31

De NSE
Aller à : navigation, rechercher

Handbook of Neuroevolution Through Erlang

Gene I. Sher

Department of Electrical Engineering and Computer Science University of Central Florida

Orlando, FL, USA

Sommaire

Foreword

by Joe Armstrong

I was delighted to be asked to write a foreword to Gene Sher's book on Neuroevolution.

To be honest I didn't have a clue what Neuroevolution was before I started reading his manuscript, but the more I read the more I became engaged in the con- tent.

Gene addresses what is a fascinating problem: How can we simulate a biologi- cal system in a computer. Can we make a system that learns from its mistakes?

Gene chose to program his system in Erlang, which is where I come in. Erlang was designed over twenty five years ago to solve a completely different problem. What we were trying to do at the time and we are still trying, is to make a lan- guage for programming extremely large systems that never stop.

Our application was telephony. We wanted to write the control software for a set of large telephone exchanges. This software should in principle run forever. The telephone networks span the planet, and the control systems for these net- works were running before the invention of the computer.

Any system like this must be able to tolerate both software and hardware fail- ures, and thus we built a programming language where failures were not a prob- lem. Our approach was to say, “well if something crashes, some other part of the system will detect the error, fix the problem and the system as a whole will not fail. ”

We also thought that such a system should evolve and change with time. It would never be the case that the software in the system would be correct, right from the start, instead we would have to change the software many times during the life of the product. And we would also have to make these changes without stopping the system.

The view of the world that Erlang presents the programmer is intrinsically dis- tributed, intrinsically changing and capable of self-repair. Neuroevolution was far from our thoughts.

Twenty years later Gene comes along and discovers Erlang - to him, Erlang processes are neurons. Well of course Erlang processes are not neurons, but they can easily be programmed to behave like neurons.

Erlang was designed to scale. So today we can run a few million processes per node, and a few dozen nodes per chip. Computer architectures have changed from the single core Von-Neumann machine, to the multicore processor, and the archi- tectures will change again. Nobody actually knows how they are going to change, but my bet is that the change will be towards network-on-chip architectures.

We're already talking about putting a few hundred to a thousand cores on a single chip, but for this to happen we have to move to network on chip architec- tures. We can imagine large regular matrices of CPUs connected into a regular switching infrastructure. So we'll soon end up with thousands of cores, each capa- ble of running millions of Erlang processes.

What will we do with such a monster computer and how are we going to program it? Well I suspect Gene has an answer to this question; he'll want to build a brain.

This book will tell you how. Will it work in the near future or will this take hundreds of years? Nobody knows, but the journey will be interesting and we might build some other useful things on the way.

Joe Armstrong

Stockholm

Dedication

To my father Ilya Aleksandrovich Sher To my mother Zinaida Lvovna Sher

Preface

We biological organisms are fragile things. We are machines whose only armor is skin, a thin layer composed of lipids and proteins, easily damageable. The cells that make us up require constant nourishment, this nourishment is supplied by our blood, which itself easily and rapidly begins to leak when our thin shield is pene- trated. We can only exist in a very limited range of temperatures. 36.7 degrees Celsius is our standard, a few degrees above that and we are uncomfortable, a bit higher and our cells begin to die and our flesh suffers permanent damage. When our temperature drops below 36.7 by a few degrees Celsius, we too can begin to suffer permanent damage unless the temperature is raised again. We burn easily. Our sensory organs are limited, we see electromagnetic radiation in only a very narrow spectrum, between 400 and 800 THz. The same for pressure waves, sound, which we only perceive between 12 and 20000 Hz. Our actuators, how and where we can move are, once again, extremely limited. The maximum speed we can achieve on land, without mechanical assistance, is 44.72km/h (record by Usain Bolt), and only for a limited amount of time, in water 2.29 m/s (record by Tom Jager). We require a constant supply of energy, food and fluid, if we do not con- sume fluids for just a few days we die, we survive only a bit longer when left without food. We are mortal, decaying, with only a few decades of being func- tional, and that only if everything else goes perfectly right.

We can study as much as we want, but at the end of the day we will still be lim- ited by our biological nature. Within our brains the signals propagate from neuron to neuron at a maximum speed of 120m/s, this cannot be improved much further. The best of our kind, Feynman, Newton, Gauss, Einstein... They are at the limit of what our specie can achieve, and yet even they are limited by time and what they can do with it. Because there is no longer an evolutionary push towards greater level of intelligence, evolution will not save us, we will not give birth to another biological specie with greater minds, and natural evolution is too slow in either case. Plus, let us be perfectly honest, the evolutionary pressure in our modern so- ciety is actually against intelligence, the Feynmans, Newtons and Einsteins of our world have less children on average than anyone else.

But we have Science on our side, and through the application of the Scientific method we are hill climbing towards deeper knowledge and understanding of what we are, how we work, and most importantly, how to improve upon our condition.

Make no mistake, there is absolutely nothing mystical about the human brain. It is but a vast graph of interconnected simple biological signal integrators. But though our brain is limited, the non biological based intelligence does not have to be so. Unlike a biological computer, the non biological one, its speed, the vastness

and the complexity it can achieve, can all be improved and increased at the speed of technological evolution, which would be under its own control. The brain is complex, and we are actively trying to reverse engineer it [1]... so we are on our way. But even if we somehow won't be able to reverse engineer our own brains, there is another approach to the creation of intelligence, by creating a new one using the basic elements that we know work, neurons, and through a process that we know works, evolution. We know that this method works, we are the proof of it. We are the proof that evolution works, that simple spatio-temporal signal processors evolved to be connected in vast topologies can produce intelligent systems. If we can supply the elements flexible enough, the environment complex enough, the evolutionary process dynamic enough, and enough computational power to simulate trillions of organisms... it will only be a matter of time... once again, we are the proof of that.

[1] The Blue Brain Project EPFL, http://bluebrain.epfl.ch

Gene I. Sher

Chapter 1 Introduction: Applications & Motivations

Abstract This chapter discusses the numerous reasons for why one might wish to study the subject of neuroevolution. I cover a number of different applications of such a system, giving examples and scenarios of a neuroevolutionary system being applied within a variety of different fields. A discussion then follows on where all of this research is heading, and what the next step within this field might be. Final- ly, a whirlwind introduction of the book is given, with a short summary of what is covered in every chapter.

One of the most ambitious and long standing goals within the field of Compu- tational Intelligence (CI), is the creation of thinking and self aware machines of human and greater than human intelligence. An intelligent system that once seed- ed, can learn, improve on itself, and then cut its own evolutionary path upwards. We have come a long way, we have made progress. Starting with symbol manipu- lation based “good old fashioned AI ” systems of the 1950s, we have advanced to artificial neurocomputation. Today, these intelligent systems can analyze images, control the walking gait of a robot, navigate an Unmanned Aerial Vehicle (UAV) through the sky, and even act as brains for artificial organisms inhabiting artificial worlds [1,2,3]. We have advanced a lot in the past few decades. Like the history of flying machines, most believed that flight could either not be achieved, or if achieved could only be done so through machines that would flap their wings or were lighter than air... Yet today, we have made super flying machines. Our tech- nological flying machines can do what the biological flying machines would not even dream off, leave and go beyond this planet. I have no doubt that the same story will repeat itself with regards to intelligent machines.

Today's most advanced approaches to computational intelligence are through neuroevolution [4,5,6], a combination of artificial neural networks and evolution- ary computation. The discipline of Neuroscience has progressed rapidly, and we've learned quite a bit about the biological neural circuits, and the process of cognition. We've also had a lot of time to experiment with Evolutionary Biology, and know very well of its power when it comes to problem solving. After all, we are one of its solutions. Neuroevolution is based on the extrapolation of the con- cepts we've learned in neuroscience and evolutionary biology, and the application of those concepts to machine intelligence.

Both, evolution and neurocomputation, are highly distributed and concurrent problem solving approaches. Evolution is the process of change in the inherited traits within populations due to the constant competition of the organisms within it. As the billions of organisms fight for resources, some survive and create off- spring, while others die out. Those that survive and create offspring pass on their traits to their offspring; and those organisms that create less offspring than others are slowly expunged from the environment. Thus evolutionary computation occurs on the entire surface of the planet, billions of organisms solving the problem of survival in a harsh environment, all in parallel. On the other hand, the carbon based cognitive computer we call our brain, is composed of over a hundred billion neurons, each neuron processing information in parallel with other neurons, each being affected to some degree by the signals it processes. The emergent property of this neurocomputation is self awareness, the mind, and the ability to learn and act upon the world.

If we are to map biological neurocomputational systems and evolutionary pro- cesses to software, we need to use a tool that makes this mapping as direct and as true as possible. Beyond the direct mapping of these processes from biological representations to their software representations, we also need to take into account their robustness, the fault tolerance of these biological processes. There is one par- ticular programming language that has all these features, the language that was developed from the very start for highly distributed and fault tolerant systems, that language is Erlang, and it will be our primary tool in this book.

        • A note from the author****

References: I have added references to a number of books and papers that I came across over the years, and which I had the chance to read. Though I have included a reference section in a number of chapters, the referenced material is neither needed, nor is it essential, for the understanding of this volume, but it is there if you'd like to take a look.

Species/Specie: The word species is both plural and singular. But when programming and having to go between different species and single species and Ids of species, and species in populations, and agents which belong to multiple species and single species … things become confusing. Thus at times I had to use the incorrect term: Specie. Nevertheless, I do believe that the use of the word Specie allowed for some discussions to be much clearer.

1.1 Motivations

I will start this book by answering the why & what-for of neuroevolution. You want to know why you should study this subject, why should you be interested in it, and what you would gain from being able to build such computational intelli- gence systems. The answer to the later stands with the fact that traditional neural networks, and thus by extension even more so with regards to neuroevolutionary systems, have already proven themselves in numerous problem domains, the fol- lowing of which is but a brief list:

1. Optimization: You have a problem to solve, you know what you want but you don't exactly know how to achieve that result, thus you need an intelligent agent to figure out the patterns of this problem for you and come up with an ef- ficient solution. [7,8,9]

2. Neurocontroller: You have a complex task that needs to be performed, the task itself changes and thus you need an intelligent agent to learn how to perform this task through experience, learn how to use the tools necessary to perform the task, and perform that task efficiently and without tiring. [10,11,12,13,14,15]

3. Invention: You have a rough idea or a database of already existing inventions and you want to create a system that will improve on the designs, creating new patents. Or you wish to explore new designs in previously unexplored fields. For this you need an intelligent agent that can extrapolate ideas from existing designs, or come up with and explore new ideas within various fields. [16,17,18]

Neuroevolutionary systems are force multipliers. Even if you yourself don't know how to program a controller that uses the robot's cameras and servomotors to move around on a rough terrain while panning & tilting its solar array to collect energy, you can evolve a brain that will perform these tasks efficiently. If you know just a little bit about financial markets, about stock market or foreign ex- change market (forex/FX), you can still create a system that learns on its own how to buy and sell profitably (but it is not easy, and thus far the profits from such sys- tems have not been staggering, nevertheless we will build such a system). The set of problem domains to which such systems are applied is growing, and as we con- tinue to advance these systems, making them more powerful and adaptable, the rewards gained from using these NN based intelligent agents will only continue to grow. Application domains like financial market analysis, robotics, art, and enter- tainment, are but the tip of the iceberg.

With regards to the why of neuroevolution, the answer stands with the fact that neuroevolutionary systems, particularly the Topology and Weight Evolving Arti- ficial Neural Networks (TWEANNS), are the most advanced forms of computa- tional intelligence creation. These systems are our best bet at creating intelligence that rivals our own, achieving what some call, a technological singularity. TWEANN systems use evolutionary processes to evolve complex systems with neural elements acting as the main building blocks. We already have proof that in- telligence through neural networks can be evolved, that proof is you and I. That which makes you who you are, that part which allows you to think, adapt, learn, is a result of billions of years of evolution. The problem domain was that of survival in a hostile and volatile environment, and the solution was a carbon based parallel computational machine, a learning system which could deal with such complex environments. These neurocomputational machines which we call brains, have been evolving for a long time, starting with a simple cell that could sense the pres- ence of some chemical percept... over the billions of years these simple systems became more advanced, these cells became more specialized for signal processing and began to resemble something we today call neurons. Evolution generated var- ious interconnected networks of these cells, various topologies, because different groups of neurons could more effectively deal with complex signals than a single neuron could... and after trillions of permutations of neural types and topologies in which they are joined, eventually a stable solution emerged... A few billions of years of building on top of that stable solution, evolving new features and neural circuits through random mutation and perturbation, and the result is a vast parallel neurocomputational system which resides within our skulls. This is the process of neuroevolution.

Why Erlang ? Though a more detailed answer will be given in Chapter-5, the quick version is that Erlang's architecture perfectly matches that of evolutionary and neurocomputational systems. Everything is concurrent and distributed, each neuron is an independent and concurrent element which processes its information along with over a hundred billion other neurons. When it comes to evolution, each organism in a population exists in parallel with all other organisms. All of these features can be easily mapped to Erlang's architecture through its process based and message passing approach to computing. And because there is such a direct match, we will not have to worry as much about how to first map these distributed and complex systems into a form that some programming language uses. Thus, because our systems will be much more concise and because our programs will so naturally represent these concurrent systems, it will be easier for us to further ex- pand them, to add new features, features that would otherwise have proven too difficult to add or even consider due to the way in which some programming lan- guage represents its programs.

By the time you finish this book, you will have created a system which can generate a population of intelligent agents which are able to interact with the real world, and simulated worlds called scapes. These intelligent agents will be pow- ered by complex neural networks (NNs), evolved synthetic brains whose topologies can expand and learn as they interact with their environment. When we get to the Artificial Life chapter of this book, these intelligent agents will be embodied through simulated robots, existing and moving around in a Flatland, interacting with each other, fighting each other, and collaborating with each other... These NN based intelligent agents will have morphologies, sensors and actuators through which the NNs will be able to learn how to interact with simulated environments and other programs... These agents will live, die, create offspring, form new spe- cies... they can be uploaded into UAVs and other mechanical bodies, embodying those machines with the ability to learn and to survive... Though some of this may sound like science fiction, it is not the case, for by the end of this book you and I will have built neurocomputational systems capable of all these things.

1.2 Applications

In this section I will provide examples and scenarios of the various problem domains to which NN based intelligent agents can be applied to. Where possible, I will cite already existing projects in that field, and the results of said projects.

When reading through the examples, you will notice that all the scenarios fol- low the basic evolutionary loop shown in Fig-1.1a. Also, when I mention neural network based intelligent agents, I simply refer to programs that use NNs to pro- cess their sensory signals and use their actuators when interacting with the world, as shown in Fig-1.1b.

Fig. 1.1 a. The standard evolutionary/neuroevol utionary loop. and b. A NN based agent. 1.2.1 Robotics

When discussing computational intelligence the first thing that comes to mind is robotics and cybernetics. How to make all the parts of a robot act intelligently and in a cohesive manner such that the robot can move around, either on wheels or legs, accomplish useful tasks, and learn from its experience. Or for example how to evolve a controller for a teleoperated robot, where the robot mimics the actions of the human agent. All of these applications can be accomplished using a neuroevolutionary system.

Chapter 1 Introduction: Applications & Motivations

Example 1: Evolving the neurocomputational brain of a robot.

To evolve a NN based intelligent agent for this particular task, we first decide on what the robot should do, and whether we want to evolve the neurocontroller (the brain of the robot) in the real world, or in a simulated world and then have it transferred into a real world robot. Whichever of the two approaches we take, we then need to agree on what types of behavioral patterns we wish to evolve, and what behavioral patterns we wish our evolved neurocontroller to avoid. For exam- ple, the longer the robot lasts without losing all of its power the better, if the robot can, when low on energy, go to an outlet and recharge itself, that type of behavior needs to be rewarded. If we wish to create a robot that services the house by clean- ing it, moving around and not bumping into anything, making tea... these things should also be rewarded. What should be punished is bumping into things and breaking stuff. Having decided on what behaviors should be rewarded and what behaviors should be punished, we then need to decide how our robot, real or simu- lated, will interface with the world.

What type of sensors and actuators our robot will use, and have access to in general? Let us in this example assume that the robot will use cameras, audio sen- sors, and pressure sensors covering its surface. For actuators it will use 2 wheels, a differential drive (like the type used in the khepera robot [19] for example).

After having decided on how the robots will interface with the environment, and what we want the robots to do, we need to develop a function that gives fit- ness points to the simulated robot when it does what we want it to do, and penaliz- es the robot when it does something we don't want it to do. A fitness function based on this reward/punishment system allows the evolutionary process to rank the NNs to see which ones are better than others, based on their comparative per- formance. We will use a simple fitness function for the ranking of how well the various simulated robots clean rooms:

Fitness = A*(# of minutes active) + B*(% of environment cleaned) - C*(# of furniture bumps) Where A, B, and C are weight variables set by the researcher, and depend on

which of these qualities the researcher finds most important. If we have decided to evolve a simulated robot in a simulated world before transferring the resulting NN into a real robot in a real world, we need to create a simulation of the environment as close to the real world as possible, an environment that should also be able to keep track of the robot's behavior. Since the simulated world has knowledge of where all the things within it are located, it can keep track of furniture collisions through collision detection. Such a simulated world should very easily be able to track the robot's fitness. After the robot runs out of energy, or after it has lived for some predetermined amount of time, the fitness tracking virtual world will score the NN that controlled the robot. In this document, we will call such self contained simulated worlds: scapes , or scape for singular.

Having now agreed on the fitness function, and having decided to simulate the robot and the virtual world using one of the more popular of such simulators, Player/Stage/Gazebo [20,21] for example, we can now run the neuroevolutionary system. The system generates an initial population of random NN based intelligent agents, each controlling its own simulated robot within a scape. After some time the neuroevolutionary system scores each NN, removes the bottom 50% of the population, and then creates mutant offspring from the surviving fit 50% (the ratio of fit to unfit agents can of course be set to a different value) of the genotypes. Af- ter the mutants have been created to replace the removed 50% of the NNs, the neuroevolutionary platform applies the NNs to the simulations again. This process goes on and on, every new generation comes with new mutants, each of which has a chance of outperforming its parent, though in most cases it will not... Given long enough time (in hours, rather than billions of years) and a flexible & powerful enough neuroevolutionary system/algorithm with a detailed enough scape, eventu- ally NN based intelligent agents will be evolved that are adapted to their chosen environment. The neuroevolutionary process will evolve NNs that can make tea, clean all the rooms, and not bump into furniture. Once highly fit NNs become pre- sent in the population, we simply extract them and import them into real robot bodies ( Fig. 1.2 ).

Fig. 1.2 Evolving fit simulated robots, and then uploading the evolved NN based controllers from a simulation into a real robot body.


The reason that we need to make our simulations of environments and robots as detailed as possible is because real sensors, actuators, environments, motors... are flawed, and there is always noise in the data which needs to be taken into account when evolving intelligent agents, so that they are ready for this noise when up- loaded to real bodies. The more detailed the simulations, the greater the chance that a NN evolved to control a simulated robot, will be able to just as effectively control a real robot.

Example 2: Evolving aerial dogfighting abilities

In another scenario, we might want a killing robot rather than a cleaning one. Lets say we want to evolve a neurocontroller for an Unmanned Combat Aerial Vehicle (UCAV), we want to evolve a neural network which will enable the UCAV to engage other fighter jets in combat. The approach will be the same as before, first we create a detailed simulation of the UCAV and the simulation environ- ment/scape. Then we develop a fitness function through which we can guide evo- lution in the right direction:

Fitness = A(Amount of damage inflicted) – B(Amount of damage sustained) + C(Efficiency) Where A, B, and C are functions specified by the researcher.

At this point we can populate the simulated skies with preprogrammed simulat- ed fighter jets against which the UCAV will fight and evolve, or, we can use co- evolution. We can create 2 separate specie populations of NNs, and instead of having the UCAVs engage in combat against preprogrammed jets, the UCAVs from one population will engage the UCAVs from another ( Fig-1.3 ). We could for example have every NN from population A, engage every individual from popula- tion B, in this manner every individual in population A and B will have a fitness score based on how many of the UCAVs it is able to destroy from another popula- tion in a 1 on 1 combat.

Then we apply selection and mutation phases to each population separately, and repeat the process... Eventually, evolution will generate capable NNs in both populations, as the NNs from each population will try to out maneuver and out- smart each other. And because we used coevolution, we could spark an arms race between the two populations in which case our UCAV neurocontrollers might achieve fitness levels even higher than when evolved against static strategies. An- other benefit of co-evolution is that both sides, specie A & B, start as incompe- tents, and then slowly improvise and improve their fighting tactics. If we had just used population A and evolved the NNs in it against static but already advanced air combat tactics, our results could have gotten stuck because the early incompe- tent NNs would have been killed too quickly, and thus not giving the scape enough time to gage the NN's performance. There could have been no evolution- ary path for the seed NNs by which to improve and score points against the al- ready existing, preprogrammed and deadly UCAVs. Coevolution allows the two species to build their own fitness landscape which provides for a smooth evolu- tionary path upwards as both species try to improve from seed, to competent level.

Fig. 1.3 Coevolution of UCAV Neurocontrollers.

With regards to using real robot systems in the evolution of NN based control- lers: To evolve the neurocontrollers directly inside the real robots, we will either: 1. Need to somehow create a supervisor program within the robot itself which would tell it when it bumps into stuff, or when the robot does not clean the room properly, or 2. We could have the NN's fitness based on how the owner responds to the robot, whether the owner for example yells at the robot (fitness point is sub- tracted) or thanks the robot (fitness point is added)... When using real robots, the evolution is slower since it has to occur in real time, and on top of that, we will not have access to all the information from which to calculate the fitness of the neurocontroller. For example, how do we establish the percentage of the house cleaned by the robot, who keeps track of this information, and how? Another problem is that real robots cost money, so it could be much more expensive as well. Do we buy a large number of robots to establish an evolutionary robotics system? Or do we buy just one robot and then try out all the different NNs in the population using that same robot, giving them access to this robot one at a time ( Fig-1.4 )... In all cases we follow a similar pattern, except that when using a scape, the scape has all the data about the robot's performance and environment, and when using real robots, we need to find another way to make those fitness value calculations.

Fig. 1.4 Evolutionary robotics experiment with 3 real robots, but a population of 6 NNs. Since only 3 robots exist, 3 NNs are uploaded into real robot bodies and have their fitness gaged, while the other 3 wait for the first 3 to finish.

There is an advantage to using real robots though. When evolving neurocontrollers inside real robots, when a solution finally is evolved, we can be sure that it will behave exactly the same during application as it did during train- ing because the real robot's fitness scores were based on its real world perfor- mance.

1.2.2 Financial Markets

Financial analysis is another area where NN based systems can be successfully applied. Because NNs are universal function approximators, if the market does have an exploitable pattern, NN based systems are our best bet at finding it. With this in mind, we can try evolving a NN based algorithmic trader.

Example 1: Automated currency trader

Unlike the stock market, Forex (FX) market deals with trading currencies and is up ~ 24/7. FX market had a turnover of roughly $4 trillion in 2011, and a huge trading volume leading to its high liquidity. It is thus reasonable to think that with a flexible enough system and a well chosen training set, we could teach a NN to buy and sell currencies automatically.

To evolve a NN based currency trader we first need to decide on how and what to present to the NN's sensors. The most direct approach would be to implement a type of sliding window protocol with regards to the traded currency's historical data. In this approach we feed the NN the historical data from X number of ticks (A tick is a calculated opening or closing price taken every K number of seconds) until the current time T, and ask it to make a decision of whether to buy, hold, or sell this currency at time T+1, as shown in We could for example inform the NN that yesterday Euro traded against the Dollar at a ratio of 1.4324, and then ask the NN to output its decision on whether we should buy, sell, or hold our Eu- ros today.

Fig. 1.5 A currency trading NN that uses a sliding window protocol.

Let's set up a system where we wish to trade Dollars against JPY (Japanese Yen). In this scenario our scape will be a simulated FX market using real histori- cal data. Our NN based intelligent agent will interface with the scape using sen- sors which read some amount and some particular combination of historical data, and then immediately output a trinary signal, -1, 0, or 1. The output will be fed in- to an actuator which will, based on this signal, make a decision of whether to buy, sell, or hold (if there is anything to hold) a certain amount of JPY. As soon as the trade request is made, the scape will move the sliding window 1 tick forward, and feed our NN the new set of historical data. This can be done for a 1000 ticks for example, after which the scape will calculate the fitness of the NN, which in this case can be the actual profit after those 1000 ticks. We can set up the fitness func- tion to be as follows:

Fitness = 300 + P

Where 300 is the amount of money the NN initially started with, and P is a pos- itive or negative number that represents the amount of profit generated during 1000 ticks.

Again we would use evolution to evolve a population of this type of currency traders, choosing those that were more profitable over those that were not, letting them create mutant offspring with yet higher profit generating potential. Once our neuroevolutionary system has finally generated a profitable NN, we would test it on new currency data to make sure that the trading NN can generalize and be used on currency data it has not yet seen, and if it can, we would then modify its sen- sors and actuators to interface with the real FX trading software instead of a simu- lated one. We will build this type of system in the Applications part of the book.

Example 2: Financial oracle

Another option is to create a financial oracle, which instead of trading directly, simply predicts whether price of the currency will go up or down during the next tick, or during the next T amount of time. We could use a setup very similar to the one we've used in Example-1, but instead of the NN's fitness being based directly on how much profit it makes, it would be based on the number of correct predic- tions. The fitness function could then be formulated as follows:

Fitness = P(correct # of predictions) + N(incorrect # of predictions)

Where P and N are weight parameters which we would choose based on how aggressive or careful we want our system to be when making a prediction. The evolved NN based intelligent agent would print its predictions to screen, which we would then take into consideration and execute the trade ourselves.

Example 3: Financial oracle committee machine

Finally, because our evolutionary approach will produce many NNs, and be- cause we can have multiple neuroevolutionary systems running at the same time, we could form NN oracle committees. Some of the champion (high fitness) NNs will have very different internal structure from other champions, we could form groups of these high fitness NNs, ask them all the same question (input signal), then weigh their votes (output signal), and base the final suggestion of the com- mittee on the weighted average of these votes.

A committee machine is simply a group of trained agents, where the final ac- tion is based on this committee as opposed to being based on any single agent. Furthermore, one can setup a committee in different ways. We could form the committee from champion NNs which were evolved on all historical information indiscriminately, and then simply ask the group of these champion NNs the same question and weigh their votes. This type of committee is called a homogeneous committee . Or we could create 7 different populations, and then evolve each of the 7 populations on a different training set. We could evolve the first population on the financial historical data of every Monday in the dataset, the second population on the financial historical data of every Tuesday, and so on. We would do this be- cause the market's behavior has patterns, and those patterns are specific to certain months and certain hours of day due to the market's dependency on seasons (when certain crops become available for example) and on active trading hours of certain countries (when USA's brokers sleep, Japan's brokers are actively trading, and vice versa). These patterns might require different trading strategies, and each population would concentrate on that particular trading strategy. After each of the- se populations has evolved a set of high fitness NNs, we would extract the cham- pions from them and put them into groups within the committee. The committee would then filter the input signals, routing the signals to the cluster of NN cham- pions which specializes in that particular data set (Monday data, or second half of the day data...). These types of committees are called filtered or specialized com- mittees . These two different types of committees are shown in Fig. 1.6 .

Fig. 1.6 The architectures of the homogenious and specialized committee machines.

Basing our trading decision on a large population of highly diverse champion NNs could yield a safer trading signal ( Fig-1.7 ), since for our whole committee's final action to be wrong, it would require for the majority of the champion NNs to all be wrong at the same time. We could further try to tune our committee system by specifying that at least X% of NNs in the committee have to agree on the trad- ing signal before the committee executes the trade... Though this might decrease the number of trades our committee machine executes in total, it could further im- prove the chance that when the trade is executed, it is a lucrative one.

Fig. 1.7. A committee machine of NN traders.

1.2.3 Artificial Life

Artificial life, or ALife, imitates traditional biology by trying to recreate bio- logical phenomena in software. The goal of ALife is to study logic and emergent phenomena of living systems in simulated environments. The organisms populat- ing these simulated worlds should also have brains and the minds that brains gen- erate. Artificial neural networks are the perfect choice for such a role. Neuroevolution allows us to populate these simulated worlds with learning organ- isms. Through neuroevolution the simulated environments allow the behavior of these artificial organisms to evolve over time, changing as the organisms interact with the environment and compete against each other.

If the software based sensors and actuators are themselves implemented as nodes, similar to how neurons are implemented, then through mutation operators they too can be added and removed to and from the NN during evolution. Through such an implementation we can then allow a neuroevolutionary system to evolve not merely the brain of the artificial organism, but also its morphology. Using this approach, the evolutionary processes will allow for the created mutant offspring to expand and try out different combinations of sensors and actuators, and thus po- tentially different types of bodily morphologies.

Example 1: Predator vs. Prey

We could populate a simulated 3d world with two populations of artificial or- ganisms. Those artificial organisms could be in the form of small tanks controlled by NNs. The prey tank population would have wheel based propulsion, and no gun turret. The predator tank population would start with track based propulsion, and a small gun turret which it could use to fire. Furthermore, each organism would start with range based sensors. Each tank would have a certain amount of energy, and a maximum lifespan of 20 simulated hours. In this simulated world, the prey organisms can only gain energy for propulsion by drinking it from the randomly scattered and self replenishing energy pools. The predator organisms can only gain energy by killing the prey, thus consuming their energy. We will ac- tually implement and use our neuroevolutionary platform in a similar, though slightly simpler, ALife simulation in Chapter-18.

In the scenario above the prey start with the following list of sensors: [Range_Sensor], and the following list of actuators: [Differential_WheelDrive]. While the predators start with [Range_Sensor] sensors, and [Differen- tial_TracksDrive,Gun_Turret] actuators. Each sensor and actuator name is a tag, a name of a program that we need to develop. These programs either act as simulat- ed sensors/actuators (if the robot itself is simulated for example), or interface with a hardware driver of the real sensors/actuators. Furthermore, each sensor and actu- ator will need to have some visual and physical representation if implemented in a virtual environment. The NNs should be able to poll the sensor programs for sig- nals, and output signals to the actuator programs, which themselves can then fur- ther post-process and act upon those signals.

In this particular example, the offspring are simply mutated versions of their fit parents. In the real world, not only the neural structures but also the organism mor- phologies evolve. Morphological evolution can be integrated as a sort of side effect of neuroevolution. We can accomplish this by extending the list of mutation opera- tors used by our neuroevolutionary system. One of these possible additional muta- tional operators could be an Add_Random_Sensor, or Add_Random_Actuator . Us- ing the sensor and actuator based mutation operators, we could generate offspring which will have a chance of integrating a new sensor or actuator into their simu- lated bodies. Through new sensor and actuator incorporation the organism's mor- phology, visual representation, and physical properties would change, and thus al- low evolution from simple organisms, to the more complex ones with regards to both, morphology and neurocognitive ability (structural morphology and neural network based brains).

To use Add_Random_Sensor and Add_Random_Actuator mutation operators, we also need to build the extended sensor and actuator lists, so that the neuro- evolutionary system will actually have some new sensors and actuators to randomly choose from when using these mutation operators. For the prey we could provide the following sensor list: [Range_Sensor,Color_Sensor], from which the Add_Random_Sensor operator could choose its sensors. And the following list of actuators: [Differential_WheelDrive,Drop_Smokebomb,Extended_Fueltank,

Change_Color,Range_Sensor_PanTilter,Color_Sensor_PanTilter]. For the pre- dators we could provide the same sensor list as for the prey, and the follow- ing actuator list: [Differential_TracksDrive,Rockets,Afterburner,Change_Color, Range_Sensor_PanTilter,Color_Sensor_PanTilter].

There is a problem though, in such ALife simulations we cannot use the “gen- erational ” evolutionary approach, where we wait for all organisms to finish their evaluation on the problem (In this case, surviving in the environment) and then calculate which ones are more fit. Instead we need to maintain a constant or semi- constant population size within the environment, we need to set up a steady state evolutionary loop . One of the ways in which to set up such a system is as follows: When a NN dies, a tuple composed of its genotype and its achieved fitness score is entered into a Dead_Pool list of some finite size. Immediately afterwards, a new offspring is generated of the same specie, with the parent of the offspring chosen from the NN genotypes stored in the Dead_Pool. The probability with which a parent genotype is chosen from the Dead_Pool is based on that genotype's fitness. In this manner the populations are constantly replenished, as soon as an organism dies, another one of the same specie is spawned.

Thus the hypothetical sequence of events in such an ALife system could go as follows: The two initial species of NNs controlling predator and prey tanks are created. Each NN has its own random minimalistic initial NN topology and set of sensors and actuators, the sensors and actuators the NN is using are reflected by the morphology and physical properties of the tank the NN is controlling in the virtual world. The NN controlled organisms/tanks interact with the environment and each other. When an organism dies, another organism is generated by select- ing a parent NN from the dead pool of that specie with the probability dependent on that NN's fitness. The offspring is a mutated version of the parent. Through statistics alone, one of the offspring undergoes a mutation of the form: Add_Random_Sensor, which adds a Color_Sensor. Though perhaps this particular organism will not have the NN capable of making any effective use of the new sensory data, and thus will die out or be just as fit as other NNs not using color da- ta, eventually, after a few thousand of such occurrences, one of the mutant NNs will have the topology capable of using the Color_Sensor at least to some degree. If the predator and prey tanks are of different colors, then the color sensing NN mutant will have an advantage over other organisms since it will be able to tell the difference between prey and predators, and know which ones to avoid. If the color sensing mutant offspring is a predator NN (NN controlling a predator tank), then it too will have an advantage over its fellow predator NNs, since it will now be able to discern the prey from the predators, and thus be able to choose its targets more effectively.

Over time, evolution will produce predators that use better weaponry (new ac- tuators) and larger NNs capable of more complex reasoning and the ability to take control of these newly integrated actuators. At the same time, only those prey will survive that can better evade such predatory tanks... and only those predators will survive which can hunt these smarter prey... In this manner, evolution will pro- duce smarter and more complex versions of predator and prey, with better strate- gies, more sensory modules, and greater offensive and defensive capabilities ( Fig- 1.8 ). Evolution will fit together through trial and error, fitter NNs and morpholo- gies of the organisms inhabiting the scape. Evolution will create these fit NNs (and their tank representations in the ALife simulation) by trying the different var- iations and permutations of neurons, sensors, and actuators.

Fig. 1.8 A possible evolutionary path of the predator and prey tanks. Evolving morphology and the NNs.

This scenario should remind you a bit of the robotics application, they are re- lated. Although in ALife the goal is to simply observe the evolved behaviors and the emergent properties, rather than to upload the evolved NNs to real robots, the transference to hardware is also possible. If these simulated sensor and actuator modules have real world counterparts, in which case this evolutionary approach will not only evolve the brains of these autonomous hunting and evading tanks, but also their morphologies by putting together a good combination of offensive and defensive modules on some standardized chassis, being controlled by the fit NN which can effectively use these modules, then the utilization of the evolved NNs based agents in actual robot systems would follow the same steps as in the robotics application example.

The scenario in this section is actually very easy to implement, and we will have a chance to develop a similar 2d version of this ALife simulation.

1.2.4 Image Analysis & Computer Vision

Image data is just like any other type of data, and thus we can evolve NNs to analyze and pick out certain features within the image. As alluded to in a previous section, where the predators and prey evolved the ability to recognize the various environmental and organism features through their color and range sensors, NNs can evolve an ability to process visual signals. In this section the possible applica- tion scenario concentrates only on the computer vision and image analysis applications.

Example 1: Evolving a NN for visual feature selection

As shown in Fig-1.9 , in this problem we encode the visual data in a manner ac- ceptable by the neural network. For this particular problem we will need to create a training set, a list of tuples composed of images, and the cluster where that im- age belongs. We then run the NN through the training set of images, and reward the NN with a fitness point every time it clusters or recognizes the image correct- ly, and give it 0 points every time it does not. In this scenario, we want the NN to cluster the happy faces into group A, and sad faces into group B. Once the NN has went through the whole training list, it will have its full fitness score.

Fig. 1.9 The image is first encoded as a vector (a bitmap for example), and then fed into a NN which decides whether the image belongs to cluster A or B.

We generate a population of NNs, each NN goes through the training set and at the end once all the NNs are scored, we again choose the most fit networks and let them generate mutant offspring. Some of these offspring will recognize or cluster the images better than others, and through this selection-mutation-application loop, eventually the neuroevolutionary process will generate highly fit NNs. Once a particular/goal fitness score is reached, or the population no longer generates or- ganisms of greater fitness, we choose the most fit NN within the population, and count it as the solution generated by our neuroevolutionary system. This champion NN can now be applied to the real world image analysis problem for which it was evolved.

Example-2: Facial stress signal analysis

To create the necessary training set for this type of problem, we would have to manually generate images of faces that show stress and those that do not, and then flag these training images with stress | no_stress tags. With this type of training set, a neuroevolutionary system can properly score the NN's performance and fitness. In a more complex version, the images do not have to be static, they could be fluid images coming directly from a camera.

Thus performing the same type of training as in Example-1, we would evolve a stress recognizing NN, which could then be used in various applications. This type of NN system can be connected to a camera at an ATM machine for example, and then used to signal duress, which might further imply that the withdrawal is being made under pressure, and that the person might require help. There are numerous other possible, and useful applications for such a NN system.

1.2.5 Data Compression

Even data compression can be done through NNs. The next example demon- strates two simple approaches to give an idea of how to tackle this type of problem.

Example 1: Transmitting compressed signals.

Shown in Fig-1.10 is a feed forward neural network composed of 3 layers. The first layer has 10 neurons, the second 5, and the third 10 neurons again. Assume we have a dataset A composed of 100 values. We divide these 100 values into sets of 10, and then evolve the weights of the NN such that it can output the same signal as its input. So that for example if the input of the NN is the vector: [0,0,0,1,0,1,0,1,1,1], then its output would be the same vector: [0,0,0,1,0,1,0,1,1,1].

Once the NN is able to output the same signals as its input, we break the NN into two parts. The first is the compressor/transmitter part composed of the first two layers, and the second is the decompresser/receiver network composed of the last, 3rd layer, as shown in Fig-1.10. When you pass the signal of length 10 to the transmitter NN, it outputs a compressed signal of length 5. This signal can then be stored or passed to another machine over the net, where the receiver NN is wait- ing. This receiver NN would accept the compressed signal of length 5 and convert it back to the original signal of length 10. This is so because the combination of the Transmitter-NN and Receiver-NN, form the original NN system which accepts and outputs the same signal of length 10.

Fig. 1.10 A simple Feed Forward NN composed of 3 layers with a topology of: [10,5,10].

Though simple, this data compression NN can produce a half sized compressed signal, which can later be decompressed by the receiver NN. This approach could be further extended. It is not known ahead of time whether the compression of this magnitude is possible (or perhaps whether it is possible to compress the signal at all), and whether it is possible with a neural network of the topology we originally chose as a guess (the [10,5,10] feed forward NN).

A better approach to this problem would be to use neuroevolution and evolve the topology of the compression NN. In such a scenario we would constrain the NN to use the sensor (input channel) which can read signals of length 10, an ac- tuator (output channel) which outputs a signal of length 10, and one hidden layer whose length is initially set to 1. Then through neuroevolution we would evolve various topologies of the hidden layers. In this manner, a compression solution can be evolved for any unknown data. Evolution will strive to produce a fit compres- sor, though perhaps not optimal, it will be evolved for that particular type of data, rather than any general dataset.

Fig. 1.11 The Feed Forward NN broken into its transmitter/compressor and receiv- er/decompressor parts.

1.2.6 Games & Entertainment

It would be great to see Non-Player Characters (NPCs) in a computer game evolve and advance over time as you play. Wouldn't it be great if the NPCs inside a game learned from their experience, and interacted with you in new and exciting ways, with you never knowing what they would come up with next? It would keep the game challenging, the game would be a world of its own, and you interacting with that living and breathing system. Because the NPCs already have morpholo- gies and the game usually has goals (gathering gold coins, fragging other players, driving better than others within the race...), it would be very easy to extract a fit- ness function for NPCs, and then use neuroevolution to evolve new NPCs as time went on. This would be similar to the Artificial Life example, here the artificial organisms are the NPCs, the game world is their scape, and their fitness functions would be based on the game's goal. Not much would have to be changed from the examples given in the ALife section, except that now you get to interact with them through your own avatar within their scape. Games would become ALife simula- tions, with you a traveler through the simulated world.

Another approach to improving games is by using a neuroevolutionary system to generate new game features, as for example is done in the Galaxy War [23] game. In Galaxy War, the neural networks determine the various properties of the player's ship weapons, such as the shape of the laser beams, the shot's trajectory, the color of the laser and plasma blasts, the speed of the projectiles... In that game, each player can further choose to randomly mutate the NN whose output parame- terizes/specifies the weapon's various properties. In Galaxy War, the player can find new weapons, and new mutated NNs specifying the weapon's properties are constantly generated, allowing for an ever changing and evolving weaponry to populate the game world. In this game, you never run out of the different types of weapons you can find, the game content evolves with the players, it never gets old, things are always changing. This game feature could further be extended in the future to also have the NPCs evolve as well, based on the experience of all NPCs and how well they performed against the players …

Another application is the generation of terrains. The NN's input could be the coordinate in a game world, and the NN's output could be the terrain feature. For example a population of NN generated terrains are released for some game to the beta tester players of the said game. The next day we ask the players to score the various terrains they tried out, to see which they liked best. After the players score the various terrains, we average the scores for each terrain, and let these average scores be the fitness scores of the terrain generating NNs. We then use neuroevolution to generate new terrains from these best scoring NNs, and repeat the process. After a few months, the beta testers themselves would have guided the evolution of the terrains in their game, terrains that they find most appealing.

Although this and the previous application requires a human in the loop, the re- sult is an exploration of possibilities, possibilities that would not have been con- sidered without the help of the evolutionary process and the NN's generating the features of the said applications. And because neural networks are universal func- tion approximators, neuroevolution provides the flexibility and the variety of the results that would have been more difficult, or perhaps even impossible to create with other methods.

1.2.7 Cyber Warfare

If you have been reading these listed application scenarios/examples in order, then you're already familiar with the pattern of applying a neuroevolutionary sys- tem to a problem. The pattern is:

1. Create a virtual environment (scape) for the problem, where if the goal is to simply train the NN on some training set (rather than interact with some simu- lation), then that virtual environment should interface with the NN's sensors and actuators, present to it the training set, and gage the NN's performance.

2. Let all the NNs in the population solve the problem, or be trained on some training set until the terminating condition is reached, at which point the scape scores each NN's performance.

3. Sort the NNs in the population based on their performance.

4. Choose some percentage of the top performing NNs, and use their genotypes to generate mutant offspring by mutating the parent NN's topology and/or weights.

5. Apply the new population composed of the top performing NNs and their mu- tant offspring to the problem again.

6. Finally, repeat steps 2-5 until some terminating condition is reached, where such a condition could be an emergence of a NN in the population with some fitness/performance level that you are seeking, or when the population is no longer generating better performing NNs, when evolution has stagnated.

In the scenario covered here, the neuroevolutionary system is used to evolve NN based computer system attackers and defenders, cyberwarfare agents.

Example-1: Cyber warfare, evolving offensive cyberwarfare agents.

The way we apply a neuroevolutionary system to a cyber warfare or network security application is similar to the way we dealt with robotics and artificial life, as shown in Fig-1.12 . In this particular scenario the simulated environment is not a 2d or 3d world, but a computer network, with various simulated hosts. Instead of having the evolving NNs control simulated tanks or UCAVs, they control offen- sive and defensive software packages, like metasploit [24] for example. In the fig- ure, these NN based cyberwarfare agents are represented with a red lightning bolt. The sensors are used to gather signals coming from the network to the local host on which the NN and the attack package is running, and the actuators use the out- put of the NN to interface with a software package that can execute attacks, a parametrized metasploit suit would work in this scenario as such an attack pack- age. Metasploit can be looked at as a large library of prepared attacks/exploits, the way to select the various attacks, methods of running those attacks and options to execute them with, can be parameterized, and the NN's output can then be used to make these various choices.

Let's say the NN sends to an actuator a vector of length 20, this particular actu- ator uses the values in this vector to decide: 1. Which particular attack vector to select from the metasploit or another custom tailored network penetration pack- age. 2. What options to use with this attack. 3. To what (IP, port...) target should this attack be applied to. The scape where we would evolve these cyber warfare agents is a simulated computer network, built using a network simulator like ns3 [25], or DETER [26] perhaps. The simulated computer networks would have simulated hosts with some intrusion detection capabilities. The scape which has access to the entire network simulation could then see whether any of the tar- gets have been compromised, and whether the attacker was traced, and thus gage how well a NN is attacking the simulated targets, and how well it is concealing it- self from detection, if at all. As before, we use an entire population of such NNs, each interacting with its own scape. After some time, when they are all scored based on their performance, we again sort them, select the best, and create a new population composed of these champion NNs and their mutant offspring. This is effectively an ALife simulation, the primary difference is the avatars controlled by the NNs, and the environment being not a 2d or 3d system, but a simulated com- puter network. As before, the NNs are simply evolving to exploit the environment they inhabit.

Fig. 1.12 Evolving a population of NN controlled network attackers. The offensive network program, like metasploit for example, is controlled by a NN and is represented as a red lightning bolt in the figure. The other simulated hosts on the network are represented as laptop computers.

As with ALife, eventually the neuroevolutionary process will produce NNs that are capable of properly using their parametrized defensive and offensive software packages. Evolution will produce NNs able to “see ” and properly react to the sig- nals intersecting its host's ports, and NNs able to select based on the situation, the proper offensive or defensive programs to protect its host, and attack others.

Example-2: Coevolution of NN controlled cyber attackers and defenders Again, as in the ALife example, we can evolve all the NNs in the same scape, interacting with, and attacking/defending against each other rather than the simu- lated static hosts as in Example-1. We could evolve two species, one composed of cyber attackers, and another composed of cyber defenders (just like the scenario where we evolved predator and prey tanks), all populating the same simulated computer network, as shown in Fig-1.13 . The goal of the cyber attackers, their fit- ness, is based on how well they can attack other hosts on the network. The fitness of the cyber defenders on the other hand can be based on how well they defend the host machine they are located on. The cyber defenders could be NNs whose sen- sors gather the signals which intersect their host, and their actuators would control the ports, either allowing those signals through, or blocking and tagging those signals as offensive. The defensive NNs could simply be evolved for the purpose of effectively controlling an intrusion detection program.

Fig. 1.13 Co-evolving cyberwarfare agents. Attackers and defenders populating the same network scape.

At the same time there should be a number of untouchable simulated hosts which send out normal, non aggressive signals to the various hosts on the network, thus simulating normal network traffic. Because the simulated network, and the operating attackers and defenders are all within the same scape, the scape will know which of the signals are non offensive (coming from the untouchable normal hosts), which are offensive (coming from the cyber attackers), and thus the scape will be able to properly distribute fitness scores to the NN based systems interfac- ing with it. The fitness functions for the cyber attackers and defenders respectively could be of the following form:

Attacker_Fitness = A(# cyber defenders compromised) – B(# times detected)
Defender_Fitness = A(# attacks blocked) + B(# normal signals passed) – C(# compromised)

In this manner we can co-evolve these two species, we hope that this co- evolution will spark an arms race between them. Since at the very start both of the species would be composed of random and incompetent NNs, both of these spe- cies can start on equal footing. Both of these species, attackers and defenders, can then slowly climb upwards as they try to out-compete each other in their environment.

1.2.8 Circuit Creation & Optimization

The way each neuron in a neural network processes information is by first ac- cumulating its incoming signals, then weighing those signals, and then adding the weighted signals together and passing the sum through an activation function (any type of mathematical function). The most commonly used activation functions are: sigmoid, sinusoidal, and Gaussian. But we could just as easily allow the activation functions to be selected from the following list: AND, NOT, and OR, and set all the neural weights to 1. If we do that, then our NN is essentially a digital circuit, and our neuroevolutionary system will be evolving not new neural networks, but new and novel circuit designs, as shown in Fig-1.14 .

Fig. 1.14 Similarities between evolving NNs and evolving digital circuits.

The neuroevolutionary platform we will be developing in this book will be very flexible, and every feature fully decoupled from the rest. We will allow the re- searcher to specify the list of activation functions from which the neuroevolutionary platform should be choosing its activation functions during the offspring creation and mutation phase, and so switching from evolving neural networks to evolving digital circuits will be as easy as specifying whether the plat- form should use list A, composed of [tanh, sin, Gaussian...] or list B, composed of [AND, NOT, OR...] activation functions.

Example-1: Evolving a circuit.

Because we evolve our circuits with a particular behavior and feature set in mind, we can easily come up with a fitness function for it. The first thing we do is build a training set, a list composed of training tuples: [{X1,Y1}, {X2,Y2}...{Xi,Yi}], because we know that for every input Xi, we want our circuit to produce an output Yi . We want to evolve not only a correct circuit, that has the logic based on the training set, but also an efficient circuit. Our fitness function should take into account the transistor cost of every gate. To take both, the cor- rectness and the efficiency of the circuit into account, our fitness function should be as follows:

Fitness = A(% of correct outputs) - B(# of AND gates) - C(# of OR gates) - D(# of NOT gates)

Where A, B, C, and D would be set up by the researcher, and be dependent on how important each of the said features is. The parameters (or even potentially functions) B, C, and D ensure that the evolved circuits which have the lowest number of gates but the same correctness as their less efficient cousins, will have a higher fitness.

Having now set up the training set, the activation function (AF) list our evolv- ing network will take its AFs from, and the fitness function our neuroevolutonary system will use, we can start evolving the circuits. As before, we start by creating a random population of minimalistic digital circuits (NNs). We let each of the networks go through the training set, and using the fitness function we let the scape calculate the circuit's fitness. Once every circuit in the population has fin- ished, we sort the circuits based on their fitness, choose the most top performing of the networks, and create their offspring. We generate the circuit offspring in the same way we did with NNs: first the parent is cloned, and then we mutate the clone's topology and parameters. The parametric mutation operators could include one which mutates one activation function into another. For example the paramet- ric mutation operator could take any logic gate in the circuit, and then change it to one of the other gates in the gate list available (OR, NOT, AND). On the other hand the topological mutation operator could add a new gate in series or in parallel with another randomly chosen gate already in the network.

Once all of the mutant offspring have been created, we apply the new popula- tion composed of the fit parents and their mutant offspring to the training set again, repeating the whole process. In this manner we evolve a circuit through a complexification process. Let's go through a possible flow of events in evolving a XOR operation from the AND, NOT, and OR gates as shown in Fig-1.15 . In that figure the initial population at generation-1, is composed of 4 minimalistic random circuits: A, B, C and D. We apply each circuit to the training set com- posed of the input: {[1,1],[-1,-1],[-1,1],[1,-1]}, to which a XOR operation is ex- pected to produce an output: {-1,-1,1,1}.

Fig. 1.15 This figure shows a possible flow of events in evolving a XOR operation from AND, NOT, and OR gates.

Let's say that in this simple example, the fitness function is: Fitness = (% of correct outputs). Each circuit in the population is fed the input signals, and the cir- cuits output is checked against the expected output for each input (Note that the NOT circuit (A) can only accept the first value of each input vector in the input vector set, and does not care about the value of the second value). Once all the cir- cuits have produced their outputs to the given inputs, using the fitness function we calculated that circuits A and B belong to the top 50% of the population, with scores 0.5 and 0.75 respectively. To maintain a constant population of 4, we re- move the bottom 50% of the population (circuits C and D), and create two off- spring from the fit circuits. Though both circuits A and B survive to the second generation, because circuit B scored higher than A, we allow it to create both of the new offspring.

Circuit B creates 2 mutant offspring called: E and F. Offspring E is created by applying two random topological mutation operators to B's clone. The first opera- tor adds a random gate before OR, in this case that random gate is AND, and the second operator adds the OR gate in parallel to the new AND gate. The resulting circuit is ((p OR q) or (p AND q)). Offspring F is created by applying one muta- tion operator to B's clone. This randomly chosen mutation operator adds a NOT gate after the existing OR gate. The resulting circuit is: NOT(p OR q). Finally, the two fit parents themselves, A and B, also both survive to the next generation.

Once all the circuits of the second generation have been created, we again feed them the Input vectors, and compare their outputs to the Expected Outputs . During the second generation, circuits B and E are the best performing, both with an equal score of 0.75. Thus both B and E survive to the next generation, and each creates a single offspring.

The third generation is composed of the circuits G, B, E, and H, with G and H being the new mutant offspring. We again test each of the circuits against the In- put and the Expected Output. During this evaluation, Circuit H gets a perfect score of 4/4, which means that it has evolved the XOR behavior and is the evolved solu- tion to the problem.

Though the flow of events could have went differently, sooner or later, even if it would have taken a few hundred generations and thousands of mutation opera- tors, a XOR circuit would have been evolved.

Example-2: Optimizing an existing circuit

If we already have an existing circuit, we could try to optimize it by attempting to decrease the number of gates used, or by creating a version of the circuit whose topology is more concise. Neuroevolution can use any mutation operators that its designer wishes to implement. Usually it works by mutating various parameters, and by adding elements to, and/or deleting elements from, the NN. Complexification is the neuroevolutionary process of starting from a minimalistic network and slowly making it more complex by adding various features and struc- tures, as the evolutionary process tries to create a fitter organism. The process of pruning on the other hand starts from a complex topology, and then slowly chips away the unnecessary elements of the network, while trying to maintain the same functionality and fitness. A complete neuroevolutionary system could use a combination of both, thus it is possible to start with an existing complex network, and re-factor it into a more concise version if that's what the fitness function dictates of evolution.

This type of problem would be tackled by initially creating a population com- posed of the circuit we wish to optimize and a few of its mutant offspring. We then apply this initial population to the training set. As would be expected, the ac- tual circuit we wish to optimize will get the perfect score with regards to function- ality, but if any of its offspring possess the same level of functionality but with a more concise topology, they will have the higher total fitness. We could use the same fitness function as in Example-1 to accomplish this:

Fitness = A(% of correct outputs) - B(# of AND gates) - C(# of OR gates) - D(# of NOT gates) Here we would make sure that A has the greatest amount of weight, so that the circuit does not lose any of its functionality during its topological optimization.

The B, C, and D of the fitness function will ensure that if through evolution one of the mutants is a more concise version of the original network, yet it still retains the same functionality, its fitness score will be higher than that of the original circuit.

The possible mutation operators that we'd want our neuroevolutionary platform to have for this type of problem are:

1. Desplice : This mutation operator deletes a randomly chosen gate in the circuit, and directly connects the wires that originally lead to and from this chosen gate respectively.
2. Perturb_Gate : This mutation operator chooses a random gate and then changes it from one type to another (from OR to AND or NOT for example).
3. Hard_Delete : This mutation operator removes one of the gates and the wires leading to and from it, potentially even breaking the circuit.

To produce offspring, we would apply a random number of these mutation op- erators to each clone. Thus by performing the same steps as in the previous exam- ple, then if it is possible, a more efficient digital circuit will eventually evolve. One of the possible open source circuits to which this could be applied is the OpenSPARC [27] CPU. It is a very large and complex circuit, perhaps applying a pruning method to some parts of this circuit would work. In the same way, one could try to evolve OpenSPARC beyond what it currently is through complexification. Finally, we could also evolve new modules for OpenSPARC, advancing the CPU even further. With a clever enough graph-evolving algorithm, this is possible, and advanced and highly effective NN branch predictors have been evolved [34] for CPUs in this manner. Perhaps soon it will be easier to evolve the next generation of CPUs and all other advanced circuits, rather than further engineer them.

1.2.9 Optimizing Shapes and Structures

Even the optimization of multidimensional shapes and structures can be ac- complished through neuroevolution. Let's say that you're trying to create some aerodynamic 3d shape, you also need to make sure that this particular structure adheres to certain constraints, perhaps some specific mass and volume constraints. How would you present this problem in such a way that you could use a neuroevolutionary system to solve it?

One way to solve this problem is by using the NN to paint the shape on a mul- tidimensional substrate ( Fig-1.16 ). A substrate is a hypercube with each of its co- ordinate axis ranging from -1 to 1 . The range for each axis is then divided into K parts, effectively determining the resolution of that axis. For example if we have a hypercube with 2 dimensions: X and Y, and we decided that each dimension will have a resolution of 3, then we divide X and Y ranges into 3 sections each. The X axis' sections will range from: -1 to -0.33, -0.33 to 0.33, and 0.33 to 1. The Y axis' sections will range from: -1 to -0.33, -0.33 to 0.33, 0.33 to 1 . We then give a coordinate to each section, placing that coordinate at its section's center. The sec- tion coordinates for X and Y will be at points: -0.66, 0, and 0.66 . What this ac- complishes is that we now have created a plane whose resolution is 3 by 3, and each “pixel ” on this plane has its own [X,Y] coordinate. For example as shown in Figure-1.16b , the pixel in the very center is 0.66 by 0.66 units of length on each side, and has a coordinate of [0,0]. The units of length used, if any, is determined by the researcher, and is chosen to be specific to the particular problem this meth- odology is applied, and scaled accordingly.

Fig. 1.16 Examples of multidimensional substrates, with a single dimension in (A), two di- mensions in (B), and 3 dimensions in (C).

Instead of a 2d substrate, we could use the same approach to create a 3d sub- strate on the X, Y, and Z axis. Once we've decided on the resolution of each axis in a new 3d substrate we wish to create, the way we can make the NN “paint ” or form the 3d shapes in this hypercube is by feeding it the coordinates [X,Y,Z] of each voxel , and use the NN's output to determine whether that voxel is expressed or not, whether it is filled or whether it's empty, and even what color it should be given. For example if we have a NN that accepts an input vector of length 3, and outputs a vector of length 1, then we can feed the coordinates (vector: [X,Y,Z]), and use the output vector (vector: [O]) to determine whether that voxel is filled in or is empty. If O >= 0, then the voxel is filled in, and if O < 0, then the voxel is empty.

This method could be extended even further, for example our NN could output a vector of length 3: [E, C, P], where E would determine whether the voxel is ex- pressed, C would determine the voxel's color, and P could determine the physical properties of that voxel. A NN used to shape and determine the properties of a substrate is shown in Fig-1.17 . In this figure a 3d substrate with a resolution of 3x3x3 is simulated in a scape. The NN interfaces with the scape, and requests a coordinate of each voxel in the substrate, and outputting a vector: [E,C,P], which specifies whether that voxel is expressed, its color, and from what material the voxel is made.

Fig. 1.17 A NN interfacing with a scape, in which it paints the form, color, and material used on a 3d substrate.

There are a number of Neural Network systems that are used for this approach, popularized by HyperNEAT [28], painting shapes on the 2d and 3d substrates. For example, a 2d version of such system is the Pic Breeder [29], and a 3d version is the Endless Forms project [30].

In the next example we will examine a possible scenario of evolving an opti- mized shape for a gas tank using a neuroevolutionary system.

Example-1: Evolving the shape for a gas tank.

We first decide on the number of dimensions. Because a gas tank is a three di- mensional object, we want to use a 3d substrate to evolve its shape. Because the substrate's axis are each -1 to 1 , and what we're evolving is a physical object, we will need to decide what -1 to 1 represents in real physical units of length, that is, how this range scales. We could decide that -1 to 1 on a substrate is equivalent to - 0.5m to 0.5m in real space. Thus our substrate, when converted to physical units, is a 1 cubic meter slab of material, on which we will carve out (evolve) the shape of the gas tank. We also want to use high resolution for each axis, high enough that the NN has a smooth enough 3d slab of “clay ” to shape, but we don't want the resolution to be too high, since we will need to pass each voxel's coordinate through the NN. If for example the resolution of each axis is 1000000, then there are a total of 1000000*1000000*1000000 coordinate triplets that will need to be passed to the NN, which might make the evaluation of the physical properties of each evolved shape too slow. For this example, let's say that the resolution for each axis is chosen to be 100. Once the shape has been evolved/optimized, we can always smooth it out through averaging methods, before creating the metal repli- ca. In summary, we're using a 3d substrate, every point [X,Y,Z] in the substrate specifies a voxel with sides of length 1cm, and we're using the NN to evolve a 3d form by painting with 1 cubic cm voxels.

Having now agreed on the substrate's properties, we need to decide on the in- terpretation of the NN's output. Let's assume that we only wish to evolve the shape of the gas tank, rather than have the NN to also play around with different materials from which this gas tank can be shaped. Then, our NNs for this problem should output a vector of length 1: [E], where E specifies whether the voxel is filled in or left empty. For every 3d coordinate of a voxel in the substrate sent to the NN, the network outputs an E, where if E >= 0 then the voxel is filled in, and if E < 0, then the voxel is empty.

Finally, we now need to come up with a fitness function by which we would gage how good the evolved gas tank shapes are. Because the physical shape of the gas tank needs to be made up to some specification, our fitness function should take various physical constraints into account. If for example we need for the gas tank to hold at least 1 litre, be no larger than 0.5 meters on each side, and also that it must be aerodynamic (perhaps this is a miniature plane wing that also holds fuel), then these constraints need to be somehow represented in the fitness func- tion, for example as follows:

Fitness = A(% fuel hold constraint) – B(% overstep on dimensional constraint) + C(aerodynamic properties).

The more fuel the evolved shape can hold, while not overstepping the length, width, and height, and being as aerodynamic as possible, the more fit is the NN. By setting A, B, and C, the researcher weighs the importance of each constraint of the design.

Having now set everything, we can begin the neuroevolutionary process like we did in other examples. We start a population of minimalistic NNs, and analyze the shapes they generate in the substrate. Each NN's crafted 3d shape is scored based on how well it fulfills the constraints, the NN's are then sorted based on their fitness, and the best of the population are allowed to create offspring. Once the new population composed of the offspring and their fit parents is created, the process repeats. Neuroevolution continues until a stopping condition is reached (a NN with a fitness level we find high enough is reached, or after the neuroevolutionary process has ran for a long enough time). Once the shape is gen- erated, we can create a physical version of it, perhaps using a 3d printer.

Example-2: Two and three dimensional shape exploration.

It is also possible to “optimize ” and explore multidimensional shapes with a re- searcher being part of the neuroevolutionary process. In this scenario a population of NNs describe two or three dimensional shapes, and we use the following steps:

1. The shapes are presented to a researcher.
2. The researcher decides which of the shapes he finds most interesting.
3. The NN which generated that shape is then chosen to create mutant offspring, which are the mutated versions of the interesting shapes, with variations of those shapes and images (if 2d).
4. Then the shape produced by the chosen NN and its offspring are again present- ed to the researcher...

In a sense, this is selective breeding of art and structure, and an approach based on Richard Dawkin's Biomorphs [31]. There are a number of such implementa- tions available to play around with online. As mentioned, an example of a neuroevolutionary system that works with 2d substrates, exploring 2d images, is the Pic Breeder [29]. Another one that explores 3d structures is Endless Forms [30].

1.2.10 Computational Intelligence & Towards Singularity

Perhaps you have picked up this book because you are as immensely interested in computational intelligence as I am. Because you wish to contribute to the field, advance it further, and get us that much closer to what some call, the technological singularity. There is no reason why human or greater than human intelligence cannot be reached through neuroevolution. After all, it has been done before, and we are the proof of that, we are the product of that. Our very own brains, carbon based neurocomputers, have been evolved over billions of years. We already know the end goal, the approach, the basic building blocks that we should use, so perhaps we could do the same thing faster in silicone, or some other substrate. To that end, the field of Neural Networks, and a related and more general field of Universal Learning Networks (to some degree, this book is in a sense the presenta- tion of Topology and Parameter Evolving Universal Learning Networks, rather than simply Neural Networks, since our nodes will be much more general than neurons) can take you towards that goal. And I'm hoping that this book will help you on your way of creating such a system, a system capable of evolving some- thing of that level of complexity, and intelligence.

1.3 A Whirlwind Overview

This book covers the theory and methodology behind a neuroevolutionary sys- tem. The aim of the book is to present new algorithms, new concepts, and to pro- vide a detailed tutorial on how to develop a state of the art Topology and Weight Evolving Artificial Neural Networks (TWEANN) platform using Erlang. This text will guide you step by step, from simulating a single neuron, to building up a complete and fully general evolutionary platform able to evolve neural network systems for any application. Source code for everything covered in this book will be provided and explained within the text, and also be available online as supple- mentary material [33].

Chapter-1 covers the goals this book seeks to achieve, and the various motiva- tions for the creation and application of neuroevolutionary systems. In Chapter-2 we begin exploring the morphological and information processing properties of a single biological neuron, followed by a brief introduction to the properties of bio- logical neural networks. We then extrapolate the important parts of neural net- works, and see how an artificial neuron can mirror these signal processing fea- tures. In Chapters 3 and 4 we discuss evolution, how it optimizes organisms over time, and how the evolutionary process can be used to optimize and evolve neural networks. With these basics covered, in Chapter-5 I will make my claims with re- gards to Erlang, and why I think that it is perfect for computational intelligence re- search and development, and why I consider it the quintessential neural network programming language.

In Chapter-6 we will take our first step in developing a concurrent neural net- work based system. We will implement a single artificial neuron, represented by a process. Then we will combine multiple such neurons into a simple feed forward neural network (NN), with each neuron an independent process and thus the whole neural network being fully concurrent. Having implemented a simple static feed forward NN, we will develop a genotype encoding for the representation of our neural network, and a mapping function from this genotype to its phenotype, which is the actual neural system.

In Chapter-7 we will implement an augmented version of the stochastic hill- climbing (SHC) optimization algorithm, and add it to the simple NN system we have created. Having created a proper optimization algorithm and a decoupled method of applying our NN system to problems through something we call a Scape, we will conclude the chapter with us benchmarking the developed optimizable NN system on the XOR emulation problem.

In Chapter-8 we take our first step towards neuroevolution. Having developed a NN system capable of having its synaptic weights optimized, we will combine it with an evolutionary algorithm. We will create a population_monitor, a process that spawns a population of NN systems, monitors their performance, applies a se- lection algorithm to the NNs in the population, and generates the mutant offspring from the fit NNs, while removing the unfit. We add topological mutation operators to our neuroevolutionary system, which will allow the population_monitor to evolve the NNs by adding new neural elements to their topologies. With these fea- tures added to our neuroevolutionary system, the chapter concludes with us now having developed a simple yet fully distributed and powerful Topology and Weight Evolving Artificial Neural Network (TWEANN) platform.

In Chapter-9 we test the various mutation operators, observing the types of to- pologies the operator produces when applied to a simple default seed NN. This chapter concentrates on debugging, testing, and analyzing our neuroevolutionary system. Because we have implemented quiet a number of mutation operators, we test how they work, and debug the problem hiding within.

Before moving forward with further expanding and improving our TWEANN platform, we take Chapter-10 to discuss a TWEANN case study. In this chapter I present a case study of a memetic algorithm based TWEANN system called DXNN which I developed through Erlang. In this chapter we discuss the various details and implementation choices made while building it. We also discuss the various features that it has, and which we will need to add to the system we're building in this book, which itself has a much cleaner and decoupled implementa- tion, and which by the time we're done will supersede DXNN. After exploring the ideas contained in the DXNN case study, we continue with advancing our own platform in the following chapters.

In Chapter-11 we modify the implementation of our TWEANN system, making all its parts decoupled from one another. By doing so, the plasticity functions, the activation functions, the evolutionary loops, the mutation operators... become in- dependent, called and referenced through their own modules and function names, and thus allowing for our system to be crowdsourced, letting anyone else modify and add new activation functions, mutation operators, and other features, without having to modify or augment any other part of the TWEANN system. This effectively makes our system more scalable, and easier to augment and improve in the future.

In Chapter-12 we extend the population_monitor process to keep track of the evolved population, building up a trace of its performance, and keeping track of the various evolutionary parameters of the evolving species by calculating perfor- mance statistics every X number of evaluations, where X is set by the researcher.

In Chapter-13 we add the benchmarker process which can sequentially spawn population_monitors and apply them to some specified problem. We also extend the database to include the experiment record, which the benchmarker uses to de- posit the traces of the population's evolutionary statistics, and to recover from crashes to continue with the specified experiment. The benchmarker can compose experi- ments by performing multiple evolutionary runs, and then produce statistical data and gnuplot ready files of the various statistics calculated from the experiment.

In Chapter-14 we create two new benchmarking problems. To be able to test a neuroevolutionary system after having made some modification requires problems more complex than the simple XOR mimicking problem. Thus in this chapter we create the pole balancing benchmark (single and double pole, with and without damping), and the T-Maze benchmarking problem.

In Chapter-15 we add plasticity to our direct encoded NN system. We imple- ment numerous plasticity encoding approaches, and develop numerous plasticity learning rules, amongst which are variations of the Hebbian Learning Rule, Oja's Rule, and Neural Modulation.

In Chapter-16 we add substrate encoding to our neuroevolutionary platform. Substrate encoding, popularized by HyperNEAT, is a powerful encoding method which uses the NN to paint the synaptic weights and connectivity patterns on a multidimensional substrate with embedded neurodes within. A Substrate Encoded NN system (SENN) offers superior geometrical regularity exploitation abilities to the NN based agent, when such geometrical regularity is present within the problem domain.

In Chapter-17 we add substrate based plasticity. We implement the ABC and the Iterative plasticity rules.

At this point we will have one of the most advanced, fully distributed, and an incredibly general TWEANN platforms to date (as you will see, this is not an overstatement). Thus we begin the applications part of the book, and apply our developed systems to two very different and interesting areas: Artificial Life, and autonomous currency trading.

In Chapter-18 we develop Flatland , a 2d artificial life simulator. We create new morphological specifications, sensors, and actuators, which can then be used by our NN based system to spawn an avatar within the flatland simulated 2d world. Through the avatar the NN will be able to interact with the environment, and other avatars inhabiting it. Afterwards, we run a number of experiments ap- plying our system to the ALife simulation, and then plot the results produced by the benchmarker process for each such experiment.

In Chapter-19 we create a Forex simulator, a private scape with which our neuroevolutionary system can interface, and evolve to autonomously trade curren- cy pairs. We perform numerous experiments, and develop a system that analyzes not merely the sliding window based input signals, but the actual graphical charts, and through substrate encoding is able to extract the geometrical patterns within those charts.

With the entire platform now developed, Chapter-20 will conclude this book with a concluding discussion on neuroevolution, the new and improved version of the DXNN platform that we've developed here, the role such systems will play in evolving general computational intelligence based systems in the future, and the movement towards singularity.

Make no mistake, there is absolutely nothing mystical about the human brain, it is nothing more than a carbon based neurocomputational machine, a vast directed graph structure composed of biological signal processing elements we call neurons. A vast parallel computing system carved out in flesh by billions of years of evolution. The goal of creating a non-biological substrate based Computational In- telligence(CI) system of similar and greater potential is not a matter of if, but of when. The projects like “The Blue Brain Project ” [32] in which large cortical col- umns are simulated on a super computer, demonstrate that the silicone based sys- tems perform just like their biological counterparts. We already know that it is possible for machines to think, you and I are the very proof of that, our brains are organic computers, chemical based computing machines and nothing more. It does not matter what performs the computation, a wet slimy cell, or an immaculate sili- cone based processing unit... as long as both elements can accept the same input and produce the same response, they will generate the same minds. It makes no difference whether this vast Neural Network system called the brain is carved on a biological substrate, or etched in a non-biological one. And unlike the supersti- tious and backwards humans, the universe itself simply does not care whether the computations are conducted in flesh, or in machine, as long as they are the same computations...

But before all these grand goals are realized though, we still need to create the tools to build systems with such potential, we still need to build the substrate, the hardware fast enough to support such a CI system, and finally we need a pro- gramming language capable of representing such dynamic, fault tolerant, and fully distributed Neural Networks and the ideas behind them. The necessary hardware is improving at a steady pace, moving in the right direction of ever increasing num- ber of cores and per-core computational power with every year, and so it is only the programming language which could offer the scalability, extendibility, and ro- bustness to the neurocomputational system that is still lacking. I believe that I found this neural network programming language in Erlang, and I will demon- strate that fact in this book.

1.5 References

[1] Bedau M (2003) Artificial Life: Organization, Adaptation and Complexity From the Bottom Up. Trends in Cognitive Sciences 7, 505-512.

[2] Edition S (2005) Artificial Life Models in Software A. Adamatzky and M. Komosinski, eds. (Springer).

[3] Johnston J (2008) The Allure of Machinic Life: Cybernetics, Artificial Life, and The New AI. (MIT Press).

[4] Gauci J, Stanley K (2007) Generating Large-Scale Neural Networks Through Discovering Geometric Regularities. Proceedings of the 9th annual conference on Genetic and evolution- ary computation GECCO 07, 997.

[5] Siebel NT, Sommer G (2007) Evolutionary Reinforcement Learning of Artificial Neural Networks. International Journal of Hybrid Intelligent Systems 4, 171-183.

[6] Gomez F, Schmidhuber J, Miikkulainen R (2008) Accelerated Neural Evolution through Co- operatively Coevolved Synapses. Journal of Machine Learning Research 9, 937-965.

[7] Back T, Schwefel HP (1993) An Overview of Evolutionary Algorithms for Parameter Optimization. Evolutionary Computation 1, 1-23.

[8] Fonseca CM, Fleming PJ (1995) An Overview of Evolutionary Algorithms in Multiobjective Optimization. Evolutionary Computation 3, 1-16.

[9] Alfredo AM, Carlos AC, Efren MM (2011) Evolutionary Algorithms Applied to Multi- Objective Aerodynamic Shape Optimization. Studies in Computational Intelligence.

[10] Alon K (2004) Analyzing Evolved Fault-Tolerant Neurocontrollers. In Proceedings of the Ninth International Conference on the Simulation and Synthesis of Living Systems. (ALIFE9).

[11] Floreano D, Mondada F (1998) Evolutionary Neurocontrollers For Autonomous Mobile Robots. Neural Networks 11, 1461-1478.

[12] Engel Y, Szabo P, Volkinshtein D (2006) Learning to Control an Octopus Arm with Gauss- ian Process Temporal Difference Methods. Advances in Neural Information Processing Sys- tems 18 c, 347-354.

[13] Kaelbling LP, Littman ML, Moore AW (1996) Reinforcement Learning: A Survey. Journal of Artificial Intelligence Research 4, 237-285.

[14] Braun H, Weisbrod J (1993) Evolving Feedforward Neural Networks. In Proceedings of ANNGA93, International Conference on Artificial Neural Networks and Genetic Algorithms. Inns-bruck: Springer-Verlag

[15] Floreano D, Urzelai J (2000) Evolutionary Robots With On-Line Self-Organization and Be- havioral Fitness. Neural Networks 13, 431-443.

[16] Boden MA (1994) Dimensions of creativity M. A. Boden, ed. (MIT Press).

[17] Bringsjord S, Ferrucci DA (2000) Artificial Intelligence and Literary Creativity: Inside the Mind of BRUTUS, a Storytelling Machine. Computational Linguistics 26, 642-647.

[18] Bentley, P., and Corne, D. (2002). Creative Evolutionary Systems P. Bentley and D. Corne, eds. (Morgan Kaufmann Pub).

[19] Khepera robot: http://www.k-team.com/

[20] The Player Project: http://playerstage.sourceforge.net/

[21] Gazebo, a modern open source 3d robot simulator: http://gazebosim.org/

[22] Sher GI (2010) DXNN Platform: The Shedding of Biological Inefficiencies. Neuron, 1-36. Available at: http://arxiv.org/abs/1011.6022.

[23] Hastings EJ, Guha RK, Stanley KO (2009) Automatic Content Generation in the Galactic Arms Race Video Game. IEEE Transactions on Computational Intelligence and AI in Games 1, 1-19.

[24] Penetration Testing Software, Metasploit: http://www.metasploit.com/

[25] A discrete-event network simulator for Internet systems, ns-3: http://www.nsnam.org/

[26] DETER Network Security Testbed: http://isi.deterlab.net/

[27] OpenSPARC, an open source 64bit CMT Microprocessor: http://www.opensparc.net/

[28] Gauci J, Stanley K (2007) Generating Large-Scale Neural Networks Through Discovering Geometric Regularities. Proceedings of the 9th annual conference on Genetic and evolution- ary computation GECCO 07, 997.

[29] Picbreeder, a collaborative evolutionary art project: http://picbreeder.org/

[30] Collaborative art, evolving 3d shapes: http://endlessforms.com/

[31] Dawkins R (1986) The Blind Watchmaker. (Norton), ISBN 0393315703.

[32] The Blue Brain Project: http://bluebrain.epfl.ch/

[33] All source code developed in this book is also available at: https://github.com/ CorticalComputer/Book_NeuroevolutionThroughErlang

[34] Vintan LN, Iridon M (2002) Towards a High Performance Neural Branch Predictor. In IJCNN99 International Joint Conference on Neural Networks Proceedings (IEEE Service Center), p. 868-873.

Part I - FOUNDATIONS

In this first part we will cover the necessary foundations for this text. We will first discuss what neural networks are and how they function, both the biological and the artificial kind. Afterwards we will briefly cover evolutionary computation, its history, and how the various flavors (genetic algorithms, genetic programming, evolutionary programming, evolutionary strategies) are related to each other. Hav- ing now covered the two main parts separately, neural networks and evolution, we will delve into how the combination of the two works, and thus start our discus- sion on Neuroevolution. We will talk about a few different approaches to neuroevolution, and the accomplishments such systems have made thus far. We will note how related they are to genetic programming, and how indeed neuroevolutionary systems can be simply considered as a variation on genetic programming systems. Finally, we will discuss why Erlang is such an important programming language for this fiel d. What benefit we will gain by using it, and why I have chosen it as the langua ge of choice for this text, and this research in general.

Chapter 2 Introduction to Neural Networks

Abstract In this chapter we discuss how the biological neurons process infor- mation, the difference between the spatiotemporal processing of frequency encod- ed information conducted by a biological neuron and the amplitude and frequency encoded signals processed by the artificial neural networks. We discuss the vari- ous types of artificial neural networks that exist, their architectures and topologies, and how to allow such neural networks to possess plasticity, which allows the neu- rons to adapt and change as they process presynaptic signals.

Our brains are biological parallel computers, composed of roughly 100,000,000,000 (one hundred billion) signal processing elements called Neurons. Like a vast graph, these neurons are connected with each other in complex topo- logical patterns. Each neuron in this vast processing graph accepts signals from thousands of other neurons, processes those signals, and then outputs a frequency encoded signal and passes it onwards to thousands of other neurons. Though each neuron on its own is relatively easy to understand, when you connect together a few billion of them, it becomes incredibly difficult to predict the outcome given some specific input. If you are careful and connect these biological signal pro- cessing elements in some particular pattern, the final output of this vast graph might even be something useful, an intelligent system for example. An output sig- nal can for example control muscle tissue in your legs, so that they move in syn- chrony and give you the ability to walk and run. Or this vast neural network's out- put can be a solution to some problem which was fed into it as an image from its sensory organs, like cameras or eyes for example. We don't yet completely know how and which neurons, and in what patterns we need to connect them to allow us to produce useful results, but we're getting there, we're reverse engineering the brain [1].

Evolution used billions of years to try out trillions upon trillions of various permutations of chemical setups for each neuron and connections between them... we and other inhabitants of this planet are the result of this vast stochastic optimi- zation, an optimization for a more fit replicator (a gene). We are, as Richard Daw- kins noted, that replicator's tools of survival, we are its survival machines [2,3].

In biological organisms born of evolution, there was only one goal, to create a copy (usually mutated due to environmental factors), to create an offspring. Bil- lions and billions of permutations of atoms and simple molecules and environ- ments on this planet eventually resulted in a molecule which was able to copy it- self if there was enough of the right material around it to do so. Of course as soon as such a molecule appears in the environment, it quickly consumes all the raw material its able to use to create copies of itself... but due to radiation and the sim- ple fact that biology is not perfect, there are variations of this molecule. Some of these mutant clones of the molecule were smaller and unable to replicate, others were able to do so more efficiently when using raw materials, yet others were even able to break apart surrounding compounds to make the missing necessary raw materials... though it's still just chemistry at this point, in essence this is al- ready competition and predation. The replicating molecules are competing against each other, not by choice, but simply because that's what naturally happens when something can make copies of itself. Anything that does not make a copy, does not take over the environment, and is either expunged from the environment, or used as raw material by replicators.

The molecules split and vary/mutate, new features are added, so that for exam- ple some new molecule is able to break apart another molecule, or merge with it. If some complex molecule does not replicate in some manner or another, it has no future... because it will not create an offspring molecule to carry its behavior for- ward in time.

These mutations, variations, collisions between molecules and atoms, all giving a chance for a more fit replicator to emerge, this was occurring on the entire sur- face of the planet, and below it. The entire planet was like a computational system, where every inch of the surface gave space for the calculations of the mutations and permutations of molecules to take place... And after billions of years, trillions upon trillions of these replications and mutations, more and more fit systems emerged. Sure, most of the mutations were harmful and produced mostly unfit off- spring that could not replicate at all, or were able to replicate but at a slower pace or lower efficiency level... But when you have trillions of opportunities for im- provement to work with... no matter how small the probability, eventually, every once in a while... a better combination of molecules results in a better replicator, able to take advantage of some niche within the environment... That is evolution.

Through these trillions of permutations, offspring and molecules combined into better replicators, some of which could defend themselves against other replica- tors, some of which could attack other kinds of replicators so that they could cre- ate more of their own kind... To know whom to attack, to know who is composed of the resources that you need to create an offspring, you need a system that is able to tell the difference between the different kinds “stuff ” out there, you need some kind of sensory setup... These adaptations continued on and on, and the competition still rages on to this day, from molecules to groups of molecules, cells, the “Survival Machines ”, tools evolved by the replicators to defend them- selves, tools growing more and more complex to deal with other rival replicators and their Survival Machines... a vast biological arms race.

Eventually, through evolution, a new information storage methods was discov- ered, RNA evolved[9]... the result of all this turmoil is what we see around us to- day. We are still banding together, we are still competing for limited resources, we are the “Survival Machines ” as Dawkins pointed out, machines used by these rep- licators, by genes, to wage war on each other and make as many copies of them- selves as possible. Their newest invention, a feature that evolved to deal with the ever changing and dangerous world, is an interconnected graph of cells that can control these Survival Machines more precisely, deal with much more complex Survival Machines, store information about the world, and keep the genes safe long enough to create more copies of them, with their own survival machine to control. One of the most significant features that arisen in biological organisms, is the parallel biological computer, the vast neural network system, the brain. Over the billions of years of evolution the brain too has been changed, evolution has trended toward more complex brains. Evolution has been slowly exploring the various neural network topologies.

This text is dedicated to the study of evolutionary methods as applied to simu- lated neural networks. Instead of using atoms and molecules as the building blocks for our evolutionary algorithms, we will use neurons. These neurons, when grouped in particular patterns and topologies, form brains. Biological computers evolved the ability to invent, imagine, scheme, and most importantly, these paral- lel computers evolved self awareness. Thus we know that such things are possible to evolve, it already happened, nature has proven it possible, we are the proof. In this book we will develop non biological neural networks, and we will apply evo- lutionary principles to evolve neural systems capable of solving complex prob- lems, adapting to artificial environments, and build a platform that perhaps, some day, could too evolve self aware NN based agents.

In the following section I will discuss in more detail the Biological Neural Networks, how they work, how each neuron processes data, how the neuron en- codes data, and how it connects to other neurons in the vast neural network we call our brain.

Our brain is a vast graph of interconnected neurons, a vast biological neural network. A neuron is just a cell that can accept signals, and based on its chemical and geometrical properties, produce an output. There are roughly 100 billion neu- rons in the human brain, with trillions of connections between them. Though it might seem surprising that they can work so coherently, the result of which is us, our consciousness and intelligence, it is not surprising at all when we take into ac- count that it took evolution billions of years and trillions of permutations to fine tune this system to get the result that we see today.

A typical neuron, as shown in Fig-2.1 , is a cell composed of three main parts, the soma (cell body), the dendrites, and the axon. The soma is a compact body containing the nucleus, and other standard cell internals, and the dendrites and ax- on are filaments that extrude from it. A single neuron usually has a large number of dendrites, all of which branch profusely but usually retain their filament thick- ness. Unlike the case with the dendrites, a neuron has only a single axon, originating from a base of the neuron called the “axon hillock ”. The axon is usually a long fil- ament which can branch and thus connect to multiple other neurons, with the ax- onal filament itself usually getting thinner the further it extends and the more it branches. “ Synaptic signals from other neurons are received by the soma and dendrites; signals to other neurons are transmitted by the axon. A typical synapse, then, is a contact between the axon of one neuron and a dendrite or soma of an- other. Synaptic signals may be excitatory or inhibitory. If the net excitation re- ceived by a neuron over a short period of time is large enough, the neuron gener- ates a brief pulse called an action potential, which originates at the soma and propagates rapidly along the axon, activating synapses onto other neurons as it goes. ” [22].

Fig. 2.1 A typical biological neuron.

It would be difficult to describe the biological neuron and its operation any more clearly than is done in the following quote [22] from the ever growing com- pendium of human knowledge, Wikipedia: “ Neurons are highly specialized for the processing and transmission of cellular signals. Given the diversity of functions performed by neurons in different parts of the nervous system, there is, as ex- pected, a wide variety in the shape, size, and electrochemical properties of neu- rons. For instance, the soma of a neuron can vary from 4 to 100 micrometers in diameter.

The soma is the central part of the neuron. It contains the nucleus of the cell, and therefore is where most protein synthesis occurs. The nucleus ranges from 3 to 18 micrometers in diameter.

The dendrites of a neuron are cellular extensions with many branches, and metaphorically this overall shape and structure is referred to as a dendritic tree. This is where the majority of input to the neuron occurs.

The axon is a finer, cable-like projection that can extend tens, hundreds, or even tens of thousands of times the diameter of the soma in length. The axon carries nerve signals away from the soma (and also carries some types of in- formation back to it). Many neurons have only one axon, but this axon may — and usually will —undergo extensive branching, enabling communication with many target cells. The part of the axon where it emerges from the soma is called the axon hillock. Besides being an anatomical structure, the axon hillock is also the part of the neuron that has the greatest density of voltage-dependent sodium channels. This makes it the most easily-excited part of the neuron and the spike initiation zone for the axon: in electrophysiological terms it has the most negative action potential threshold. While the axon and axon hillock are generally involved in information outflow, this region can also receive input from other neurons.

The axon terminal contains synapses, specialized structures where neuro- transmitter chemicals are released to communicate with target neurons. ”

The neuron to neuron signaling is a three step electrochemical process, as shown in Fig-2.2 . First an ion based electrical signal is propagated down the axon, and towards every branch of that axon down to the axonal terminals. At the synaptic cleft of those axonal terminals, where the axon is in very close proximity to the cell bodies and dendrites of other neurons, the electrical signal is converted into a chemical one. The neurotransmitters, chemical signals, pass the distance between the axon terminal of the presynaptic neuron, and the dendrite (or soma, and some- times even axons) of the post-synaptic neuron. How excited the post-synaptic neu- ron gets, the strength of the signal that the dendrites perceive from these neuro- transmitters, all depend on the number of receptors that are present on the surface where the neurotransmitters contact the postsynaptic neuron. Thus, it is the num- ber of, and type of receptors found on the soma and dendrites that weigh the in- coming chemical signal, and decide whether it is excitatory when combined with other signals, or inhibitory. The receptors convert the chemical signals they per- ceive, back into electrical impulses. This train of signals continues its journey down the dendrites and towards the soma. Thus, as we can see, the complete signal is an electrical one, converted into a chemical one, and then converted back into an electrical one.

Fig. 2.2 Neuron to neuron signaling, a three step electrochemical process.

Furthermore, the way the signals are perceived is not based on a single spike, a single electrical impulse that some neuron A sends to neuron B, but the signal's frequency. The message is encoded not in the amplitude, but in the frequency. Evolutionary this makes perfect sense, in biological systems it would be difficult to regulate a perfect amplitude as it passes down the wires, but frequency is much simpler to manage using the imperfect biological wetware.

A neuron B could have hundreds to thousands of axons connecting to its soma and dendrites. The way a neuron calculates whether it should produce an output signal, also called action potential or simply spike, at any given time, depends on the intensity of the electrical signal at the axon hillock at that time, as shown in Fig-2.3 . Since the intensity of the signal experienced by the axon hillock (trigger zone) depends on how many spikes at that moment excite that region at the same time, the signal is based not only on how many spikes there are, but also on the shape of the neuron and the timing of the signals. The neuron performs a spatio- temporal integration of the incoming signals. If the excitation level at a given time surpasses its threshold, an action potential is generated and passed down the axon.

Furthermore, the output signal's amplitude is independent of signals arriving at the axon hillock, it is an all-or-none type of system. The neuron either produces an action potential (if there is enough excitation at the trigger zone), or it does not. Ra- ther than encoding the message in the action potential's amplitude, it is encoded in the frequency, and the frequency depends on the spatiotemporal signal integration and processing that occur within the soma and at the axon hillock.

The signal is based on the spatial properties of the incoming spikes, because if the axon hillock is located in a strange position, or its properties are distributed in space within the neuron differently, it will perceive the incoming signals in a dif- ferent way. For example, thinking purely mathematically, if the trigger zone is somehow spread thinly over a great area, then to trigger it we would need to send electrical signals that intersect on this wide area, the distribution of the incoming action potentials would have to cover this wide area, all the different places of the axon hillock that sense the electrical signals. On the other hand, if the axon hillock is concentrated at a single point, then to produce the same output we would need to send just a few of the signals towards that point.

On the other hand, the neuron's signal processing is temporal based processing because, if for example 10 spikes come across the axon hillock, each at a rate of 1ms after the other, the axon hillock feels an excitation of only 1 spike every 1ms, which might not be enough excitation beyond the threshold to trigger an output action potential. On the other hand, if 10 spikes come from different sides, and all come across the axon hillock at the same time, the intensity now is 10 spikes ra- ther than one, during the same single ms, which will overcome the biological threshold and the neuron will send an action potential down the axon.

Thus, the output signal, an electrical spike encoded signal produced by the neu- ron, is based on the spatial and temporal properties of its input signals. Something similar is shown in Fig-2.3 , where I loosely defined the timings of when the spikes will arrive at the trigger zone using t which defines the arrival at the trigger zone, t-1 which defines arrival at the trigger zone in 1 delta, t-2 which defines the arrival at the trigger zone in 2 deltas, and so on. At t-1 we see that there will be 4 spikes, at t-2 only 2. If it requires 3 spikes to overcome the threshold (which itself is defined by the shape and chemical properties at the axon hillock) and to set off an action potential down the axon, then the signals arriving at t-2 , when they do finally arrive at the hillock in 2 deltas (time units), will not trigger an action poten- tial, while the signals currently at t-1 will generate a spike when they finally arrive at the trigger zone.

Fig. 2.3 Spatiotemporal signal integration.

Furthermore, the neurons don't just accept incoming signals and produce out- going signals, the neurons also change over time based on the signals they pro- cess. This change in the way neurons respond to signals by adding more receptors to the dendrites, or subtracting receptors from the dendrites, or modifying the way their receptors work, is one of the processes by which a neural network learns and changes its excitability towards certain signals, it is how we accumulate experi- ence and form memories. Other ways by which a neural network learns is through the axons branching and making new connections, or breaking old connections. And finally the NN changes in the way it processes signals through having the very fluid in which the neurons are bathed changed and chemically modified, through drugs or other means for example.

The most important part to take away from this chapter is that the biological neurons output frequency encoded signals, and that they process the incoming fre- quency encoded signals through spatiotemporal integration of those signals. And

2.2 Artificial Neural Network

that the neurons can change over time based on the signals they process, the neu- rons change biologically, they change their information processing strategies, and they can form new connections to other neurons, and break old ones. This process is called neuronal plasticity , or just plasticity. In the next section we will discuss artificial neural networks, how they function, and how they can differ from their biological counterparts.

Artificial neural networks (NN), as shown in Fig-2.4 , are simulated biological neural networks to different levels of precision. In this section we will cover the typical artificial neural network, which are not perfect simulations. A typical arti- ficial neuron, aka neurode, does not simulate a biological neuron at the atomic, or even molecular level. Artificial neurons are abstractions of biological neurons, they represent the essentials of biological neurons, their nonlinear signal integra- tion, plasticity, and concurrency.

Fig. 2.4 An artificial neural network.

As shown in Fig-2.5 , like a biological neuron, an artificial one accepts signals through its artificial dendrites, processes those signals in its artificial soma, and outputs the processed signals to other neurons it is connected with. It is a concise representation of what a biological neuron does. A biological neuron simply ac- cepts signals, weighs each signal, where the weight depends on the receptors on the dendrites on which the axons from other neurons intercepted, then based on its internal structure and chemical composition, produces the final frequency encoded output and passes that output onwards to other neurons. In the same way, an artifi- cial neuron accepts signals, weighs each signal using its weight parameters, integrates all the weighted signals through its activation function which simulates the biological neuron's spatiotemporal processing at the axon hillock, and then propa- gates the final output signal to other neurons it is connected to.

Fig. 2.5 A detailed look at an artificial neuron's schematic.

As can be seen from Fig-2.5 , there are of course differences. We abstract the functionality undertaken by the receptors on the dendrites with simple weights, nevertheless, each incoming signal is weighted, and depending on whether the weight is positive or negative, each incoming signal can act as an excitatory or in- hibitory one, respectively. We abstract spatiotemporal signal integration that oc- curs at the axon hillock with an activation function (which can be anything, and as complex as the researcher desires), nevertheless, the weighted signals are integrat- ed at the output point of the artificial neuron to produce the final output vector, which is then passed onwards to other neurons. And finally, we abstract the func- tionality undertaken by the axon with simple signal message passing, nevertheless, the final output signal is propagated, diligently, to all postsynaptic artificial neu- rons.

The biological neural network is a vast graph of parallel processing simple bio- logical signal integrators, and the artificial neural network too is a vast graph of parallel processing simple signal integrators. The neurons in a biological neural network can adapt, and change its functionality over time, which too can be done in artificial neural network through simulated neural plasticity, as we will discuss in later sections, and eventually implement in the NN systems we will build ourselves.

There is one thing though that differs significantly in the typical artificial neu- ral networks, and the biological neural networks. The neurons in a biological NN frequency encode their signals, whereas in the artificial NNs, the neurons ampli- tude encode their signals. What has more flexibility? Frequency encoded NN sys- tems or the amplitude encoded ones? It is difficult to say, but we do know that both, biological and artificial neural networks are Turing complete [4], which means that both possess the same amount of flexibility. The implications of the fact that both systems are universal Turing machines is that even if a single artifi- cial neuron does not do as much, or perform as a complex computation as a single biological neuron, we could put a few artificial neurons together into an artificial neural circuit, and this artificial neural circuit will have the same processing power and flexibility as a biological neuron. On the other hand, note that frequency en- coding signals takes more time, because it will at least take the amount of time be- tween multiple spikes in the spike train of the signal for the message to be for- warded (since it is the frequency, the time between the spikes that is important), whereas in an amplitude encoded message, the single spike, its amplitude, carries all the information needed.

How much of the biology and chemistry of the biological neuron is actually needed? After all, the biological neuron is the way it is now due to the fact that it was the first randomly found solution , the easiest solution found by evolution. Wetware has no choice but to use ions instead of electrons for electrical signal propagation. Wetware has no choice but to use frequency encoding, instead of amplitude encoding, because wetware is so much more unreliable than hardware (but the biological neural network as a whole, due to a high level of interconnec- tions, is highly fault tolerant, reliable, and precise). The human neuron is not a perfect processing element, it is simply the processing element that was found through evolution, by chance, the easiest one to evolve over time, that's all. Thus, perhaps a typical plasticity incorporating artificial neuron has all the right features already. We have after all evolved ALife organisms with just a few dozen neurons that exhibited interesting and evolutionary appropriate behaviors with regards to food foraging and hunting [5,6,7]. We do know one thing though, the limits of speed, signal propagation, neural plasticity, life span of the neuron, integration of new neural systems over the organism's lifetime, are all limited in wetware by bi- ology. None of these limitations are present in hardware , the only speed limit of signal propagation is that of light in a hardware based neural computing system. The non biological neural computer can add new neural circuits to itself over life- time, and that lifetime span is unlimited, given that hardware upkeep is possible.

I think that amplitude encoded signaling is just as powerful, and the activation functions of the artificial neurons, the integration of the weighted signals, is also as flexible, or can be as flexible as the spatiotemporal signal integration performed by a biological neuron. An artificial neuron can simulate different kinds of recep- tor densities on the dendrites by different values for weights. An artificial neuron can simulate different kinds of neuron types through the use of different kinds of activation functions. Even plasticity is easy to add to an artificial neuron. And of course, there are also artificial spiking neural network systems [23,24,25], which use frequency encoding like a biological neural network does. There is absolutely no reason why artificial neural networks cannot achieve the same level of perfor- mance, robustness, and intelligence, as biological neural networks have.

2.2.1 The Neurode in Detail

In this section we will do a more detailed analysis of the architecture of an artificial neuron, how it processes an incoming signal, and how such an artificial neu- ron could be represented in software. In Fig-2.6 we use the schematic of an artifi- cial neuron in a simple example where the neuron receives two incoming signals. Each of the signals is a vector. The third signal is not from any other neuron, but is simply a bias value, which modifies the neuron's processing. The neuron pro- cesses the signal based on its internals, and then forwards its output, in a vector form, to postsynaptic neurons. In the figure, the “axon ” of the neuron branches in- to 3 strands.

Fig. 2.6 An artificial neuron in action, receiving signals from two other elements, a and b.

Artificial neurons accept vector input signals, and output a vector signal of length 1. Each input signal is weighted; each element in the input vector is multi- plied by a weight in a weight list associated with that input vector, and that partic- ular element in the input vector. Thus, the integration of the incoming signals is done by calculating a dot product of the incoming vectors and the weight vectors associated with those input vectors. In the above figure, there are two incoming signals from other elements, and a bias signal (which we'll talk about next). The incoming signal from element ‘a' is a vector signal of length 2, the signal from el- ement ‘b', is a vector of length 3, and the bias signal is a vector of length 1. The neuron has a weight list for each incoming signal. The weight lists weigh the im- portance of each input vector. The way we integrate the input signal is by calculat- ing a dot product of the weights and the input signals. Once the dot product is cal- culated, we compute the output of the neuron, Output = F(X), where F is the activation function, and X = Dot_Product + Bias. The neuron then packages this result into a vector of length 1, like so: [Output], and then fans out this output vec- tor to the elements that it is connected to. A sigmoid function, or hyperbolic tangent, is the typically used activation function in artificial neurons. A multi-layered feed forward neural circuit composed of neurons using sigmoid activation functions can act as a universal function approximator [8], which means that a neural net- work composed of such neural circuits can do anything.

Now regarding the bias input, it is simply an input vector which is used to in- crease the flexibility of the neuron by giving it an extra weight that it can use to skew the dot product of the input signals. Not every neuron needs to have a bias input, it's optional, and if the weight for the bias input is 0, then that is equivalent to a neuron that does not have a bias input at all. The neuron can use the bias to modify the point at which the weighted dot product produces a positive output when passed through the activation function, in which case the bias acts as a threshold. If the bias is a large positive number, then no matter what the input will be, the neuron has a much greater chance of outputting a positive value. If the bias is a negative number, then the incoming signals will have to be high enough to overcome this bias for the neuron to output a positive value. In essence, the bias controls how excitable in general the neuron is, whereas the weights of the non bi- as inputs control how significant those inputs are, and whether the neuron consid- ers them excitatory or inhibitory. In future figures we will use a much simpler neuron schematic than the one we used in Fig-2.6 . Having now demonstrated the inner workings of a neuron, in the future when diagramming a neuron we will use a circle, with multiple inputs, and an output link that fans out the neuron's output signal.

When we connect a few of these neurons together in the right topology and set their weights to the right values, forming a small neural network like the one in Fig-2.7 , such a neural network could perform useful tasks. In Fig-2.7 for example, the neural circuit composed of 3 neurons calculates the XOR of the inputs. We can demonstrate that this neural circuit does indeed calculate the XOR of its inputs by feeding it the signals from a XOR truth table, and comparing its output to the proper output of the XOR logical operator. The input signals, in this case a single vector of length 2, is fed from the truth table to the neurons A and B, each neuron calculates an output signal based on its weights, and then forwards that signal to neuron C. Then neuron C calculates an output based on the inputs it receives from neuron A and B, and then forwards that output onwards. It is this final output, the output of the neuron C, that is the output of the neural circuit. And it is this output that we will compare to the proper output that a XOR logical operator would produce

Table 1. The XOR truth table, and the vector form which can be used as input/output signals of a NN. In this table, 1 == true, -1 == false.

Pattern

1

2

3

4

[X1, X2, Y]

[-1,-1,-1]

[-1, 1, 1]

[ 1,-1, 1]

[ 1, 1,-1]

Input: [X1, X2]

[-1,-1]

[-1, 1]

[ 1,-1]

[ 1, 1]

Output: [Y]

[-1]

[ 1]

[ 1]

[-1]


when fed the same input signals as the neural circuit at hand. The XOR truth table is shown in the following table, where X1 and X2 are the inputs to the XOR logical operator, and Y is the XOR operator's output.

We will now walk through the neural circuit, neuron by neuron, step by step, and calculate its output for every input in the XOR truth table. As shown in Fig- 2.7 , the neural circuit has 3 neurons, A, B, and C. Neuron A has the following weight vector: [2.1081,2.2440,2.2533], where: W1=2.1081, W2=2.2440, and Bi- as=2.2533. Neuron B has the following weight vector: [3.4963,-2.7463,3.5200], where W1=3.4963, W2=-2.7463, and Bias = 3.5200. Finally, Neuron C has the following weight vector: [-2.5983,2.7354,2.7255], where W1=-2.5983, W2=2.7354, and Bias=2.7255. With this information we can now calculate the output of the neural circuit for every input vector, as shown in Fig-2.7 .

Fig. 2.7 Calculating the output of the XOR neural circuit.

As can be seen in the above figure, the neural circuit simulates a XOR. In this manner we could even build a universal Turing machine, by combining such XOR neural circuits. Another network of neurons with another set of activation func- tions and neural weights would yield something different...

The main question though is, how do we figure out the synaptic weights and the NN topologies needed to solve some problem, how for example did we figure out the weights for each of these 3 neurons to get this neural circuit to act as a XOR operator? The answer is, a learning algorithm, an automated algorithm that sets up the weights. There are many types of algorithms that can be used to setup the synaptic weights within a NN. Some require that we have some kind of train- ing sample first, a set of inputs and outputs, which a mathematical function can then use to set up the weights of a neural network. Other algorithms do not require such prior knowledge, all that is needed is for each NN to be gaged on how well it performed and how its performance on some problem compares to those of other NNs. We will discuss the various learning algorithms in section 2.4, but before we move on to that section, we will first cover the standard Neural Network terminol- ogy when it comes to NN topological structures, and discuss the two types of basic NN topologies, feedforward and recurrent, in the next section.

2.3 Neural Networks and Neural Network Based Systems

A neuron by itself is a simple processing element. It is when we interconnect these neurons together, in parallel and in series, when we form a neural network (NN), that true computational power emerges. A NN is usually composed of mul- tiple layers, as the example shows in Fig. 2.8 . The depth of a NN is the number of layers that compose it.

Fig. 2.8 A multi-layered NN, with a NN composed of 3 layers. The first layer has 3 neurons, the second layer has 1 neuron, and the third layer has 3 neurons.

Using layers when discussing and developing NN topologies gives us an ability to see the depth of a NN, it gives us the ability to calculate the minimum number of neurons the input has to be processed by in series, before a NN produces an output. The depth tells us the minimum amount of non parallel processing that has to be done by a distributed NN. Finally, assigning each neuron a layer allows us to see whether the connections from one neuron to another are feed forward, meaning some neuron A sends signals to a neuron B which is in front of neuron A, or whether the connection is recurrent, meaning some neuron A sends a signal to neuron B which itself is behind A, and whose original output signal is either fed directly to neuron A, or was forwarded to other neurons and then eventually got to neuron A before it itself produced its output signal (the recurrent signal that it sent back to neuron B). Indeed in recurrent NNs, one can have feedforward and feed- back loop based neural circuits, and a neuron B could have sent a signal to neuron A, which then processed it and sent its output back to neuron B... When a neural network is composed of neurons whose output signals go only in the forward fac- ing direction, such a network is called a feedforward NN. If the NN also includes some recurrent connections, then it is a recurrent NN. An example of a feedforward and a recurrent neural network is shown in Fig-2.9 .

Fig. 2.9 An example of a Feedforward and a Recurrent neural network.

As can be seen in the recurrent NN example, neuron A receives a signal from somewhere, processes it, sends a signal to neuron B, which processes the signals sent to it and then sends an output signal to neuron C, D, but also a recurrent sig- nal back to A and itself.


2.3.1 Recurrent Neural Networks and Memory Loops

What is significant about recurrent neural networks is that they can form memory circuits. For example, the Fig-2.10 shows four examples of a recurrent NN. Note that in 2.10A, the neuron sends a signal back to itself. This means that at every moment, it is aware of its previous output, and that output is taken into ac- count when producing a new output. The neuron has memory of its previous ac- tion, and depending on the weight for that recurrent connection, its previous signal at time step T can play a large or a small part in its output at a time step T+1. In 2.10C neuron 1 has a recurrent connection to neuron 2 , which outputs a signal back to neuron 1 . This neural circuit too forms a memory system, because this cir- cuit does not simply process signals, but takes into account the information from time step T-2, when making a decision with regards to the output at time step T. Why T-2?, because at T-2 neuron 1 outputs a signal to 2 rather than itself, it is

Fig. 2.10 An example of recurrent NNs that could potentially represent memory loops.

A is a general, 3 layer recurrent neural network, with 3 recurrent connections. B is a self recur- rent neuron, which thus has a trailing memory of its previous output, depending on its weight with its own recurrent connection. C is a two layer recurrent NN, with neuron-2 receiving a signal from neuron-1, which processes the signal that came from neuron-2 in the first place, thus neuron-2 receives a signal that it itself produced T-2 steps before, processed by neuron-1. Finally, D is a one layer recurrent NN, which has the topology of a flip flop circuit.

then at T-1 that 2 outputs a signal to 1, and it is only at time T that 1 outputs a signal after processing an input from some other element, and a signal it output at T- 2, which was processed by 2 before coming back to 1 again. Thus this memory loop is deeper, and more involved. Even more complex systems can of course be easily evolved, or engineered by hand.

2.3.2 A Neural Network Based System

We have discussed neural networks, and in all figures I've shown the NNs as having input signals sent to them from the outside, but from where? In real im- plementations the NNs have to interact with the real or simulated world, and the signals they produce need to be somehow used to accomplish useful tasks and act upon those real or simulated worlds. For example, our own brain accepts signals from the outside world, and signals from our own body through the various senso- ry organs, and the embedded sensory neurons within those organs. For example our eyes, our skin, our nose... are all sensory organs with large concentrations of sensory elements that feed the signals to the vast neural network we call our brain. These sensory organs, these sensors, encode the signals in a form that can be for- warded to, and understood by, the brain.

The output signals produced by our brains also have no action without some actuators to interpret those signals, and then use those signals to act upon the world. The output signals are made sense of by the actuators, our muscles for exam- ple evolved to know how to respond when receiving signals from the motor neu- rons, and it is our muscles that perform actions upon the world based on the sig- nals coming from the biological NN.

Thus, though it is the NN that thinks, it is the NN with sensors and actuators that forms the whole system. Without our sensory organs, our brain is in the dark, and without our muscles, it does not matter what we think, because we can have no affect on, and no way to interact with, the world.

It is the same with artificial NNs. They require sensors, and actuators. A sensor can be a camera, which can package its output signals in a way that can be under- stood by the NN, for example by representing the sensory signals as vectors. An actuator can be a motor, with a function that can translate the NN's output vector into electrical signals that controls the actual motor.

Thus it is the whole thing, the sensors connected to and sending the sensory signals to the NN, and the NN connected to and sending its output signals to the actuators, that forms the full system, as shown in Fig-2.11 . In this book we will re- fer to such a complete and self contained system, the Sensors connected to the Neural Network, which itself is connected to Actuators, as the NN based system,

2.4 Learning Vs. Training

or NN based agent. It is only when we are discussing the NN in isolation, the to- pology of a NN for example, that I will use the term NN on its own. When it's clear from the discussion though, the two terms will sometimes be used inter- changeably.

Fig. 2.11 The Biological and the Artificial Neural Network Systems compared.

Having now discussed the basics of NNs, the different types of topologies, and what a complete NN system is, and what parts form a NN system, we now move forward and briefly cover how the classical, typical NNs learn and get trained. In the following sections we will discuss the typical algorithms used to modify the weights of the neurons belonging to some NN applied to a problem, and the dif- ference between the term learning and training.

Though most of the time you will hear the terms learning and training used in- terchangeably when people discuss the processes and algorithms that modify the weights and the topology of a NN such that it is more fit, such that it is able to solve some problem it is applied to, in this book we will discriminate between the two. Take for example the Back Propagation (BP) Learning algorithm we will discuss in the next section. In that algorithm we have a list of tuples of inputs and ex- pected outputs. The inputs are the vectors we would feed to a NN system, and the outputs are the expected outputs we'd like the NN system to produce. The way the BP learning algorithm works is by letting a neural network output a vector based on the input, and then use a gradient descent method to change the weight para- meters of the neurons based on the difference of the NN's actual output, and the expected output. Through the application of the BP algorithm, eventually the difference between the NN's output and the expected output, is minimized. Once the error, the difference between the NN's output and the expected output is below some threshold, we apply this NN to data that it has not yet seen, and use the NN's output as the result, hoping that the NN can generalize from the training set to this new real world data. When using this algorithm, is the NN really learning?

When we think of learning, we think of studying, of one looking at the data, and then through logic, and explanation to oneself, coming to a conclusion that something should work this way or that way. The starting NN, and the end result, are the same NN, and the change to the reasoning, and thus to the topology and synaptic weights is self initiated and self inflicted. We are the same before and af- ter we learn something, in a sense that this change in our logic in our perception was not done from the outside by some external system, but instead, it was us that has done the change, it was us that had worked and came to the conclusion that another way of thinking is better, or that something works this particular way... That is learning. In the BP algorithm we just discussed above, the NNs are static, they are not learning. We simply bring into existence a NN, see whether how it behaves now is appropriate and whether it represents the answer to some question, and then we change its weights, the synaptic weights of the neurons are modified and optimized from the outside, by an outside supervisor. The NN is trained. This is something that is referred to in the standard Neural Network literature as Super- vised Learning, where the NN has a supervisor that tells it whether its answers are right or wrong, and it is the supervisor (an external algorithm) that modifies the NN so that the next time it will hopefully produce a better answer.

In true learning, the NNs are able to change on their own through experience. The NN only lives ones, and during that lifetime it is modified through experi- ence. And what experience it is exposed to is to a great degree guided by the NN itself. In the way that what we choose to expose ourselves to, influences what we learn, and how our perspectives, how we think, and what we know, changes. The phenomenon of the neural networks changing and adapting through experience, is due to neural plasticity. Neural plasticity is the ability of the neuron to change due to experience. Thus for example if we create a large NN system composed of plas- tic (those possessing plasticity) neurons, and then release it into a virtual environ- ment and it improves on its behavior, it learns how to survive in the environment through experience... that is what I would refer to as learning. This is called Unsu- pervised Learning, and indeed that is completely possible to do in artificial neural networks, by for example giving each neurode the functionality which allows it to change its information processing strategy based on the signals it processes.

Thus the main idea to be taken from this section with regards to the difference between what I call training and learning, is this: The process of training a neural network is accomplished by changing its weights and topology from the outside, by some algorithm external to the NN based system. On the other hand, a neural network is learning if it is adjusting and improving itself of its own volition, through its exposure to experience and the change of its NN topology and neural parameters. Thus it would be possible to bootstrap a NN system, by first training some static system, then adding plasticity to the NN, and then releasing this boot- strapped NN system into some environment, where based on the bootstrapped part it is able to survive, and as it survives it is being exposed to the environment, at which point its plastic neural system changes and adapts, and the NN learns. We will explore this further in later chapters, after we've built a neuroevolutionary system that can evolve NN systems, and where the NN systems are then released into some simulated environment. We will evolve NN systems which have plastic- ity, we will evolve them so that they can use that plasticity to learn new things on their own.

In the following two sections we will discuss the typical supervised and unsu- pervised training and learning algorithms respectively.

Supervised learning is a machine learning approach to inferring a target func- tion from a training data set composed of a set of training examples. Each training example is composed of an input vector, and a desired or expected output vector. The desired output vector is also referred to as the supervisory signal. When ap- plied to neural networks, supervised learning, or training, is an approach to the modification and automation of weight setting of a neural network through the use of a supervisor, or external system, that compares the NN's output to a correct, pre-calculated output, and thus expected output, and then based on the difference between the NN's output and the expected output, modifies the weights of the neurons in the NN based on some optimization algorithm. A supervised “learning ” algorithm can only be applied to problems where you already know the answers, where you can build a training set . A training set is a list of tuples, where every tuple is composed of the input vector, and the expected output vector: [{Input, ExpectedOutput}...]. Thus we need to know the outputs ahead of time, so that we can train the neural network before we can use it with input signals it has not yet seen. Note, this is not always possible. For example, let's say we wish to create a neurocontroller for a robot, to survive in some environment. There is no training set for such a problem, there is no list of tuples where for every camera input that act as robots eyes there is an expected and correct move that the robot must make.

That is usually never the case, in fact, we do not know what the right move is, if we knew that, we would not need to create the neurocontroller. Another example is the creation of a neurocontroller that can make a robotic arm reach for some point in space. Again, if we knew what the right combination of moves that the motors needed to make, we would not need for the NN to figure that out.

The most widely used of such supervised algorithms, is the Error Backpropaga – tion algorithm [10]. The backpropagation algorithm uses gradient descent to look for the minimum error function between the NN's output, and the expected output. The most typical NN topology that this algorithm is applied to, is a standard feed- forward neural network (though there is a BP algorithm for a recurrent NN topol- ogy too). As we discussed, a supervised learning algorithm trains a NN to approx- imate some function implicitly, by training the NN on a set of inputs and expected outputs. The error that must be minimized is the error between the NN's output, and the expected output.

Because we will concentrate on neuroevolution, we will not cover this algo- rithm in great detail. But an extensive coverage of this supervised learning algo- rithm can be found in: [11,12]. In summary, the training of the NN through the backprop algorithm works as follows:

1. Create a multi-layered feed forward neural network, where each neuron has a random set of weights. Set the neurons in the first/input layer to have X number of weights, plus bias, where X is the vector length of the input vectors. Set the last/output layer to have Y number of neurons, where Y is the length of the ex- pected output vector.
2. For every tuple(i) in the training list, DO:
3. Feedforward Phase:
1. Forward Input(i) vector to the neurons in the first layer of NN.
2. Gather the output signals from the neurons in the last layer of NN.
3. Combine the gathered signals into an Output(i) vector.
4. Backprop Phase:
1. Calculate the error between the NN's Output(i) and ExpectedOutput(i)
2. Propagate the errors back to the neurons, and update the weights of the neurons based on their contribution to that error. The weights are updat- ed through gradient descent such that the error is decreased.
3. The errors are propagated recurrently from the last neural layer to the first.
5. EndDO
6. Repeat steps 2-5 until the average total error between the NN's outputs and the expected outputs is less than some chosen value e .

Schematically, the feedforward phase and the error backprop phase, is demon- strated in Fig. 2.12 .

Fig. 2.12 The schematic of the backprop learning algorithm.

The steps of recursively updating the synaptic weights of all the neurons in the feedforward NN based on the error between the NN's output and the expected output is demonstrated by the figure through the step numbers. Starting with step 1 , the NN's output is O , and the expected output is X . If there were more than one output neurons, then each neuron i would produce an output Oi , and for each Oi there would be an expected output Xi. The meaning of the steps is elaborated on in the following list:

1. The neuron in the output layer of the feedforward NN produces an output O.
2. The error of the neuron's output as compared to the expected output is e, calculated as: e = Xi-Oi where Xi and Oi are the output of neuron i, and expected output i, respectively, if there are i number of output neurons in the NN.
3. We calculate b (beta) by multiplying the derivative of the activation function by e: b = e*AF'(S) , where S is the dot product of the neuron's input signals and synaptic weights for those input signals.
4. We then calculate the delta (change in) weight for each weight i as follows: dw(i) = n*b*Xi, where n is a learning parameter chosen by the researcher (usu- ally between 0.01 and 1), b is the value calculated in step-3, and Xi is the input i to the neuron, associated with the weight i.
5. We updated every synaptic weight i of the neuron using the appropriate dw(i) for each Wi . The updated weight is produced through: U_Wi = Wi+dw(i) .
6. The next e (error) is recursively calculated for every presynaptic neuron using the equation: e=Wi*b , where Wi is the synaptic weight associated with the neu- ron whose output was Xi.
7. We calculate b (beta) by multiplying the derivative of the neuron's activation function by e: b = e*AF'(s) .
8. We then calculate delta weight for each weight i as follows: dw(i) = n*b*Xi where n is a learning parameter chosen by the researcher (usually between 0.01 and 1), b is the value calculated in step-7, and Xi is the input i to the neuron, associated with the weight i.
9. We updated every synaptic weight i of the neuron using the appropriate dw(i). The updated weights of the neurons are calculated through: U_Wi = Wi+dw(i) .

This procedure is continued recursively to the other presynaptic neurons, all the way to, and including, the first layer neurons.

Thus, to optimize the neural network's weights for some particular task for which we have a training set, we would apply the backprop algorithm to the NN, running it through the training set multiple times, until the total error between the NN's output and the expected output is low enough that we consider the NN's synaptic weights a solution. At this point we would apply the NN to the real prob- lem for which we have been training it.

As noted, this can only be applied to the problems for which we already know the answers, or a sample of answers. This algorithm is used only to train the neural network, once it is trained, its weights will remain static, and the neural circuit is used as a program, unchanging for the remainder of its life. There are numerous extensions and improvements to this basic algorithm, covered in the referenced texts. But no matter the improvements, at the end of the day it is still a supervised approach, and the resulting NN is static. In the next section we will briefly discuss unsupervised learning algorithms, the addition of plasticity to the neurons of a NN, and other methods which allow the NN to self organize, and adapt and change through the interaction with the environment, and/or data it comes across.

2.6 Neural Network Unsupervised Learning Algorithms

Unsupervised learning refers to the problem of trying to determine structure in incoming, unlabeled data. In such a learning algorithm, because the input is unla- beled, unlike the case with the training data set discussed in section 2.5, here there is no error or reward signals which can be used to guide the modification process of neural weights based on the difference between the output and the expected output. Instead, the NN self modifies its parameters based on the inputs and its own outputs through some algorithm. There are two general kinds of such learning algorithms; a learning algorithm can either be a system that has a global view of the NN, and which uses this global view to modify neural weights (kohonen, competitive …), or a learning algorithm can be a local on, embedded in each neu- ron and letting it modify its own synaptic weights based on its inputs and outputs (hebbian, modulated …).

Our brains do not have an external supervisor, our brain, the biological neurons that compose it, use different types of unsupervised learning, in a sense that they have plasticity and they change based on their experience. There is evidence that the hippocampus plays a very important role [13] in the formation of new memories, which means that a neural circuit like the hippocampus can modulate, or af- fect the topology and neural weights located in other parts of the brain, other neu- ral networks. Thus, in a sense there is also modulation of learning algorithms at a more global scale of the neural network, and not just at the level of single neurons. Our brains of course have evolved the different features, the different rates of neu- ral learning through experience, and the different neural circuits within our brain which affect and modulate other parts of our brain...

Though we could include a form of hebbian learning in neurons (discussed next), and create a large homogeneous hebbian or kohonen neural network... it will still be nothing more than a clustering network, there will be no self aware- ness within it. To create a truly intelligent neurocomputing system, we need to combine static neurons, neurons with plasticity, and different forms of unsuper- vised learning algorithms... all into a vast neural network. And combine it in a way that all these different parts work together perfectly, and allow for the whole emergent NN system to truly learn, which is the case with evolved biological neural networks.

In this section we cover the unsupervised learning approaches, how to make neurons plastic, how to allow neurons to change their own weights through expe- rience... The actual method of putting all these various systems together into a vast network that can have the potential of true learning, will be the subject of the rest of this book, with the method taken to accomplish this goal, being evolution. For the sake of exposure, and because we will use these particular unsupervised forms of NN learning once we've developed our basic neuroevolutionary platform and began expanding it beyond the current state of the art, we will briefly cover 4 particular unsupervised learning algorithms next.

2.6.1 Hebbian Learning

In 1949 Donald Hebb proposed a computational algorithm to explain memory and the computational adaptation process within the brain, he proposed a rule we now refer to as the Hebbian learning. The Hebbian learning is a neural learning algorithm that emulates plasticity exhibited by neurons, and which has been con- firmed to a great extent to exist in the visual cortex [14].

As Hebb noted [26], “The general idea is an old one, that any two cells or sys- tems of cells that are repeatedly active at the same time will tend to become ‘asso- ciated', so that activity in one facilitates activity in the other. ”, or more concisely: “neurons that fire together, wire together ”.

The basic Hebbian rule for associative learning can be written as follows: For every weight w(i) in a neuron B, we add to the weight w(i) the value dw(i) where dw(i) = x(i)*O . This equation simply states that if we have a neuron B, which is connected from a number of other elements, and which produces an output O after processing the presynaptic x(i) input signals, and has a weight w(i) for every input signal x(i), then the change in the weight w(i) is x(i)*O. We can see that if both x(i) and O have the same sign, then the change in synaptic weight is positive and the weight will increase, whereas if x(i) and O are of opposite signs, then the weight will decrease for the synaptic connection between neuron B and the pre- synaptic element which sent it the signal x(i). So then for example, imagine that we have a NN with 2 neurons, in which neuron A is connected to neuron B. If neuron A sends a positive signal to neuron B, and this makes neuron B output a positive signal, then B's synaptic weight for the connection coming from A in- creases. On the other hand, if A's signal to B makes B produce a negative signal, then B's synaptic weight associated with A's signals is lowered. In this manner the two neurons synchronize. Fig-2.13 demonstrates this scenario and shows the hebbian rule in action.

Fig. 2.13 Neuroplasticity through Hebbian learning.

In the above figure, we can see that just from one signal coming from Neuron A , a signal that was positive and thus producing a positive delta weight with re- gards to the positive synaptic weight of B, B's neural weight nearly doubled for the connection with A . A few more signals from A , and the weight aw1 would have grown significantly larger, and eventually drowned out any other weights to other links. Thus the problem with the original and very simple Hebbian learning rule is that it is computationally unstable. For example, as noted, the weights do not saturate, they can continue growing indefinitely. If that does occur, then the weights that grow fastest will eventually drown out all other signals, and the out-put of the neuron, being a sigmoid of tanh, will always be 1. Thus eventually the neuron will stop truly discerning between signals, since its weights will be so large that no matter the input, 1 will always be the neuron's output. Another problem is that, unlike in a biological neuron, there is no weight decay in the original Hebb's rule, there is no way for the synaptic weights for the incoming signals to become weaker.

New learning algorithms that fix computational instabilities of the original Hebbian rule have been created. For example, three versions of such rules are the Oja's rule [15], the Generalized Hebbian Algorithm (GHA) aka Sanger's rule [16], and the BCM rule [17]. The Oja's and BCM rules in particular, incorporate weight decay, and are more biologically faithful. In Fig. 2.14 I demonstrate how a neuron using the Oja's learning algorithm updates its synaptic weights after hav- ing processed its input vector.

Fig. 2.14 Neuroplasticity through Oja's rule.

As can be seen in Fig-2.14 , unlike the original Hebbian rule, Oja's rule pro- duces a smaller weight increase, but more importantly, if we process a few more signals, then we would notice that the weight does not grow indefinitely. This computational system is stable, the weight eventually saturates at some viable val- ue, and if that particular synapse, and thus the synaptic weight associated with it, is not stimulated any further by incoming signals (the incoming signals are not as high in magnitude), the weight begins to decay, memory slowly deteriorates.

The only problem is that, if there were to have been more than one synaptic weights (w1, w2...wi), they would all still follow the same type of rule, the same learning rate ‘n'. To make that rule even more flexible, we can employ neuromodulation, which allows for every synaptic weight to update differently from every other, making the plasticity of the neuron even more flexible and realistic. This form of unsupervised learning is discussed next.

2.6.2 Neuromodulation

In biological neural networks, neuromodulation refers to the process of the release of several classes of neurotransmitters into the cerebrospinal fluid, which then modulate a varied class of neurons within reach of the released neurotrans- mitters. In this manner, a neural circuit that releases the neurotransmitters into the cerebrospinal fluid, can affect some area of neural tissue, augmenting its behavior by making it more easily exited or inhibited for example. Neuromodulation can al- so be direct, when one neuron is connected to another, and depending on this modulatory neuron's signals, the behavior of the modulated neuron, the way it processes information, is modified.

In artificial neural networks, the same can be accomplished. We can allow a neuron or a neural circuit to use its output to modulate, or control the plasticity type and the adaptation pace (learning parameter for example) of another neuron or neural circuit. Thus for example assume that we have 2 neural circuits, A and B, which form a neural network based system. Circuit A is connected from a set of sensors, and to a set of actuators. Circuit B is also connected to the same set of sensors, but its output signals, instead of going to the actuators, are used to modu- late and dictate how the weights of the neurons in circuit A change and adapt over time, as shown in Fig-2.15 . Thus, in this neural network system circuit B modu- lates circuit A, and controls that circuit's ability to learn, pace of learning, and the learning algorithm in general. Since a neural network is a universal function approximator, this type of learning algorithm can be highly versatile and robust, and the modulatory signals produced by the neural circuits can be of any form.

Fig. 2.15 A Neural Network based system with plasticity through neuromodulation. In this figure, Circuit-B modulates Circuit-A's learning algorithm.

The equation used to add plasticity to a neuron through neuromodulation is: DWij = L = f*N(A*Oi*Oj + B*Oi + C*Oj) . DWij is the delta weight, change in the synaptic weight of neuron j for the link coming from neuron i. N is the learning rate, which dictates the general magnitude of weight change after the neuron processes a signal. A, B, and C are parameters weighting the contribution of the output signal coming from the presynaptic element i and the output signal produced by the postsynaptic neuron j, and together forming the non linear plasticity factor. Finally, the value f is a further modulatory signal which dictates how rapid- ly, and in what direction the weight will change based on the learning rule L. In standard neuromodulation, the value f is produced by the modulating neuron or neural circuit, and the parameters N, A, B, and C are set by the researcher, or evolved and optimized through a neuroevolutionary process. But the parameters N, A, B, and C can also be produced by the modulatory neural circuit B in vector form for each neuron in circuit A, to modulate and give those neurons even more dynamic neuroplasticity.

In a sense, we can think of Circuit-B as being the biological part of circuit A, that part which produces plasticity. We can recreate the neural network shown in Fig-2.15 to be composed not of two separate neural circuits, one which does the actual processing (A) and one which does neuromodulation (B), but instead com- posed of one neural network, where every neuron has an embedded circuit B, which gives it plasticity. This type of neural architecture is shown in Fig-2.16 .

Fig. 2.16 Another type of neuromodulatory architecture.

From the above figure, we can see that each neuron now has the functionality of Circuit-B embedded inside of it. Also, there is a small change in how this new Circuit-A functions. The embedded Circuit-B does not use as input the signals coming from the two sensors, but instead uses as its input the same input as the neuron to which it adds plasticity. In this manner the modulatory circuit sees the input signals of the neuron which it modulates, making the modulation signal spe- cific to the data of the neuron in which it is embedded.

An actual example of the steps taken in processing signals by a neural network with plasticity shown in Fig-2.15 is presented in Fig-2.17 . The sequence of events in such a NN is demonstrated by the numbers given for the various steps, and is further elaborated in the following list:

1. The two sensors produce signals, and forward them to the neurons in the first layers of Circuit-A and Circuit-B.
2. The two neurons of Circuit-A process the signals from the two sensors, and produce outputs. The neuron of Circuit-B also at the same time processes the signals from the two sensors, producing the output and forwarding it to the neu- ron in the next neural layer of Circuit-B.
3. The second neuron in the Circuit-B processes the signal coming from the pre- synaptic neuron.
4. Circuit-B produces the modulatory signal, sending it to all neurons of Circuit- A. Since the first two neurons in Circuit-A have already processed their input signals, they use this modulatory signal and then do both, update their synaptic weights based on this modulatory signal, and update their learning rule parame- ters, where the used learning rule might be: General Hebbian, Oja's Rule, or some other.
5. The neuron in the second layer of Circuit-A produces an output after pro- cessing the signals sent to it by the two presynaptic neurons in the first layer of Circuit-A.
6. The neuron in the second layer of Circuit-A uses the modulatory signal sent to it by Circuit-B in step-4 to update its synaptic weights, and learning rule pa- rameters.
7. The sensors produce another set of signals and forward those signals to the neu- rons they are connected to. The loop repeats itself.

Fig. 2.17 Neuromodulation in action.

At this point you are probably asking yourself the following question: Sure, now we can allow for some neuron to learn and adapt, to possess plasticity... but plasticity is controlled by another type of neural network, so how do we set up that other neural network's synaptic weights and parameters so that it can actually produce the modulatory signals that are useful in the first place? That is a valid question, in fact, for example in the figure above where we modulate two neurons, instead of having to set up those neuron's synaptic weights, we have to set up the weights of the neurons in Circuit-B, each possessing 2 weights. We can do this through evolution. Evolution can optimize the synaptic weights and the various parameters needed by the modulatory neural circuits, which would then modulate effectively the other neural circuits of the complete neural network.

Unlike the static simple neurons, the neurons shown in the above figure are complex, plastic, but highly robust and adaptive elements. A neural network of such elements, evolved to work coherently as biological neural networks do, would have quite a significant amount of learning ability. We will build such sys- tems and variants of it in later chapters, we will embed such adaptive and plastic neural networks in artificial organisms when we'll apply our neuroevolutionary system to ALife simulations, and as you will see, such systems do indeed have high potency, and might be exactly the building blocks needed when the goal is to evolve an intelligent neurocognitive system.

2.6.3 Competitive Learning

Competitive Learning [18] is another form of unsupervised learning, but unlike the Hebbian and the neuromodulation methods which add plasticity to each neu- ron, this one requires some system/process that has a global view of all the neu- rons forming the neural network undergoing competitive learning. In competitive learning we have a set of neurons, each of which is connected to a given list of sensors, and where each neuron competes with the others for the right to respond to a subset of sensory signals. Over time, competitive learning increases the spe- cialization of each neuron for some particular set of signals, and thus allows the NN to act and spontaneously form a clustering/classification network.

A NN which uses competitive learning (CL) is realized through the implemen- tation of the following set of steps:

1. Choose j number of sensors whose signals you wish to cluster or classify.
2. Create i number of neurons, each connected to all j sensors, and each neuron using a random set of synaptic weights.
3. DO:
1. Propagate the signals from sensors to the neurons.
2. Each neuron processes the sensory signals and produces an output signal. 3. An external CL process chooses the neuron with the highest output signal magnitude.
4. The CL updates the synaptic weights of that neuron by applying to it a form of Hebbian learning (by using the Oja's rule for example).
4. UNTIL: The network begins to cluster signals, and a pattern begins to emerge. This is a simple learning rule that can be used to see if there is a pattern in the data, and if those signals can be clustered. Fig-2.18 shows a diagram of a NN sys- tem utilizing the competitive learning algorithm.

Fig. 2.18 A neural network employing competitive learning.

2.6.4 Kohonen/Self Organizing Map

A Kohonen map [19], also known as a self organizing map (SOM), is a type of neural network that in a sense represents a hypercube or a multidimensional grid of local functions, and through the use of a form of competitive learning the SOM performs a mapping of data from a high dimensional space into a lower dimen- sional one, while preserving that data's topology. These types of neural networks originated in the 80s and are loosely based on associative memory and adaptive learning models of the brain. Like the competitive learning neural network, a SOM system requires a process that has a global view of the NN, so that learning can be achieved. An example of a 2d SOM system is shown in Fig-2.19 .

Fig. 2.19 A self organizing map, where the SOM_LA process performs SOM based Learn- ing Algorithm computations, and synaptic weight updates.

To set up a Kohonen map we create a hypercube based substrate with embed- ded neurons within, where each neuron has a set of weights and a coordinate with- in the substrate. Each axis of the hypercube ranges from -1 to 1, and the neurons are embedded regularly within the substrate (this is somewhat similar to the hy- percube representation we discussed in Chapter 1.2.10, which is used by the HyperNEAT system). The actual density, the number of neurons forming the SOM, is set by the researcher. Finally, each neuron in this hypercube is connected to the same list of sensors.

The learning algorithm used by a SOM is somewhat similar to one utilized by the competitive learning we discussed in the previous section. When the sensors propagate their vectors to the neurons, we check which of the neurons within the hypercube has a weight vector which is closest to the input vector based on a Car- tesian distance to it. The neuron whose weight vector is the closest to the input vector is called the best matching unit, or BMU. Once this neuron is found we apply the weight update rule: Wv (t + 1) = Wv (t) + Θ (d)* α (t)*( I (t) – Wv (t)) , to all neurons in the hypercube, where Wv (t+1) is the updated weight vector, Wv (t) is the neuron's weight vector before the update, (t) is a monotonically decreasing learning coefficient similar to the one used in simulated annealing [20,12], I (t) is the input vector, and Θ (d) is usually the Gaussian or the Mexican-Hat function of the distance between the BMU and the neur on in question (thus it is greatest for the BMU neuron, and decreases the further you move away from the BMU) . Once the new weight vector is calculated for every neuron in the hypercube, the sensors once again fanout their sensory vectors to the neurons. We continue with this pro- cess for some maximum X number of iterations, or until (t) reaches a low enough value. Once this occurs, the SOM's output can be used for data mapping. A trip through a single iteration of the SOM learning algorithm is shown in Fig-2.20 .

Fig. 2.20 Self Organizing Map in action.

The following list elaborates on each of the algorithm steps in the above figure:

1. The sensors forward their signals to all the neurons in the substrate (each neuron has its own coordinate).
2. Each neuron in the substrate processes the incoming signals, and produces an output.
3. A process by the name SOM_LA which has a global view of all the neurons in the substrate, compares the neural weights to the input vectors for every neu- ron.
4. SOM_LA finds the neuron whose synaptic weight vector is closest to the input vector coming from the sensors.
5. SOM_LA updates that neuron's synaptic weights, and updates the synaptic weights of the neurons around it, with the synaptic weight update decreasing in magnitude proportionally to the distance of those other neurons to the win- ning/chosen neuron.
6. The sensors forward their signals to all the neurons in the substrate... The loop repeats itself.

There are numerous variations on the original Kohonen map, for example the General Topographic Map (GTM) and the Growing Self Organizing Map (GSOM), are two of such advanced self organizing maps.

2.6.5 Putting it All Together

In this chapter we have discussed 4 different types of unsupervised learning al- gorithms. There are of course many others, like the Hopfield memory network that models associative memory, and the Attenuated Resonance Theory (ART) NN that models a scalable memory system. We can see that such unsupervised learn- ing algorithms add plasticity to the neurons, and the neural networks in general. But, these types of learning algorithms themselves have parameters that need to be set up before the system can function. And what about the general topology of the NNs which possess plasticity? After all, we can't simply add some unsupervised learning algorithm to a random NN structure, and then expect it to immediately possess intelligence. The way neurons are connected to one another in the NN, the topology itself, is just as important, if not more so, than the synaptic weights of the neurons. A neurocognitive system possessing intelligence will certainly have to utilize many of these types of NNs and the different plasticity types they possess. This possible future neurocognitive system will integrate all these learning neural circuits into a single, cohesive, synchronized, vast neural network system, pos- sessing the topology and architecture similar to an example shown in Fig-2.21.

Fig. 2.21 A possible vast NN composed of neurons with and without plasticity, and different interconnected neural circuit modules. The flexibility of an evolved NN based system which draws upon and uses all the available learning algorithms, encodings, neuron types... could potentially be immense.

How can we figure out how to put these modules together, how to connect the neurons in the right manner, how to bootstrap a NN system so that it can take over from there, and so that its own intelligence and ability to learn can continue the work from that point onwards? That problem has already been solved once before, we are the result; the solution is evolution.

2.7 Summary

In this section we have discussed how biological neurons process information, their ability to integrate spatiotemporal input signals, and change their signal pro- cessing strategy, a process called plasticity. We then discussed how artificial neu- ral networks process signals, and that the most common such neural networks deal with amplitude encoded signals, rather than frequency encoded signals as is the case with biological neural networks. Although as noted, there are artificial neural networks called spiking neural networks, which like biological NNs deal with fre- quency encoded signals.

We then discussed the various topologies, architectures and NN plasticity rules. We discussed how a recurrent NN exhibits memory, and how the Hebbian, Oja's, and neuromodulation learning rules allow for NNs to adapt and change as they in- teract and process signals. Finally, we discussed how the various parameters and topologies of these NNs can be set, through evolution, allowing for the eventual vast NN to incorporate all the different types of learning rules, plasticity types, to- pologies, and architectures.

With this covered, we move to the next chapter which will introduce the sub- ject of evolutionary computation.

2.8 References

[1] The Blue Brain Project: http://bluebrain.epfl.ch/

[2] Dawkins R (1976) The Selfish Gene. (Oxford University Press), ISBN 0192860925.

[3] Dawkins R (1982) The Extended Phenotype. (Oxford University Press), ISBN 0192880519. [4] Hornik K, Stinchcombe M, White H (1989) Multilayer Feedforward Networks are Universal Approximators. Neural Networks 2, 359-366.

[5] Sher GI (2010) DXNN Platform: The Shedding of Biological Inefficiencies. Neuron, 1-36. Available at: http://arxiv.org/abs/1011.6022.

[6] Parisi D, Cecconi F, Nolfi S (1990) Econets: Neural Networks That Learn in an Environment. Network Computation in Neural Systems 1, 149-168.

[7] Predators and Prey in simulated 2d environment, Flatland: http://www.youtube.com/ watch?v=HzsDZt8EO70&list=UUdBTNtB1C3Jt90X1I26Vmhg&index=2&feature=plcp

[8] Hassoun MH (1995) Fundamentals of Artificial Neural Networks. (The MIT Press).

[9] Lynch M (2007) The Origins of Genome Architecture S. Associates, ed. (Sinauer Associates Inc).

[10] Haykin S (1999) Neural Networks: A Comprehensive Foundation J. Griffin, ed. (Prentice Hall).

[11] Rojas R (1996) Neural Networks: A Systematic Introduction. (Springer).

[12] Gupta MM, Jin L, Homma N (2003) Static and Dynamic Neural Networks From Fundamen- tals to Advanced Theory. (John Wiley & Sons).

[13] Di GG, Grammaldo LG, Quarato PP, Esposito V, Mascia A, Sparano A, Meldolesi GN, Picardi A (2006) Severe Amnesia Following Bilateral Medial Temporal Lobe Damage Oc- curring On Two Distinct Occasions. Neurological sciences official journal of the Italian Neu- rological Society and of the Italian Society of Clinical Neurophysiology 27, 129-133.

[14] Kirkwood A, Rioult MG, Bear MF (1996) Experience-Dependent Modification of Synaptic Plasticity in Visual Cortex. Nature 381, 526-528.

[15] Oja E (1982) A Simplified Neuron as a Principal Component Analyzer. Journal of Mathe- matical Biology 15, 267-273.

[16] Sanger T (1989) Optimal Unsupervised Learning in a Single-Layer Linear Feedforward Neural Network. Neural Networks 2, 459-473.

[17] Bienenstock EL, Cooper LN, Munro PW (1982) Theory For The Development of Neuron Selectivity: Orientation Specificity and Binocular Interaction in Visual Cortex. Journal of Neuroscience 2, 32-48.

[18] Rumelhart DE, McClelland JL (1986) Parallel Distributed Processing M.I.T. Press, ed. (MIT Press).

[19] Kohonen T (1982) Self-Organized Formation of Topologically Correct Feature Maps. Bio- logical Cybernetics 43, 59-69.

[20] Kirkpatrick S, Gelatt CD, Vecchi MP (1983) Optimization by Simulated Annealing. Science 220, 671-680.

[21] Cerny V (1985) Thermodynamical Approach to The Traveling Salesman Problem: An Effi- cient Simulation Algorithm. Journal of Optimization Theory and Applications 45, 41-51.

[22] An excellent discussion of neuron and synapse: http://en.wikipedia.org/wiki/Neuron

[23] Gerstner W (1998) Spiking Neurons. In Pulsed Neural Networks, W. Maass and C. M. Bishop, eds. (MIT-Press), pp. 3-53.

[24] Ang CH, Jin C, Leong PHW, Schaik AV (2011) Spiking Neural Network-Based Auto- Associative Memory Using FPGA Interconnect Delays. 2011 International Conference on FieldProgrammable Technology, 1-4.

[25] Qingxiang W, T MM, Liam PM, Rongtai C, Meigui C (2011) Simulation of Visual Atten- tion Using Hierarchical Spiking Neural Networks. ICIC: 26-31

[26] Hebb DO (1949) The organization of behavior. Wiley, eds. (Wiley)

Chapter 3 Introduction to Evolutionary Computation

Abstract In this chapter we discuss biological evolution, and the way it has evolved the organisms and structures that we see around us today. We then extract the essentials of this natural stochastic search method, and discuss how one could implement the same, or an even more efficient version, in software. Once the standard evolutionary algorithm methods are introduced (genetic algorithms, ge- netic programming, evolutionary strategies, and evolutionary programming), we also discuss the slightly lesser known memetic algorithm approaches (hybrid algo- rithms), and how it compares to the already discussed methods. Finally, we dis- cuss the equivalency between all these methods, and the fact that all of them are just different sides of the same coin.

Evolutionary computation algorithms are population based optimization algo- rithms inspired and derived from the Darwinian principles of biological evolution. Instead of simply listing the general steps of an evolutionary algorithm, let us dis- cuss evolution in general, and then extract the essence of it, and see how such principles can be implemented in software, and used to evolve solutions to any problem.

As has been mentioned to some degree in the first and second chapter, evolu- tion is the driving force, the phenomenon that has birthed us. What are the princi- ples of evolution, how does it work? Let us again discuss the history of life on earth, the path it took, and the evolutionary principles that shaped it.

3.1 Evolution

At its core, replication, creating copies, is really the main thing [1,2,3]. There can really be no other way for life to emerge, without it all starting with a replica- tor. After all, whatever does not replicate, does not make copies, and thus will eventually be drowned out by a system that uses up all the available resources and makes copies of itself. But how do we get to a replicator, even the most simplest of ones? It is for this reason why it took billions of years, once you have a replica- tor things get easy. A simple replicator is just a molecule that through chemical reactions can make a copy of itself, given that there are resources around it that can be used. Through physical principles alone, such a molecule simply makes copies of itself [4,5], a physical chain reaction. But how do we get to such a mole- cule in the first place is of course the question.

DOI 10.1007/978-1-4614- 4463 - 3_3 , © Springer Science+Business Media New York 2013

If such a molecule is small and simple enough, then, given enough time (bil- lions of years), if you randomly bang atoms against each other, in different envi- ronments, under different circumstances, trillions upon trillions of times per se- cond, eventually you'll hit the right combination and such a molecule will arise. Now there are trillions of stars in our universe, multiple planets for each star, each planet with a different environment. If simple molecules and atoms are banging against each other, and normal chemical reactions take place on those planets, and they do, then given a few billion years, by statistics alone, there is 100% chance that at one point or another, the right combination of simple inorganic molecules and atoms will combine just right to create such a simple replicator. And then it starts...

There's no avoiding it, probability itself demands it. If there is a chance that through standard physical properties a particular combination of molecules and at- oms can come together to form a replicator, then given enough permutations and chances, that combination of atoms will occur. It might be so rare though, that we might even be the first ones, but we won't be the last. Given enough time, sooner or later another planet will have the right conditions, and be at the right place at the right time, that after a few billion years on its surface too, the right combina- tion will occur through probability and chance alone. And that is the spark, that is the start, after that, there is no choice but for evolution to start carving out a path toward complexity.

When making copies, sooner or later, there will always be an error. At some point or another, either due to the environment the replicator is in, or due to a high energy photon slamming into the replicator, there will be an error, the copy will be slightly different. Certainly, majority, almost all the errors will end up in the mu- tated copy being damaged in one way or another. But every once in a while, the mutant copy will be an even better replicator. It might perhaps be a bit more ro- bust, and not affected by errors as much (thus being able to produce more surviv- ing offspring), or it might be more efficient. Whatever the minuscule advantage, if there is one, then it and its kind will make more copies of itself than its parent rep- licator, in the long run. Of course at this point there is now not a homogeneous pool of replicators using up all the available resources, but different kinds, varia- tions of each other. We now have competition, even if the replicators themselves don't know this. After all, the replicator wants to make a copy of itself, and its fit- ness is defined by how many copies it makes... If another replicator is using up the same resources, or space, then this is competition.

At this point the replicators could have spread out through the entire planet, slowly using up the resources, converting the resources into copies of themselves. And again, due to probability alone, due to stochastics alone, one of the errors dur- ing replication will end up generating a clone offspring that has the ability to break apart other replicators around, through its chemical properties alone. This would be a great evolutionary discovery, this would be the discovery of predation. At the same time, this would spark a new type of drive towards complexity, because now replicators can affect each other, they are more unstable too, because they can break each other apart. There is also now a direct competition, if you can be bro- ken apart by another replicator merely because you are proximal to it, you will not create any offspring. Any error, where the error is simply a mutation in the created offspring, that leads to the offspring ability to not be absorbed, or be able to somehow chemically sense the aggressive replicator, will be of great benefit. Sooner or later, that combination of mutations will occur.

When such a mutant offspring is birthed, it will be able to create many more offspring than its parent. Which also means that the amount of replicators that can be broken up by the aggressive replicators will dwindle... Another mutation that results in the aggressive replicator having the ability to break apart these new re- sistant replicators, will certainly be useful. Though of course, it would again take trillions of mutations that lead no where... but sooner or later, the right mutation combination, if such a combination of mutations is possible, will occur. This will lead to a more and more complex type of replicators. It is an arms race between replicating molecules. It is evolution.

Now remember, all of this is guided simply by the fact that replicators repli- cate. There is no thinking, it's simply what they do, create copies. This is a com- pletely natural process; there is no other way for evolution to be. It has to be repli- cation, and replication has to be at its core. This is what the natural world is based on. As soon as a replicator is introduced to the environment through stochastic processes, there is no other path that can be taken than a path towards an arms race and evolution towards complexity as the replicators make copies of themselves, make errors during the creation of those copies, and unknowingly compete for limited resources as each replicator makes a copy.

At some point a mutation might lead to a replicator to form a sort of molecular boundary around itself. This will give it an advantage against predation. Again complexity will have to rise. Stochastic processes will try out different combina- tions of morphologies. Different combinations of attacking and defending... The older parents, the first ones, do not necessarily have to be wiped out. But some will, and some will remain.

We are now at a level of an organism, a replicator with a shell, a replicator with some kind of protective boundary. Its complexity is increased, this is at the level of single celled organisms. From there, multi celled organisms are just a few mil- lion years, a few trillions of combinations and mutation attempts away.

With every new species, with every new biological invention through evolu- tion, the organisms are not just interacting with each other, they are interacting with the environment itself. If we start off with a completely flat and boring envi- ronment, and all the organisms are adept at existing in that simple environment, and then suddenly one of the organisms evolves an ability to modify it, by for ex- ample digging holes... then that takes everyone to a whole new level of complexi- ty. On the one hand, the organism that can dig holes, does not only make the environment more complex, but also creates its own niche, and therefore could be- come more fit and create more offspring. If that organism eventually becomes dominant through replication, then evolution will drive other organisms towards species that can deal with this more complex environment... which would require some kind of chemical or other type of sensory ability. As new organisms arise that can deal with the new environmental complexity, and have abilities to attack or defend themselves from environment modifying organisms... new evolved in- ventions will follow.

The organisms modify the environment, and the environment grows more complex, requiring the organisms to evolve new methods of dealing with it, of sensing it, of traversing and surviving in it. The organisms change the entire planet through their interaction with it. This is a never ending drive towards complexity, and as you can see, there is really no other way for it to happen. This is what com- petition and replication leads to.

The stochastic processes of mutation, and the never ending replication and the resulting competition for resources to replicate continues for billions of years, tril- lions upon trillions of failed attempts at improvement, and a few successful ones... until finally you see in front of you what you see today. The replicators have evolved through arms race a new type of boundary around themselves for protec- tion against each other.... that eventual end result, through the slow path of time and mutation, through evolution, that protective boundary and that invention is you and I, or as Richard Dawkins put it: one of the ultimate survivor machines [6] that the replicators had evolved.

Now before we begin extracting the algorithm from this biological phenome- non of evolution, and see how we can do the same thing with software, let us tack- le the final question, which will also be useful for our understanding of the evolu- tionary process. The question is, why are there broken links in the organisms we see today? Why don't we see a smooth spectrum, all the way from the first repli- cator, to the complex organisms like mammals?

There are many reasons for this phenomenon, we will cover two of the sim- plest. The first way in which such breakages in the phylogenetic tree can occur is through extinction. Some organism might just be very good at consuming a par- ticular species. If that organism consumes some other species at a fast enough rate, that other species will become extinct. If the environment continues to change as the new, and younger species interact with it, the older species from eons past, might not be fit for it, and they will become extinct. For example, as plankton and other oxygen releasing organisms covered earth, more and more oxygen was re- leased, and higher and higher concentration of it were present everywhere. Any species to which oxygen was toxic, for some reason due to its chemical makeup for example, that species would become extinct if it did not evolve fast enough in- to an oxygen tolerant species, and evolution is just stochastics, so not every spe- cies would have the chance to evolve. Any species that needed a high carbon dioxide content in the atmosphere, would become extinct as the concentration of oxygen increased at the expense of carbon dioxide...

The second reason for the breakages in the phylogenetic tree is due to the evo- lution of the species as a whole through interbreeding. For example, let's say there is some ape that creates an offspring. That offspring has had just the right muta- tions which gave it a slightly larger brain. Its genetics are different, we have just witnessed the birth of a new type of ape. Now that ape is still close enough genet- ically to its parents, and so it can breed with the apes in that species. His genetic mutation is advantageous. This new ape will breed, and some of his offspring will inherit this genetic pattern that gives them a larger brain. Eventually that new mu- tation will spread through the population, over the next hundreds of generations. Thus, this whole population, the community of apes, will integrate this new com- plementary mutation, and the whole population evolves together. The old popula- tion, the old species, changes into a new one as it integrates new genetic patterns. Through such mutations we can go from species A to species B smoothly, over thousands of generations, without even leaving a trace of there ever being an A, because through slow change the population has become B. Then today, we would only see a species B.

With regards to reason 2, you might then pose a question, what about the fact that we do sometimes see species A and B? That also can occur, as another exam- ple, let's say we have a population, and as it explores the environment and spreads through it, it separates into a few different groups. The mutation is rare, and it will occur in one of the groups, in one of those subpopulations. If those subpopula- tions, if those groups do not interbreed, because for example they have become separated by a large distance, geographically, then when one of these populations A begins to evolve towards B, the other groups will still stay as A. At some later time, after thousands of generations, we have a population that has evolved from A to B, and another that might have not acquired any new mutations, or have went into another evolutionary direction. And so we would have A, B, and perhaps some population that evolved in another direction, C for example. Then we would have 3 species, the original A, and the mutated but sufficiently different species B and C that can no longer breed with A or each other.

We have now covered the biological properties, how evolution occurs and the path it takes. In the next section, we will figure out and extract the most essential parts of evolution, and then see how to create the same form of optimization in non biological systems.

The evolutionary process discussed in the previous section needs the following elements to work: We have to have a population of organisms/agents, and some


way of representing their genome. We need a way to represent their genome be- cause the process of evolution requires variation, it requires for a process of creat- ing offspring that have a chance of being different from their parents. The way we create mutated offspring is by applying mutation operators to the genomes of their parents, by for example first cloning the parent genome and then mutating that cloned genome. Another element that is required is the process of selection. In bi- ological evolution, the most fit organisms are the ones that create the most off- spring, indeed the ability to create offspring (which requires the organism to have enough resources, and be able to successfully compete against other organisms) is that which defines the organism's fitness.

But when we evolve programs, they are not always applied to the problem where they have control over their replication processes. Neither do we want to add any more complexity to the agents, the complexity which would be needed for their ability to decide on when and how to replicate. Finally, each agent will not know his own fitness with relation to the rest, thus each agent will not know how many offspring it should create if any, and we would not want to leave such deci- sions in that agent's “hands ” in the first place, since we want to be able to guide the evolution ourselves. Thus, the selection process is a method of choosing those agents, or those genomes, that are more fit than others.

If you think about the genotype being in a search space, and the phenotype be- ing in the solution space, with the evolutionary algorithm trying out genotypes in the search space, and mapping each genotype to a phenotype in the solution space, to see whether it is, or how close it is, to the right solution, then an evolutionary algorithm is simply an advanced search algorithm. It is an optimization algorithm, where the term optimization simply means, searching for the best. The algorithm conducts a search for solutions, trying out different genotypes, and checking if they represent better solutions or not, as shown in Fig-3.1.

Fig. 3.1 Evolution as a search algorithm, from search space to solution space.

Thus, to sum it up, an evolutionary process requires a population of agents and a way to represent their genotypes in the search space. It requires reproduction, a way for parents to create offspring by creating new variations of their own geno- types. It requires a mapping system, to convert the genotype (in the case of biological organisms, DNA & RNA) in the search space, to phenotypes (the actual organism, its morphology, and its behavior) in the solution space. Finally, it requires a selec- tion method, a way to choose the fit agents, a way to discern the fit from the unfit. Thus, evolution is a directed parallel search, directed by the principle of survival of the fittest, however that fitness is defined.

In the biological evolution, there was a point before the first replicator emerged, when various atoms and molecules simply collided with each other ran- domly... until one of those combinations was a replicator. But in artificial evolu- tion, we do not need to have that moment, we can create the initial population of simple organisms, and decide how their genotypes are represented, so that they can be mutated, mapped to their phenotypic forms, and evaluated for fitness. Based on all these essentials, the evolutionary process can be represented with the following set of steps:

1. Create a seed population of simple organisms. The organisms must have both, genotypic and phenotypic representations, and a function capable of mapping between the two. The genotype must have a form such that mutation operators can be applied to it, thus allowing the population of organisms to evolve.

2. Create a fitness function, a function that can give each agent in the population a fitness score, such that the agents can be compared against one another, to see which is more fit.

3. DO:

1.

2.

3.

4.

5.

Evaluate each organism's fitness in the population. Give each organism a fitness score using the fitness function, based on that organism's perfor- mance.

Choose the fit organisms within the population.

Remove the unfit organisms from the population.

Create offspring from the fit agent genotypes. The offspring are variations on the genotype of their fit parents. The offspring can be created through the process of cross over, mutation, or both. For example an offspring can be created by first creating a clone of a fit parent, and then applying vari- ous mutation operators to the clone, and thus generating a mutant of the fit parent.

Compose the new population from the offspring and their parents (this method is called an elitist selection, because the fit organisms of each gen- eration always survive to the next generation).

4. UNTIL: Termination condition is reached. Where that condition can be some goal fitness score, or some maximum amount of computational power or time expanded on the problem to which the evolutionary algorithm is being applied.

Now when I say evaluate each organism's fitness, I of course refer to the phe- notype of the organism. The genotype is the representation, it is that to which we apply mutation operators, it is that string or program from which the actual pheno- type of the organism can be derived. The genotype is a mutatable encoding of the organism, a point in the search space of all possible genotypes. The phenotype is the actual characteristics of the organism, its behavior and morphological proper- ties, a point on the solution space, of all possible solutions.

For example, in the case of biological organisms, our DNA is our genotype. Evolution occurs through the mutation of our genotype. We, you and I, are pheno- types mapped from our genotypes. A genotype is converted to the phenotype by using some type of mapping function. In the case of biological organisms, it is the biological process of translating from genes to proteins, and the composition of those proteins into a cohesive and synchronized organism. It is the phenotype whose fitness is judged, it is the phenotype that interacts directly with the envi- ronment it inhabits. And if it is fit enough to survive, and thus to create offspring, it passes on its genome, sometimes in a mutated form so as to explore other points in the search space, and thus giving its offspring a chance to perhaps find a better position in the solution space and achieve an even greater fitness than its parent.

                • Note********

Evolution is the process undertaken by a population of agents. What is evolving is the popula- tion rather than an individual. It is the population as a whole that is changing, through the gen- eration of new mutant individuals within the population, some of which have the traits more fit for the environment in which they exist. In a sense, an individual within the population repre- sents the current state that the population achieved through evolution at that point in its evolu- tionary history. When it is clear from the content, I will at times state “as the agent evolves … ”, because we can look at any given agent and concentrate only on that agent's evolutionary path by back-tracing it through its ancestors, or its earlier forms. Or if we concentrate only on a genotype's future offspring which are more fit than it is, and thus trace forward the path of the genotype through its surviving offspring, the term “as the agent evolves … ” also applies.

Though we now know the essentials of evolution, and even the evolutionary algorithm and the necessary parts that allow it to create variation, and explore genotypic variations in search for a more fit one, there are still the following two linked questions we need to answer before we can put this knowledge to use: How do we represent these genotypes and phenotypes in software? and how do we for- mulate the problem we're working on, in evolutionary terms?

In the next section we will discuss the task of problem formulation, and geno- type/phenotype representation.

3.3 Formulating a Given Problem in Evolutionary Terms

Let us first set out in concrete terms the problem and solution that biological evolution solves in the real world.

Problem: If you are a replicator, how do you create as many copies of yourself as possible in a dynamic and hazardous environment?

Solution: Put the replicator inside a survival machine, an adaptable system able to create copies of the replicator with its own similar copy of a survival ma- chine that can deal with the hardships the environment produces. The replicator is the gene, as noted by Richard Dawkins.

The way biological evolution solved the problem is by creating trillions of so- lutions, and then evaluating how good those solutions were. The way to evaluate the solution is to simply let it interact with the hazardous environment, those solu- tions that are able to replicate, have a high fitness (dependent on how many copies it was able to make). The replicator offspring will differ slightly from the parent, and in this way new solutions are explored again and again, in a hill climbing fashion. Hill climbing in the sense that if an inferior solution to the current one is found, it usually does not do well when compared (when it competes) to the cur- rent solutions, and so it does not survive. There are millions of possible solutions, ranging from single celled survival machines without any type of adaptive capa- bilities, to multicellular survival machines that have some embedded information about the environment, a predisposition of learning certain essential patterns with- in the environment, and an organ that can remember and adapt to new challenges within the environment.

Thus, the way the problem is solved is: Take the problem, and let it evaluate the fitness of the different solutions. Create multiple solutions, and let the problem decide which are good and which are bad. Then take the good ones, create new variations of those good solutions, and then repeat this cycle.

We see that the answer depends heavily on the problem to which we wish to evolve a solution. Let us now take a non biological problem, and formulate it in such a way that we can apply an evolutionary algorithm to it, and evolve a solu- tion. For example, let's say that we have the following problem: We are given some unknown graph, as shown in Fig-3.2 , and we want to find a mathematical function that represents this graph. How can this problem be formulated in evolu- tionary terms?

Fig. 3.2 A graph, the problem to which a solution, a function which describes it, needs to be evolved.


An organism in this problem is some function f(x). The function f(x) is the genotype of the individual organism. The solution space for this problem is one of all possible graphs, where we are searching for the graph shown in the figure above. Thus we map the genotype f(x), to its phenotype, the actual graph of that function. Finally, the fitness is decided by the solution space, and is based on how close the graph represented by the organism is to the one in the above figure.

Thus, to evolve a solution to the problem, we would first need to create a seed population of very simple agents, where each agent is simply a function. We then compare the graph of each of the functions to the above graph, calculating the Cartesian distance between each point on the given graph, and the coordinate on the graph the agent function represents. The fitness is then 1/Cartesian_Distance. The smaller the total Cartesian distance, the greater the fitness of the agent, and the more offspring it should be allowed to create. The offspring are mutated ver- sions of the parent equation; we compose them by taking the fit function, and then adding, subtracting, multiplying, and dividing it by other primitive functions and constants. Once the low fitness agents are removed and replaced by offspring agents, we re-evaluate the whole population again, as per the evolutionary algo- rithm discussed in the previous section. Eventually, by chance, there will be off- spring agents, offspring functions, which represent graphs that are closer to the given graph than their parents. Thus, through evolution, eventually we will find better and better approximation to the given graph. Eventually, we will evolve a solution, a function whose phenotype, the graph, is very close to, or even exactly as, the above given graph.

The above sounds like it should work, and indeed it does work, it is using an evolutionary approach. We have simply replaced the biological organism's DNA by a string f(x) representing the function, and the organism's phenotype by the function's graph. Each organism can have a higher or lower fitness, and that fit- ness is evaluated based on its performance in the environment, how close the graph produced by f(x) is to the given graph. Although in this case, unlike in the biological system, the creation of offspring is done not by the organism/agent it- self, but by the researcher, or some other outside system created by the researcher. Nevertheless, the offspring creation process is still dependent on the fitness of the organism, and thus the main features of evolution are retained. Finally, as men- tioned, the environment in this problem is not a three dimensional world, but in- stead is a scape . The environment is the given graph in a sense, and the way to survive in this environment is to possess a graph that is as close to the given graph as possible. The only thing we have not yet decided on is how to represent the genotypes of these agents.

We need to create a genotype to represent these agents such that it makes it easy for us to apply mutation operators to them, such that it makes it easy for us to create mutant offspring. The phenotype on the other hand is what is getting evalu- ated. The phenotype can be the same thing as the genotype, and depending on the


problem, that might even be optimal. For example, the problem might be such that the goal is to create a particular genotype, in which case we could then evaluate the fitness of the genotype directly by comparing it to the provided goal genotype. In this problem though, the phenotype is the actual graph that the function paints, and we do not know what the genotype of the goal phenotype is. Thus, our agent's genotype is the function f(x), and the phenotype of our agent is the graph that f(x) represents, a graph which can be compared to the goal graph.

One of the representations for a computer program, or a function, which yields easily to mutation and variation, is through the use of trees, as shown in Fig-3.3 . This is the representation used in Genetic Programming popularized by John Koza [7,8,9,10].

Fig. 3.3 Function as a tree structure, the genetic programming way.

In such a representation, the non-leaf nodes represent mathematical operators, and the leaf nodes represent inputs, whether those inputs be variables or constants. We can then evaluate this tree structure by following the paths from the leafs to the root, executing the mathematical operators on the input values.

The leaf nodes can be elements like: 0, 1, Pi, X(i) input. The non leaf nodes are the mathematical functions, and other types of programs. When the problem posed is mathematical in nature, we can use the following nodes: tanh, +,-, %, /, *, sin, cos, tan … basically any standard mathematical operators, and trigonometric func- tions that can be used to form various functions.

This representation makes it very easy to create mutant offspring. For example we can mutate agents by simply adding new nodes to the tree, in random locations of that tree. Or we can take two or more trees and swap branches between them,


thus creating an offspring not through mutation but through crossover. The follow- ing figure shows the agents with tree like genotypes, the functions that those geno- types evaluate to, and the mutated genotypes produced through the application of mutation operators and cross-over.

Fig. 3.4 Evaluating and mutating tree encoded genotypes.

Thus, having now decided on the genotype representation, on the phenotype representation, and on the fitness function (graph similarity through Cartesian dis- tance between the agent graph and the goal graph), we can employ the evolution- ary process to evolve a solution to this problem. We would start with a seed popu- lation of very simple functions, whose genotypes are represented as trees. We would then map each of the genotypes to their phenotypes, the graphs they repre- sent in two dimensional space. For each graph, we would see how close it is to the wanted graph, and using the fitness function give each of the individuals its fitness score. Based on these fitness scores we could then choose the top 50% of the pop- ulation, the most fit of the population, and discard the bottom 50%. We then allow each of these fit individuals to create a single offspring. The offspring can be mu- tated clones of their parents, or some crossover between the fittest agents in the population. Once the offspring are created, we form a new population composed of the fit parents and their offspring, this new population represents the next gen- eration. The new population is then once again evaluated for fitness... The evolu- tionary loop continues.


This could continue on and on, and through evolution, through the search con- ducted in the solution space, more and more fit individuals would arise, those whose graphs, whose phenotypes, are closer to the wanted graph.

Using these same set of steps we could apply the evolutionary algorithms to other problems. For example we could create a genotype encoding for antennas, where the fitness of those antennas is their performance, signal clarity of their re- ception. In this way we could then evolve new antennas, better at catching some particular signal. Or we could create an encoding for circuits, after all, circuits are simply graphs, and graphs are trees with more than one root, as shown in Fig-3.5 .

Fig. 3.5 Multi-rooted trees as graphs, and graphs as circuits.

The fitness function could then be the lowest number of logic gates or transistors, while maintaining the same functionality as the original circuit. Then, through evolution, we could evolve more efficient circuits, more efficient CPUs...

We are then only limited by our imagination, and coming up with how to en- code the genotypes of something we wish to evolve. There are numerous varia- tions and “sub categories ” of evolutionary algorithms. In the next section we will briefly discuss them, and how they differ from one another.

3.4 The Different Flavors of Evolutionary Algorithms

You already know how to apply evolutionary computation to various problems. You need only create a genotype encoding with a mapping to phenotype, a fitness


function, a selection algorithm, and the mutation operators you wish to apply dur- ing the reproduction/offspring-creation phase to create variation in the offspring. The evolutionary process takes care of the rest. Indeed it is so robust, that even a poorly formulated approach will still usually lead to a good enough solution.

Granted, some approaches do get stuck in local optima. Being deceived by the environment, by the solution space, and lead to local optimal solutions, never being able to jump out of that area using the given mutation operators. Thus there are all kinds of extensions, and advancements made to the simple formulation we've covered. The increasing number of phases during evolution, the specialized muta- tion operators to give a chance for the offspring to jump far enough from its parent in the solution space such that it can explore other areas, which might lead to the global optima... There are approaches that divide the population into species, with each species only competing with others of its kind, and thus not letting any one particular highly fit organism to take over. An idea similar to niche finding in evo- lutionary biology. There are variations on the original algorithm that make fitness functions take into account how different the genotype, or even the phenotype, of the agent is from everyone else in the population, giving extra fitness points to agents that are different from others. Such advanced evolutionary computation al- gorithms can dramatically improve the speed at which solutions are found, and the final fitness of the solution.

For the sake of completeness, in this section we briefly discuss the four most commonly known variations of evolutionary computation (EC) flavors. The four most common EC flavors are: Genetic Algorithms, Genetic Programming, Evolu- tionary Strategies, and the Evolutionary Algorithms.

3.4.1 Genetic Algorithms

Genetic algorithms (GA) is one of the most well known approaches to evolu- tionary computation. Although computer simulation of evolution was being ex- plored as early as 1954, they became popularized in early 1970s, due to John Hol- land's book Adaptation in Natural and Artificial Systems [11]. In his attempt to discuss natural evolutionary processes, and emulate the same in artificial systems, his algorithm implements the standard features of evolution, selection, crossover, mutation, and survival of the fittest.

The algorithm primarily relies on crossover. Putting the fit individuals into a “matting pool ” from which two random fit parents are then drawn, and their geno- type is then used to create an offspring. The offspring is created by taking the gen- otype of individual A, choosing a random sequence slice of it (if represented as a string), of random length, then doing the same with individual B, and then finally creating a new offspring C by putting the two sequence slices from A and B to-


gether. Mutation is also used for the purpose of creating offspring, but is usually only used lightly, and only for the purpose of maintaining some minimal level of diversity within the population.

A simple example of evolving individuals through crossover, where the geno- type is represented as a binary string, and the phenotype as simply the translation of the bits 0 and 1 to colors green and white respectively, is shown in Fig-3.6 .

Fig. 3.6 Evolving an individual that can blend in with a green background, with offspring creation through crossover.

In the figure above we use GA to evolve an individual capable of blending into its background, which we have decided to be green. This is somewhat similar to the story of light and dark moths [12]. The dark moths had an advantage over their lighter variants due to the background on which the moths would sit being dark, letting the darker moths blend into their background more effectively than their lighter cousins. Eventually most of the moths in the location in question became dark, as the lighter ones stood out in their environment, got eaten by predators, and thus on average produced less offspring. In this simple setup, the genotype is repre- sented as a binary string. The phenotype is created by mapping 0 to white, and 1 to green. If the fitness function is then based on how well an individual is able to blend into a green background, we would then assume that green individuals are more fit in such environments, than white individuals which would stand out. In this example we use crossover. The population is composed of 4 individuals, and each generation we choose 2 of the 4 individuals that are most fit, and use those fit individuals to create 2 offspring for the next generation. The offspring are created by taking parent A and cutting its genotype in two, then taking B and cutting its genotype in two, finally, we create the two offspring by connecting the random half from A to a half from B, creating the two new offspring. As can be seen from the example, eventually, when generation 3 is reached, one of the offspring is completely green. At this point we would have reached the termination condition, having achieved the maximum fitness level within the green environment.


Genetic algorithm systems also use mutations, and indeed this same problem can be just as easily solved through mutation alone, as shown in the next figure.

Fig. 3.7 Evolving an individual that can blend into a green background, through mutation alone.

The main drawback with old GA approaches is that they primarily utilized stat- ic sized genotypes. The genotypes could not grow or shrink, if you had started with a genotype string of size 4, that is all you would have to work with. Later on this problem was eliminated by simply adding a few extra mutation operators that could add and subtract sequences from the string encoded genotype, resulting in a dynamically sized genotype.

3.4.2 Genetic Programming

Genetic programming (GP) is a specialized type of GA that deals with not string encoded genotypes or chromosomes, but tree based programs. GP evolves programs represented as trees, similar to the types we developed in Section-3.2 when evolving a solution for the graph problem. GP also comes with a specialized set of mutation operators, like branch swapping between two individuals to create an offspring through crossover. Node mutation, node addition, and other types of mutation operators that the researcher decides to implement. Also, unlike the orig- inal GA, the genotypes in GP are of variable length, GP can expand and grow programs/trees of any size. Of course this can also be done by modifying the GA approach as we discussed, by adding mutation operators to grow and shrink the string encoded genomes.

GP was originally introduced by Cramer [13], but popularized by Koza [1992]. John Koza has also further modified the genotypes and specialized GP to be ap- plied in evolution of new materials, antennas, and other structures [14,15]. Having access to a large cluster of machines, a cluster which he dubbed the “ Invention Machine ” [16], John Koza is able to run parallel GP algorithms with large popula- tions, evolving new programs, and applying the approach to the evolution of pa- tentable materials and hardware. This invention machine has produced numerous


artifacts for which patents were granted. Thus when used to its full potential, evo- lution is easily able to generate results rivaling, or on par and competitive with, the innovations produced by human inventors.

3.4.3 Evolutionary Strategies

Thus far we spoke of mutation operators, and said that there is a certain chance that so and so mutation is applied, or so and so number of offspring are created from a fit parent... In actual evolutionary systems, we have to specify these proba- bilities. For example, let us say we have a GP system which creates offspring purely through the application of mutation operators. For such a system, we can specify the following evolutionary parameters:

 Only the top X% of the population gets to create offspring, where X = 50.

 Each fit individual creates Y number of offspring, where Y is proportional to the individual's fitness in comparison to others, and the available amount of free space in the population, based on the number of individuals removed due to being unfit, where: FreeSpace% = 100% - X% , and thus the number of off- spring that can be created during the generation to fill up the provided free space: TotalNumberOfNewOffspring = (FreeSpace%)*PopulationSizeLimit .

 Offspring are created by first cloning the fit parent, and then applying K ran- domly chosen mutation operators from the following list: [Add_New_Node, Remove_Node, Swap_Node] . Where each mutation operator has a probability M(i) of being chosen.

 The program representing the non leaf node is chosen randomly from the fol- lowing list: [*, /, %, +, -, sin, cos, tan, tanh], each being chosen with a proba- bility of N(k) .

 The program representing the leaf node is chosen randomly from the following list: [Input(n), 1, 0, Pi] , each being chosen with a probability of L(I).

As you can see, there are lots of different parameters that need to be set up be- fore we can run the evolutionary algorithm. The parameters specify the mutation probabilities, selection probabilities... Evolutionary Strategies (ES), is another var- iation on the simple GA approach, which evolves not only the genotype, but also these evolutionary parameters, the evolutionary strategy itself. Thus, as each or- ganism evolves, the probabilities with which mutation operators are applied, and various other evolutionary strategy parameters, are also mutated and evolved. The evolutionary algorithm itself, can change over time for every agent. Thus, we end up not just with a population of genetically different individuals, but genetically different individuals which also evolve at different rates, and use different proba- bilities for choosing various mutation operators during their reproduc- tion/offspring-creation phase.


Evolutionary strategies [17,18,19] were originally proposed and introduced by Schwefel, and then continued being researched by Rechenberg a decade later. There are advantages to this approach. The evolutionary process itself evolves as well, which is a phenomenon that exists in biological evolution. After all, we are all susceptible to mutation rates at different levels. Our DNA differs in robustness from one another, though to a low degree. So there are certainly features in the bi- ological world where evolutionary strategy changes, if not due to the way DNA itself is structured, then at least due to the fact that different environments on this planet have different levels of exposure to mutagens, radiation... Which means that or- ganisms living in different environments, will have different number of mutation operators applied to their genotype, and at different intensities. The intensities can be based on the level of exposure and presence of mutagens in the particular envi- ronmental niches that the organisms inhabit.

3.4.4 Evolutionary Programming

Evolutionary Programming (EP) is a search algorithm closely related to ES, but developed earlier, independently, and specialized to the evolution of state transi- tion tables for finite state machines (FSM). Developed by Lawerence Fogel [20] in the 1960s during the rise of Artificial Intelligence popularity, and developed for the purpose of evolving Finite State Machines (FSM) with ability to predict next actions and environmental states, this particular search algorithm eventually be- came less and less used, until it finally fell into obscurity. In the 1980s though, it again gained popularity as it was further advanced and reintroduced to the compu- tational intelligence community by David Fogel.

This is another variation and specialization of evolutionary algorithms. Like genetic programming and genetic algorithms, this approach simply specializes in evolving a different type of genotype. Instead of applying the evolutionary algo- rithm to the evolution of strings, or tree structure encoded programs, it is instead used to evolve FSM based systems, with an inclusion of the self adaptability used by evolutionary strategies.

3.5 A Few Words on Memetic Computing

Memetic Algorithms (MA) sometimes also referred to as Hybrid Algorithms (HA), are evolutionary algorithms that separate the evolutionary process into two phases, global search and local search. The global search part of the evolutionary algorithm might be the type that produces offspring that are spread far and wide within the solution space by creating those offspring through the use of powerful


and high impact mutation operators. The local search would then allow each such individual in the population to self optimize, to tune in further and explore its local space/optima and find the best position on it. These types of algorithms have been found to be extremely efficient, more so than the standard, single phase algorithms [21,22].

Let us consider an example where we wish to find/evolve a single small trigo- nometric function as shown in Fig-3.8 , using a memetic algorithm. Unlike the problem discussed in Section-3.3 though, this one is a very simple function, and we will use a very short, static string based genotype encoding. This simple geno- type encoding will be as follows: A+B*F(X*C+D)^E , where A, B, C, D, and E are variables, and F is a function. The variables span the entire number line, whereas F is a function that can be chosen from the following trigonometric function list: [Sin, Cos, Tan, Tanh] .

Fig. 3.8 Evolving a function using a memetic algorithm to match the goal graph.

Since the evolution is separated into two phases in memetic algorithms, we will also have two different classes of mutation operators, one for global search, and one for local search. The global search would then search through all the types of


trigonometric functions. The list composing the mutation operators (in this case just a single one) for the global search, is as follows: [ChangeF] . The global search mutation operator list is composed of a single mutation operator that changes the individual's trigonometric function from the one it is currently using to a new one in the trigonometric function list. Each trigonometric function is very different from every other in its nature, and thus the agents/graphs produced from the different trigonometric functions will be very different from one another. The local search is composed of the following mutation operator: [Perterb_Value]. Where the mutation operator is applied to one of the values in the genotype: [A, B, C, D, E] , adding a random value to one of these variables to see if the new phe- notype, the graph, is a better match to the goal graph. The local search, the explo- ration of the local solution space by tuning in the values of the function, allows us to explore each local space of the trigonometric function used, before deciding on its final fitness score. As can be seen, while the change in the trigonometric func- tion drastically changes the phenotype, the mutation and perturbation of the A, B, C, D, or E variables, results in the tuning and small/local changes of the same phenotype.

When using standard evolutionary algorithms, sometimes evolution might pro- duce the perfect genotype, it's just that a few of the parameters need to take on different values before the phenotype can truly shine. But because evolutionary algorithm is just a “one-off ” type of deal, if this genotype does not have the right values right there and then, it is given a low fitness score, and then most likely discarded off. Think of it this way, there might be a position on the solution space where there is a high fitness position, but the evolutionary algorithm created a so- lution that is right at the bottom of the this fitness hill. It would be more effective for each solution to search locally, optimize itself, to truly achieve the potential that its global parameters possess. This is graphically represented in Fig-3.8 , which again shows the mapping from the genotype/search space to the solution space, with the solution space also showing the various fitness scores.

Fig. 3.9 A global search, and a local search. Moving up the fitness hill.


For example, walking only on two legs requires a significant amount of bal- ance, it leaves our hands free to use tools. What if a genotype specifying a pair of legs came into existence, but the part of the brain that deals with balance was still solving a problem for a quadrupedal system? The phenotype would be unable to balance, it would thus be given a poor fitness score, and would disappear from the population. For the bipedal system to work, it would have to be reinvented by evo- lution with the right balancing neurocontroller at the same time, before it would be given its true fitness. Solving two problems at the same time, like rolling two dice to try and hit the same number at the same time, has a much lower probability than rolling a single die, and then re-rolling the second die until the same number is hit again. The probability of evolving the right set of parameters in the genotype to yield a balancing neurocontroller for a bipedal form might be very low. The prob- ability of evolving bipedal morphology might also be very low. Thus, evolving both at the same time, to be just right to work together, might have an especially low probability of occurring. On the other hand, if we have came across the geno- type for bipedal morphology, and then tune in the various parameters to achieve the best performance with this morphology and thus produce a neurocontroller to make this morphology achieve its full potential, is much easier (has a higher prob- ability of being achieved through pure stochastic processes). That is the pattern of operation when it comes to memetic algorithms, separating the global and local searches, however they are defined for a particular genome encoding.

In a sense, a memetic algorithm gives each individual's genotype in the popula- tion a chance to demonstrate its best possible fitness. It allows each individual to show what is possible using its general genotypical properties, given that we can tune its “local ” parameters to work best with its more globally affecting genotypic structures.

The final example I'll give for memetic computing, and this is us jumping slightly ahead to the subject that will be discussed in detail in the next chapter, is one of evolving neural network systems. In neural networks in particular, as we saw in Chapter 2, we can make a clear distinction between the neural network to- pologies, the manner in which the neurons are interconnected, and the synaptic weights those neurons possess. The global search part could then be done through the mutation operators which modify the NN's topology itself. While the local search can then be accomplished through the tuning of synaptic weights. This would provide us with scenarios where we could, through mutation, add and con- nect a few new neurons to an existing NN. But of course simply connecting new neurons to a NN, each with its own randomly selected activation function and synaptic weights, will disrupt that NN system, and most likely make it less fit. So if we then take the time during a separate phase to see if we can tune the synaptic weights of these new neurons, we can try to make these newly added neurons work complementary with the already existing and fit NN system.


Through this approach, we give each NN topology a chance to show its true fitness. We give it time to adjust the synaptic weights, and are then able to judge the actual topological structure, rather than judging how a random topology works with a random set of synaptic weights. This two phase memetic approach has proven to work exceptionally well, and is the method being popularized by the to- pology and weight evolving systems like DXNN [23] and EANT [24,25].

Are all these systems different: GA, GP, ES, EP, MA? Or are they simply dif- ferent sides of the same coin? The concluding section of this chapter discusses just that, before we take our first step and dive into neuroevolution.

3.6 The Different Sides of the Same Coin

Today there are advanced versions of GA, GP, ES, EP, and MA. The lines be- tween these flavors of evolutionary computation are blurred to the point where it's difficult to pinpoint where one system ends and another begins. Indeed, they are all technically the same. After all, GP is just GA that operates on a tree algorithm. And if we add the ability to also mutate evolutionary strategy parameters, then we've just went from GA to ES. EP is just GA specialized for finite state ma- chines, and Memetic Computing (MC) is simply a variation on the standard evolu- tionary computing, which splits the local and global search into separate phases...

How we represent the genotype and phenotype, and how we map from geno- type to phenotype, is entirely up to us. Whether we also wish to include the muta- tion of the evolutionary strategy, the mutation operator probabilities and the par- ticular list of mutation operators; all of this is up to the researcher to choose. Thus, all of this is simply evolutionary computation.

For example, there are GP approaches that evolve not trees but strings, and the- se strings encode operations taken by a computer, a program in linear form. This type of approach is called linear genetic programming. On the other hand, if in- stead of a tree we use a graph, the approach is called Cartesian GP. A Cartesian GP can use any type of function or program for the nodes. If we decide that each of the nodes should first sum up the incoming signals and then apply the tanh function to the sum, our Cartesian GP system becomes a neuroevolutionary sys- tem. Yet in a neuroevolutionary system, the neurons can also use tanh, sin, cos, or really any other function as well. Learning networks which use different kinds of activation functions, not just tanh, are referred to as Universal Learning Networks rather than Neural Networks ...

We can use GA to evolve building architectures, by for example encoding in the string the position of the various elements of the building, and then evolving a population of such genotypes for some particular fitness, for some particular size and aerodynamic properties for example. We could also represent the genotype of

3.7 References

the building as a tree, or as a graph, to make it easier for the GA to apply certain mutation operators... We could also split the single phase of reproduction where offspring are created through mutation, into two phases. In one phase the offspring are created through the application of large scale, high intensity mutations. Then during another phase the offspring could use something like hill climbing, a local search algorithm, to tune in the various parameters of their phenotype, looking around in close proximity to their position on the solution space. At this point our GA would be called MA...

As you can see, everything is pretty much blurred together, a few modifications to your algorithm, modifications that simply make it easier to encode the genotype for some particular project you are working on, and there will be those who will start calling your evolutionary algorithm approach by a new name. These different names simply make it a bit easier to describe quickly some of the features of your evolutionary approach. But in reality, when you continue to advance any evolu- tionary algorithm, trying to make it more agile, more robust, applicable to more problems... you will continue adding more and more features, including graph en- coding, linear encoding, multi phase evolution, evolutionary parameter adaptation, the ability to use functions like tanh, logical operators like XOR, AND... and pro- grams like IF, While... for the nodes within the genotype... An advanced evolu- tionary system, one that can be applied to any problem and self adapt to it, seeing on its own which features, which encoding, and which evolutionary strategy is best for it, will most likely need to have all of these features.

In conclusion, you can encode the genotype of a system using any type of data structure you want, as long as you can also create the mutation operators to oper- ate on that data structure, and a mapper that can map the genotype from that data structure to its phenotype. It does not matter what that data structure is, or what set of mutation operators you're using, whether you're also using crossover or not, and whether your evolutionary parameters themselves adapt or not. It's all the same evolutionary system, tailored to your problem, by you.

The next chapter introduces neuroevolution, the evolutionary process applied to the evolution of neural networks.


[1] Cracraft J, Donoghue MJ (2004) Assembling the Tree of Life J. Cracraft and M. J. Do- noghue, eds. (Oxford University Press), ISBN 0195172345.

[2] Lewontin RC (1970) The Units of Selection. Annual Review of Ecology and Systematics 1, 1-18.

[3] Kimura M (1991) The Neutral Theory of Molecular Evolution: A Review of Recent Evi- dence. Japan Journal of Genetics 66, 367-386.

[4] Tjivikua T, Ballester P, Rebek J (1990) Self-Replicating System. Journal of the American Chemical Society 112, 1249-1250.


[5] Graur D, Li WH (2000) Fundamentals of Molecular Evolution D. Graur and W.-H. Li, eds. (Sinauer Associates), ISBN 0878932666.

[6] Dawkins R (1976) The Selfish Gene. (Oxford University Press), ISBN 0192860925.

[7] Luke S, Hohn C, Farris J, Jackson G, Hendler J (1997) Co-evolving Soccer Softbot Team Coordination with Genetic Programming. Proceedings of the First International Workshop on RoboCup at the International Joint Conference on Artificial Intelligence 1395: 398-411.

[8] Koza JR (1992) Genetic Programming: On the Programming of Computers by Means of Nat- ural Selection. (MIT Press), ISBN 0262111705.

[9] Koza JR (1994) Genetic Programming II: Automatic Discovery of Reusable Programs. (MIT Press), ISBN 0262111896.

[10] Koza JR et al. (1998) Genetic Programming. Morgan Kaufmann Publishers. ISBN 1558605487.

[11] Holland JH (1975) Adaptation in Natural and Artificial Systems J. H. Holland, ed. (Univer- sity of Michigan Press).

[12] Mike M (1998) Melanism: Evolution In Action. (Oxford University Press).

[13] Cramer NL (1985) A Representation for the Adaptive Generation of Simple Sequential Pro- grams. In Proceedings of an International Conference on Genetic Algorithms and the Appli- cations, J. J. Grefenstette, ed. (Lawrence Erlbaum Associates), pp. 183-187.

[14] Koza JR, Bennett FH, Andre D, Keane MA (1999) Genetic Programming III: Darwinian In- vention and Problem Solving (Morgan Kaufmann), Springer. ISBN 1558605436.

[15] Koza JR, Keane MA, Streeter MJ, Mydlowec W, Yu J, Lanza G (2003) Genetic Program- ming: Routine Human-Competitive Machine Intelligence. (Kluwer Academic Publishers), Springer. ISBN 1402074468.

[16] Koza JR, Keane MA, Yu J, Bennett FH, Mydlowec W (2000) Automatic Creation of Hu- man-Competitive Programs and Controllers by Means of Genetic Programming. Genetic Programming and Evolvable Machines 1, 121-164.

[17] Hans S. (1974) Numerische Optimerung von Computer-Modellen. (PhD thesis).

[18] Back T, Hoffmeister F, Schwefel HP (1991) A Survey of Evolution Strategies. In Proceed- ings of the Fourth International Conference on Genetic Algorithms, L. B. Belew and R. K. Booker, eds. (Morgan Kaufmann), pp. 2-9.

[19] Auger A, Hansen N (2011) Theory of Evolution Strategies: a New Perspective. In Theory of Randomized Search Heuristics Foundations and Recent Developments, A. Auger and B. Doerr, eds. (World Scientific Publishing), pp. 289-325.

[20] Fogel LJ, Owens AJ, Walsh MJ (1966) Artificial Intelligence through Simulated Evolution L. J. Fogel, A. J. Owens, and M. J. Walsh, eds. (John Wiley & Sons).

[21] Moscato P (1989) On Evolution, Search, Optimization, Genetic Algorithms and Martial Arts: Towards memetic Algorithms. citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.27.9474 Ac- cessed March 20 2012

[22] Krasnogor, N. (1999). Coevolution of Genes and Memes in Memetic Algorithms. Proceed- ings of the 1999 Genetic And Evolutionary Computation Conference Workshop Program, 1999-1999.

[23] Sher GI (2010) DXNN Platform: The Shedding of Biological Inefficiencies. Neuron, 1-36. Available at: http://arxiv.org/abs/1011.6022.

[24] Kassahun Y, Sommer G (2005) Efficient Reinforcement Learning Through Evolutionary Acquisition of Neural Topologies. In Proceedings of the 13th European Symposium on Arti- ficial Neural Networks ESANN 2005 (ACM Press), pp. 259-266.

[25] Siebel NT, Sommer G (2007) Evolutionary Reinforcement Learning of Artificial Neural Networks. International Journal of Hybrid Intelligent Systems 4, 171-183.

Chapter 4 Introduction to Neuroevolutionary Methods

Abstract Neuroevolution is the machine learning approach through neural net- works and evolutionary computation. Before a neural network can do something useful, before it can learn, or be applied to some problem, its topology and the synaptic weights and other parameters of every neuron in the neural network must be set to just the right values to produce the final functional system. Both, the to- pology and the synaptic weights can be set using the evolutionary process. In this chapter we discuss what Neuroevolution is, what Topology and Weight Evolving Artificial Neural Network (TWEANN) systems are, and how they function. We also discuss how this highly advanced approach to computational intelligence can be implemented, and what some of the problems that the evolved neural network based agents can be applied to.

We have talked about how the genotype encoding, the mapping from genotype to phenotype, and the phenotypic representation of that genotype, is all completely up to us. We can apply the evolutionary process to any problem, just as long as we can come up with a good genotype encoding and a set of mutation operators that we can use to generate offspring from the fit individuals in the population.

In Chapter 2 we discussed neural networks, ways to optimize and train them (through back propagation for example), and how to imbue them with plasticity, such that they can truly learn and self organize, and adapt on their own, as biolog- ical neural systems do. Still though, the self organizing maps, and the Hebbian learning rule based plasticity imbued neurons, are not general. Self organizing maps (SOM) can only be applied to some specific problems, and when we are cre- ating a SOM, we don't really know what map density we should use, what param- eters we should use with the map, and what set of inputs to feed it with. The same goes for competitive neural networks. Finally, with regards to plasticity imbued neurons, though each neuron can adapt and learn, it is the neural network as a whole, all its various parameters, and the NN's topology that determines what the cognitive system does. Each neuron might have plasticity, but it only learns if eve- rything is synchronized in the neurocognitive computer, only if the parameters the neurons are using and the topology of the NN as a whole, all have the right set- tings to do something useful.

Thus, we still need a way to set up all these various parameters in the NN, and to be able to grow and interconnect vast neural networks composed of neurons which have plasticity. For complex problems, supervised learning like the back- propagation will not work, plus we need something to set the right topology and NN parameters, not just the synaptic weights of the neurons...

DOI 10.1007/978-1-4614- 4463 - 3_4 , © Springer Science+Business Media New York 2013


There is one optimization algorithm that can do all of this, that can optimize and grow neural network (NN) systems. That algorithm is of course, the evolu- tionary algorithm. In this chapter we discuss how we can combine neural networks and evolutionary computing into a single system: a neuroevolutionary system.

4.1 Neural Network Encoding Approaches and Mutation Operators

As we discussed, an evolutionary approach can be used with anything, as long as we can come up with its genotype representation, genotype encoding, and a set of mutation operators (programs that can change the genotype). Neural networks are graphs, which themselves are simply multi rooted trees. All that is necessary is for us to come up with how to represent the genotype, how to encode it, and a set of mutation operators that is flexible enough for the evolutionary process to be able to evolve any genotype A into a genotype B, given the chance to apply some set of these mutation operators in some order.

4.1.1 Neural Network Mutation Operators

What would be a good set of mutation operators for the evolution of neural network systems? Well, we know that we need a way to tune the weights of the neurons, thus we need some kind of mutation operator that can modify a synaptic weight. We also need the NN to be able to expand, grow, and have new neurons added to it. Thus, we need a mutation operator that adds new neurons. There are different ways of adding a new neuron: for example we can select an existing lay- er in the NN and then add a new neuron to that layer and connect it randomly from some randomly chosen neuron and to some randomly chosen neuron in the NN. Or, we can grab two neurons that are already connected, and then disconnect them and reconnect them through a newly created neuron. Finally, another mutation op- erator that could be of use is one that adds new synaptic connections. For this new synaptic connection establishing mutation operator, we choose a random neuron in the NN, and then connect it to another randomly selected neuron in the NN. These types of mutation operators are referred to as: “complexifying ”, they add to the ex- isting NN and make it more complex, they grow the NN. A few examples of these types of mutation operators being applied to a simple NN, are shown in Fig-4.1 .


Fig. 4.1 Complexifying mutation operators.

Sometimes it is also a good idea to be able to prune, or remove connections and neurons from a NN. After all, environments change, and a phenotypic feature that might have been useful at one point, might no longer be useful in the new envi- ronment. A way to simplify a NN, or a graph in general, can be useful in evolutionary computation and allow it to get from any type of graph A to a graph B, where graph B is a subgraph of A. Thus, a neuroevolutionary system should also have access to mutation operators that can randomly select and delete neurons in the NN, and randomly select and delete connections between neurons and other ele- ments in the NNs, as shown in Fig-4.2 .

Fig. 4.2 Deleterious mutation operators.


In Chapter-2 we also noted the distinction between NNs and NN systems, where the NN systems are NNs connected to the sensors and actuators from which and to which they receive and send signals respectively. Thus, the ability of evolu- tion to integrate and add new sensors and actuators to the NN system is of course a must. After all, though we might start the evolutionary process with a simple NN system, evolving a neurocontroller for a robot when that robot only has access to a camera for a sensor, and a set of motors connected to wheels for actuators, over time there might be new sensors and actuators that are built and become available to that robot. Thus, we want our neuroevolutionary system to have the mutation operators that can integrate and try out these new sensor and actuator systems with the evolving NN over time, to test out if the resulting NN becomes more robust and more fit when taking advantage of its new hardware. A diagram example of such evolutionary paths, and mutation operators applied to a NN, is shown in Fig-4.3 .

Fig. 4.3 Mutation operators that add new sensors and actuators.

The way we implement all these various mutation operators (MO) will of course depend on the genotype encoding we use, it will depend on how and what kind of data types we use to represent the NN system. MOs are a set of programs that operate on the genotype, to produce the mutation effects we've just discussed. In the following section we will briefly discuss a few ways in which we can en- code the NN genotypes.

4.1.2 Neural Network Genotype Encoding

There are any infinite number of ways that we can come up with to store/encode and represent neural network systems. They are after all graphs, and so we could choose any standard manner in which graphs are usually stored. We


could encode our NN systems using a string format, strings composed of tuples that contain the input and output node ids, and synaptic weights. A string encoding method is for example the method used in a neuroevolutionary system called NEAT [1]. An example of this genotype encoding method, and the small NN it maps to, is shown in Fig-4.4 .

Fig. 4.4 A string encoded neural network.

You can note that just by reading the string, it is not so easy to immediately form the image of what the phenotype, what the actual NN looks like based on its genome. The genotype is composed of two parts, a string of tuples that specify the node ids and node types, and a string of tuples that dictate the connections and synaptic weights between the neurons.

The mutation operators we would need to implement to work with this type of genome encoding would need to be able to add and delete such tuples. For exam- ple a mutation operator that adds a new connection between two nodes would first randomly choose the ids of two nodes from the node string, and then compose a tuple that specifies the link between these two nodes, and a random synaptic weight for the postsynaptic neuron of this connection. A synaptic link deleting mutation operator would simply remove any one of the tuples in the connection string. A mutation operator that adds a new node and connects it randomly to other nodes in the NN, would first simply create a new tuple in the node string, with its own unique id, and then add two tuples to the connection string. One tuple would specify the connection from some existing presynaptic node to the new node, and the other tuple would specify a synaptic connection from the new node to some other random postsynaptic existing node in the neural network. Two ex- amples of mutation operators being applied to this form of genotype encoding, and the resulting phenotypes, are shown in Fig-4.5 .


Fig. 4.5 Applying mutation operators to a string encoded neural network.

Another encoding method, which is relational database friendly, human reada- ble, and is thus much easier to reason about and change/mutate, is the one used by DXNN [2] neuroevolutionary system. The encoding simply specifies all the details of each neuron in its own tuple, where the simplified version of this type of tuple has the following format: {Id, Input_IdPs,Output_Ids}, where Input_IdPs is a list of tuples of the form: [{Input_Id,SynapticWeights}....], and where Output_Ids is a list of ids: [OutputId1,OutputId2...]. Each tuple is self contained, possessing all the information needed to define a processing element it represents, whether it be a neuron, a sensor, or an actuator. Each node representing tuple keeps track of the the Ids of the presynaptic elements and the synaptic weights associated with them, and a list of postsynaptic Ids. This genotype encoding makes it very easy for a human to read it, allowing the person to comfortably see which elements are con- nected together, and what the general topology of the NN system is. Finally, be- cause each node representing tuple has its own Id, this encoding also has the per- fect form for being stored in a relational database. An example of a neural network genotype encoded using this method, and the neural network system it maps to, is shown in Fig-4.6 .

You will notice in the following figure that in the case of the sensor and actua- tor, their ids are respectively as follows: {sensor_function_name,1} and {actua- tor_function_name,5}. The ids include not only unique identifiers (1 and 5), but also sensor and actuator function names. Like neurons, the sensor and actuator nodes can be represented as processes, and these nodes could then execute the sensor or actuator functions through the function names. If we are talking about a sensor node, then when it executes the function sensor_function_name , it would


produce sensory signals that it would then forward, by message passing, to the neurons it is connected to. If we are talking about an actuator, then the actuator would accumulate the incoming signals from the presynaptic neurons, and then use this composed vector as a parameter when executing the actua- tor_function_name function. It is a much more dynamic and flexible approach than the above demonstrated string encoding. The few extra bytes of space this tu- ple encoding method takes, is well paid for by the utility, flexibility, readability, scalability, and usability it provides. Today, even a laptop can easily hold over 16Gb of ram. Why make a system less understandable, and thus much more diffi- cult to work on, expand, and advance, by using a poorly formed encoding that lacks the agility, and does not provide an easy way to think about it? The simpler and more architecturally direct the encoding, the easier it is for the researcher to improve it, work with it, and use and apply it to new areas.

Fig. 4.6 A tuple based neural network genotype encoding.

This tuple encoded genotype makes no attempt at emulating a biological ge- nome. It takes full advantage of the much greater flexibility that software pro- vides. Since the genotype is very similar to how graphs are usually represented, the mutation operators are also very easy to develop and apply. For example, to create a synaptic connection between two neurons, the program simply chooses two random neurons A and B from the NN, and adds a synaptic link from neuron A to B by adding the id of B to A's output_ids list, and by adding the id and a new synaptic weight for A to neuron B's input_idps list. To add a new neuron to the genotype, the mutator program simply generates a new neuron tuple, with a new unique id C, and sets up a link between C and some randomly chosen presynaptic neuron A, and to some randomly chosen postsynaptic neuron B. These mutation operators are just graph operators. A few examples of mutation operators being applied to a tuple encoded NN, and the resulting phenotypes, are shown in Fig- 4.7 .

In Fig-4.7a I show the genotype of the initial NN system, and its phenotype. I then apply a mutation operator where I add a new neuron with id {neuron,6} to the initial NN system, the added parts of the genotype are presented in bold in Fig- 4.7b , with the phenotype shown on the right side of the updated genotype. Finally, in Fig-4.7c I apply to the initial genotype a mutation operator that removes a random


neuron from the NN, in this case this neuron's Id is {neuron,3} . The updated geno- type and the phenotype is then shown, with the removed parts of the genotype highlighted with red (looks as bold font in the black & white printed version).

Fig. 4.7 Mutation operators applied to a tuple encoded NN genotype, and the resulting phenotypes.

Undoubtedly there are many other ways to encode neural network systems. Back in the day when storage space and ram was limited, there were a number of encoding approaches that tried to minimize the amount of space used by such en- codings. But today, storage space and ram are no longer the limiting factors. Our current computational intelligence systems are only limited by our ability to rea- son about them, expand them, and improve them. Thus, we should strive to repre- sent the genotypes in manners that are not most space conservative, but in ways which makes them most easily accessible to being reasoned about. We should cre- ate the genotypes and phenotypes in a way that makes it easy to work with, to think about, to expand, and to improve.


When it comes to neural networks, tuple encoding is one such genotype encod- ing method, and as we will find out in the next chapter, Erlang gives us the ability to create a phenotypic representation that is just as easy to reason about and work with as its genotype, if not more so. But more importantly, the phenotype imple- mentation in Erlang is a direct mapping of the NN representation, making it un- precedentedly easy to work with, scale, and improve. Neural networks are vast graphs, and so we use an encoding approach that is prevalent in graph representa- tions [3], and one that yields to graph operations most easily. We can go through the genotype and quickly see what type of phenotype, what type of neural network structure that it represents. And because it is so easy to see it, to visualize it, it is easier for us to modify it, it is easier for us to, in the future, expand this encoding method, and to add to it as our knowledge of neuroevolutionary systems and com- putational intelligence increases.

The ability to easily visualize the system, gives us the ability to think about it more clearly and without us having to constantly map the NNs in our mind back and forth between the way which makes it easy to reason about, and the way in which they are actually implemented in some programming language that does not have the same architecture as the NN systems.

4.1.3 The Neural Network Phenotype

The phenotype is the actual, functional representation of the neural network. It is the system that can process the input signals, and based on those input signals produce output signals from the neurons in the output layer (those neurons which are connected to the actuators). The actual input signals that the NN acquires as input is based on the sensors that it uses to sense the environment, and the actions the NN system takes within the world is done through its actuators, which act up- on the world using the control signals coming from the presynaptic neurons. In bi- ological organisms, our DNA is our genotype, and our nervous system is part of our complete phenotype.

The mapping from genotype to phenotype can be direct, where the genotype specifies every part of the phenotype directly, or indirect [4], where the genotype has to go through some kind of developmental cycle, or use some other external information to construct the actual phenotype. We have seen the direct encoded neural networks in the previous subsection. The genotype directly represented the phenotype, every synaptic weight, every connection, and every neuron, was speci- fied within the tuple encoded and string encoded genotypes discussed. Thus, next we will briefly discuss what the indirect encoded NNs are.


In indirect encoding the resulting phenotype is not directly based on the geno- type, but is merely controlled or guided by it as it develops. The development pro- cess is the mapping of genotype to phenotype, sometimes through multiple stages, and where the phenotype can also be affected or produced by incorporating multi- ple features generated by stochastic processes and through the system's initial in- teraction with the environment in which it is developing, as shown in Fig-4.8 . In this manner, the phenotypes might also be somewhat environment specific, since their development can be influenced by th e environmental features during the mapping process.

Fig. 4.8 Environment affected development of a neural network.

The above figure shows two scenarios; in Fig-4.8a a NN is controlling an agent in an ALife environment where most of the agents around it are green, thus the NN created has all 5 of its sensors be green color sensing, such that the agent can better discern the various elements within its environment, and have a higher abil- ity to extract features in a world where most of the things are green. In Fig-4.8b , a neurocontroller is created in an area where there are agents and features in the en- vironment of all kinds of colors, and so the same neurocontroller as in Fig-4.8a

now has a different set of sensors, each concentrating on a different color, and thus allowing the agent to have the ability to discern between all the different colored agents within the environment. If we are to create an agent with an ability to see multiple colors in a green environment, its green resolution is too low, and it would not compete well in that niche where its color sensing abilities are not needed. If we are to create the green color specializing agent in an environment composed of agents of different colors, it would be blind to anyone that is not green, and thus not be able to compete with color discerning agents. But because the phenotype is not directly specified by the genotype, and because as the geno- type is developing/being-mapped to a phenotype, it is being affected by the chem-


istry of the environment, the phenotype is specialized for the environment in which it is born. If these organisms were biological, and there were two environ- ments, an underground tunnel environment, and a color filled jungle environment, it would be advantageous for the organism born in the tunnel environment to start off with sonar sensors, but color sensors when born in the jungle environment.

Another type of indirect encoding is shown in Fig-4.9 . Here it is the particular mapping and the NN implementation that defines what the resulting phenotype is, and how it behaves. The genotype shown is a composition of a tuple encoded NN system shown in the previous subsection, with an extra third and fourth element in the NN. The third element is a list: [3,5,2] , and the fourth element is the tuple: {SensorList,ActuatorList} . What is interesting here is that it is the third list, [3,5,2] that defines the actual sensory signal processing, and actuator signal producing, NN. The [3,5,2] list defines a neural substrate composed of 3 layers, with 3 neurodes in the first, 5 in the second, and 2 in the third. Furthermore, it is implicit that this neural substrate is two dimensional, and each neurode in this substrate is given a Cartesian coordinate based on its position in the layer, and its layer's posi- tion in the substrate. The program that constructs the phenotype, spaces out these layers equidistantly from each other, with the first layer of density 3 positioned at Y = -1, the second at Y =0, and the last at Y = 1 end.

Fig. 4.9 Substrate encoded neural network system.


It is further implicit that the first layer of density 3 defines the sensory signal outports, rather than neurons, and that all neurodes in each layer are equally spaced from each other, and thus each neurode has its own coordinate [X,Y]. It is also this first neurode layer that specifies the sensory resolution, but is independ- ent of the neural network, and thus can be specified through development based on the environment (after all, higher resolution requires more processing power, and some environments might not require high resolution …). Finally, the implementa- tion is such, that the directly encoded NN is fed the coordinates of these substrate embedded neurodes (and sensory outports), and the output signal produced by the direct encoded NN is the synaptic weight between the connected neurodes in the substrate. Thus the synaptic weights of the neurodes are defined by this secondary NN.

At the end we end up with a substrate where each embedded neurode has the synaptic weight for its presynaptic connection. The first layer at Y = -1 represents the sensory outports, with the sensors specified in the SensorList (in this case just a single sensor), and the output layer at Y = 1 is connected to the actuators speci- fied in the ActuatorList (in this case a single actuator which can accept a signal of vector length 2). This is in essence a very simplified version of substrate encoding popularized by the HyperNEAT system [17]. We will cr eate a much more advanced substrate encoded system in Chapter-16.

Though slightly more complex than the direct encoded NN discussed earlier, we also note that there is a lot of information implicit in the architecture. The gen- otype does not on its own dictate every part of this NN, even the synaptic weights of the embedded neurodes depend on the density of each layer, which might change without us having to re-specify the synaptic weights for each of these neurodes. The substrate density can depend on the environment, or be generated stochastically. Thus, with this type of encoding, it is possible to create very dense neural substrates with thousands of neurons and millions of synaptic connections and weights, yet use a relatively little amount of information to specify all the synaptic weights and connections between the embedded neurodes, if the synaptic weights and connection expression is all dictated by the much smaller directly en- coded and specified NN. Through indirect encoding, a genotype can specify a NN based system of a much greater complexity than it could through direct encoding.

On top of the standard mutation operators we would use to mutate the directly encoded NN which specifies the synaptic weights of the substrate embedded neurodes, we could add the following:

 Increase_Density: This mutation operator chooses a random layer, and increas- es its density.

 Increase_Dimensionality: This mutation operator adds a new dimension to the substrate, and updates the necessary features of the NN needed to deal with this extra dimensionality, and thus the increased length of the input vector fed to the directly encoded NN (For example moving from 2d to 3d would result in switching from feeding the NN [X1,Y1,X2,Y2] to [X1,Y1,Z1,X2,Y2,Z2]).


 Decrease_Density: This mutation operator chooses a random layer, and de- creases its density.

 Decrease_Dimensionality: Same as the “Increase_Dimensionality ” function, but in reverse.

 Add_Coordinate_Preprocessor: The directly encoded NN is fed the coordinates of the two connected neurodes, but we can also first preprocess those coordi- nates, changing them from Cartesian to polar, or spherical... or instead of feed- ing the coordinates, we can feed a vector composed of the distances between the Cartesian points where the neurodes are located... This mutation operator performs a task analogous to the Add_Sensor and Add_Actuator, but adds pre- processors used by the NN.

 And of course we could add many other mutation operators, problem specific or general in nature.

For example, you and I are not a direct representation of our DNA, instead, we are based on our DNA, the development process, the chemical environment of the womb, and the nutrition we've been provided with at an early stage of our devel- opment. All of this influenced our various phenotypic features, body size and type, and even intelligence. For example, given the same DNA, a fetus in a womb of a mother who consumes alcohol and methamphetamines, will certainly be born with different features than if the mother did not consume these drugs during pregnan- cy. Thus the mapping from DNA to the actual organism is an indirect process, it is not one-to-one. There are a number of systems that use indirect encoding to pro- duce the resulting neural networks, by simulating chemical diffusion [5,6,7], by using multiple development phases and phase timing [8,9], and many other approaches. And we too will discuss and deve lop an indirect encoding approach in this book, though in a much later chapter.

In general, the phenotype, the way the neural network system is represented, is at the discretion of the researcher. We can implement the phenotype in hardware [10,11,12,13], on FPGAs for example, in software, or even in wetware using the technology that allows us to grow and connect biological neurons on a substrate [14,15,16]. When implementing in software, we can decide whether the phenotype is fully distributed, like a biological neural network system, or whether it will ac- tually be all processed in series, on a single core. Whatever the choice, as long as the information is processed in the same way, as long as the plasticity and the weights are the same, as long as the events occur at the same time and at the same rate in all these implementations, it does not matter whether the phenotype is hardware, software, or wetware based. The results will be the same; at the end, it's all just information processing.


The mapping from genotype to phenotype itself is done by a program that can read the genotype and then produce the phenotype based on that data. The mapper for the direct or indirect encoded NN system can also be a program implemented in many different ways, and the implementation is of course dependent on the genotype and the phenotype chosen. If for example the genotype is tuple encoded, stored in the Mnesia database, and the phenotype is a process based neural net- work system, and all of it is written in Erlang, then the mapper simply reads every tuple representing the neurons, sensors, and actuators from the mnesia database, and creates a process for each such node with the properties specified within the tuples. Each process then knows the Ids of the elements it is connected from and from which it should be expecting signals, and the elements to which it is con- nected to and to which it should be sending its output signals. Had the genotype been string encoded, and the actual neural network represented on an FPGA, then the mapping would have to be completely different.

                • Note********

In a biological organism, a ribosome reads the information in RNA and produces proteins mak-

ing up and used by our bodies. Thus a ribosome, and all the machinery needed for it to work, is

a part of the biological mapper program.

4.2 Neuroevolution Through Genetic Algorithms

In this section we discuss how a standard, single phased, genetic algorithm based neuroevolutionary approach works. We've seen in the previous section two different ways to encode a NN genotype, and the possible mutation operators needed to evolve populations composed of NNs encoded using such methods. Given this, how do we solve problems through neuroevolution?

To demonstrate how neuroevolution works, we will apply it to the following three example problems: 1. Creating a neural network system that functions as a XOR gate, 2. Creating a neural network system that balances a pole on a moving cart, and 3. Evolving a NN based intelligent system, with all the perks of neural plasticity, the ability to learn, and even the ability to modify its own neural struc- ture, its own brain.

4.2.1 Example 1: Evolving a XOR Logical Operator

As we have seen in Section 2.2.1, if we connect 3 neurons in just the right manner, and set their synaptic weights to the right values, the neural circuit will behave as a XOR operator. But how do we figure out what the right neural net- work topology is, what is the most optimal topology, and what the right synaptic weights for each neuron in that topology should be set to?

                • Note********


How do we apply a NN to some specific problem? There is no need to change the actual neural

network genotype encoding or its phenotype representation for every different problem. The

problems solved do not depend on the encoding we use, since it is not how the NN system and

its genotype are represented that matters, but it's the ability of a NN to interact with the prob-

lem or environment at hand. How the NN system can interact with the environment, problem,

or other programs, is solely specified by the sensors and actuators it uses. Like biological or-

ganisms, the way you interact and interface with the real world is through your sensors, and

through your muscles/actuators. We can use the same NN, but its functionality will differ de-

pending on whether the signals are fed to it from a camera sensor, a database sensor, a gyro-

scope sensor... and what it does will differ depending on whether the NN's signals are used to

control a differential drive actuator, the servos of a robotic hand, or the position of a mouse

pointer on a screen.

For each problem, we need only to create and set up a set of sensors and actua- tors that the NN system will use to interface with the environment/scape of that problem. We need only one sensor for this problem, one that can read from the XOR truth table a value, then feed it to the NN in a vector form, and then move to the next row in the truth table. We also need only a single actuator, one to which the NN passes its signal, and where the actuator then simply prints that value to console. The environment or scape in this case, is not some kind of 3d simulation, but instead is the truth table itself, a small database that contains the inputs and the expected outputs, and a small program that monitors the NN system's output, cal- culates how close it is to the correct output, and then gives the NN a fitness score based on the total error. The system setup is diagrammed in Fig-4.10 .

Fig. 4.10 A neuroevolutionary system setup to evolve a XOR logic operator.

What set of steps will a neuroevolutionary system follow to evolve a solution? The set of events would go something like this:


1. We develop a general neuroevolutionary system, which uses a tuple based gen- otype encoding, and which uses a fully distributed, process based, NN system implementation. The mapping from tuple encoded genotype to phenotype is performed by reading the tuples from the database, and converting each tuple to a process. Each such process expects signals coming from some set of ids, and after it has processed all the expected signals, it outputs a signal to the ids in its output_ids list, or it executes some function (if it is an actuator for example), and then begins the wait for the input signals anew.

2. We develop a set of general mutation operators, whose names and functionality are as follows:

add_Neuron

Generates a new neuron, and connects it from some randomly chosen neu- ron in the NN, and to some randomly chosen neuron in the NN. add_SynapticConnection

Selects a random neuron, which then randomly selects and adds either an input or output synaptic link to another randomly selected element in the NN system (neuron, sensor, or actuator).

splice

Chooses a random neuron, then a random element that the neuron is con- nected to, disconnects the two, and then reconnects them through a newly created neuron.

add_Bias

Chooses a random neuron without a bias, and adds a bias to its weights list.

3. We set up a fitness function for this problem. In this case, since we wish to minimize the general error between the NN system's output and the correct output based on the truth table, the fitness function is as follows: Fitness = 1/Error_Sum , where: Error_Sum = sqrt((Output1-ExpectedOutput1)^2 +(Output2-ExpectedOoutput2)^2 …) . The fitness is an inverse of the Er- ror_Sum because we wish for higher fitness to be represented by higher values, and through this fitness function we can ensure that the lower the error then the higher the fitness score.

4. We create an initial/seed population of very simple neural networks, randomly generated, with random synaptic weights. Let us imagine that in this example the population size is only 4, and our seed population is composed of the fol- lowing genotypes:

NN_1: [{s_id1,[n_id1]},{a_id1,[n_id1]},{n_id1,[{s_id1,0.3},{bias,0.2}],[a_id1]}]

NN_2: [{s_id2,[n_id2]},{a_id2,[n_id2]},{n_id2,[{s_id2,0.5}],[a_id2]}]

NN_3: [{s_id3,[n_id3]},{a_id3,[n_id3]},{n_id3,[{s_id3,-1}],[a_id3]}]

NN_4: [{s_id4,[n_id4]},{a_id4,[n_id4]},{n_id4,[{s_id4,-0.2}],[a_id4]}]


Genotype encoding is tuple based: [{sensor_id, fanout_ids}, {actuator_id, fanin_ids}, {neuron_id, input_idps, output_ids} …] . The possible initial popu- lation might be composed of the above genotypes, where each genotype is just a single neuron NN connected from a sensor and to an actuator, and where the single neuron in NN_1 starts off with a bias value in its weights list.

5. We convert each genotype in the population to its phenotype, and then calcu- late the fitness of each individual/phenotype by going through each input vector in the truth table and having the NN produce an output, which is then compared to the correct output of the truth table. The fitness of the phenotype is based on the general error of the 4 outputs the NN system provides for the 4 inputs it senses from the truth table. The fitness is calculated using the equation in step 3.

6. We choose 50% of the most fit individuals in the population for the next step, and remove the remaining least fit 50% of the population. Because the popula- tion size in this example is 4, this translates into choosing the two most fit agents within the population, and removing the 2 least fit agents.

7. We then use the genotype of these two individuals to create two offspring. To do so, we first clone both genotypes, and then apply X number of mutation op- erators to each of the clones to produce the mutant clones which are then des- ignated as offspring of the fit NNs. For each clone, the value X is chosen ran- domly to be between 1 and the square root of total number of neurons the NN is composed of. Thus, the larger the NN, the greater the range from which we choose X (we might apply just 1 mutation operator, or as many as sqrt(Tot_Neurons) to create the offspring). This method gives the mutant

clones a chance to both, be genetically close to their parents if only a few muta- tion operators are applied, and be far out on the search and solution space if a large number of mutation operators is applied.

8. We compose the new population from the two parents, and the two mutant off- spring. This new population is the new generation, and is again of size 4.

9. Go to step 5, until one of the NN systems achieves a fitness of at least 1000 (An error of less than 0.001).

In evolution it's all about exploring new genotypes and phenotypes, and the more permutations and combinations of synaptic weights and NN system topolo- gies that is tried out, the greater the chance that an offspring superior to its parent is evolved. The greater the population size, the more varied the mutation opera- tors, and the more varied the number of said mutation operators applied to the clone to produce an offspring, which results in greater NN population diversity. In an actual neuroevolutionary system, we would use a population size greater than 4. Though of course, each individual in the population requires computational


time to think, and so the larger the population, the more computational time re- quired to evaluate it... Whether it is better to have a larger population or a smaller one, whether it is best to choose the top 90%, top 50%, or only top 10% of the population to produce offspring from... are all dependent on the system and the mutation operators used. And even all of these features can be dynamic and evolv- able, as is the case with the previously briefly discussed approach called “evolu- tionary strategies ”.

By going through steps 1-9, we would eventually evolve a neural circuit that acts as a XOR logic operator. Assuming that the available activation functions to the neurons in this hypothetical neuroevolutionary system are the functions com- posing: [tanh,cos,sin,abs,gaussian,sgn], one of the possible evolutionary paths towards a solution is shown in the following figure.

Fig. 4.11 A possible evolutionary path towards a XOR logic operator.


4.2.2 Example 2: Evolving a pole balancing neurocontroller

The problem: Let us say that we have been given a problem of having to devel- op a way to balance two poles on a cart, where the cart itself is positioned on a 2 meter track, as shown in Fig-4.12 . Our only ability to affect the environment is by pushing the cart with a force we choose, either back or forth on the track, every 0.02 seconds. Furthermore, every 0.02 seconds we are allowed to measure the an- gle that the two poles make with the vertical, and the cart's position on the track. Finally, the solution must satisfy the following constraints: The cart must always remain on the track, the poles must never be more than 35 degrees from the verti- cal, the poles must be balanced for at least 30 minutes, and we can only interact with the track/cart/poles system by pushing the cart back and forth on the track. What we are seeking is to create a neurocontroller that can produce the force val- ues with which to push the cart to balance these poles, how can this be done through neuroevolution?

Fig. 4.12 The double pole balancing problem.

Let us do the same thing we did in the previous section. For this problem we will decide on what needs to be set up, before evolution can take its toll and evolve a neural network system that solves the given problem by sensing the sys- tem state, and outputting force values to control the cart.

The Problem representation: If we are given the actual hardware, the physical track, cart, and the poles attached to it, we would not really be able to apply an evolutionary process to it to create a neurocontroller that can push and pull the cart to balance the poles, since it would require us to create physical pushers and pull- ers, each controlled with its own neurocontroller... It would be a mess. Not to


mention, since we have to at least balance the pole for 30 minutes, and everything is done in real time, it would take us an enormous amount of effort, real hardware, and time to do this. Every evaluation of a NN's performance would take from a few seconds to 30 minutes, and it could take thousands of evaluations in total (thousands of offspring tested) before one of high enough fitness is found. Finally, we would also need the same number of cart/track/poles systems as the number of NNs in the population, if we wish to perform the said tasks in parallel. Otherwise we would need to perform the evaluations of the NNs one after the other, and thus further decrease the speed at which we would evolve a fit NN.

What we could do instead is create a simulation of the whole thing. Since the cart has to stay on the track, and the pole is attached with a hinge to the cart, it is constrained to two dimensions. The physical simulation would be rather simple, and require just a few equations to implement. Also, because we can use a simula- tion of the whole thing, we do not have to run it in real time, we can run the simu- lation as fast as our computer allows. So then, the way we can represent this prob- lem and evolve a neurocontroller to solve it, is through a simulation of the track- cart-poles system, with the ability for the cart to be pushed back and forth on the track. The next question is, how do we represent the neural network genotype en- coding?

The Genotype: We can use the same NN genotype encoding as in the previous problem, it is general enough to be used almost for anything, easy to read, easy to operate on, and easy to store in databases. In fact, the NN genotype encoding is independent of the problems we apply the NN based controllers to. Using the same NN genotype encoding, we need only to figure out how our NN would inter- face with the track-cart-pole simulation scape so that the NN can push the cart.

The Interface: We've created the simulation of the problem we'd like to solve. We know how to encode our NN genotypes. Our goal is to evolve a NN system that decides when to push the cart on the track, and from which side. So then, we know that our NN can have access to the cart's position on the track, and the an- gles between the poles and the vertical every 0.02 seconds of the simulation. We need to create a way for our NN to interface with the simulation, and that is where sensors and actuators come into play. It is the sensors and actuators that, in a sense, define what the NN is applied to and what it does. Our brain might be a highly adaptive system, capable of learning and processing very complex data in- puts, but without our body, without our arms, legs, muscles, without our eyes, ears, and nose, we cannot interface with the world, and there would be no way of telling what any of those output neural signals mean if they were not connected to our biological actuators.


For the NN to be able to interface with the simulation of the track-cart-poles system, we create a sensor that can gather a vector signal from the simulation eve- ry 0.02 seconds and feed it to its NN, which then processes that signal and sends its output to the actuator. The actuator of the NN needs to be able to use this signal to then push the cart back or forth. Thus, the actuator interfaces with the simula- tion, and tells the physical simulation from which side of the cart the force should be applied to push it. Furthermore, the sensor will forward to the NN a sensory signal encoded as a vector of length 3: [CartPos,Pole1Ang,Pole2Ang], where CartPos is cart position value, Pole1Ang is the first pole's angle to the vertical, and Pole2Ang is the second pole's angle to the vertical. The actuator will accept a connection from a single neuron in the NN, because it only needs one value, be- tween -1 and 1, which dictates which way to push the cart, and with what force. Thus the NN will forward to the actuator a vector of length 1: [F], where F is force. The only remaining thing to consider is how to set it all up into a neuroevolutionary process.

The setup: Having decided on all the parts, the NN encoding, the problem for- mulation (A simulation) so that we can solve it through evolution, and the way in which the NN systems will interface with the simulation so that we can assess their phenotypic fitness, we can now put it all together to evolve a solution to the given problem. A diagram of the setup of how a NN system would interface with the pole balancing simulation through its sensors and actuators, is shown in Fig-4.13 .

Fig. 4.13 This figure shows a neural network system interfacing with the pole balancing simulation. The NN based agent can sense the position of the cart, and the pole_1 & pole_2 angles with respect to the vertical, and push the cart on the track using its sensors and ac- tuators respectively, to interface with the pole balancing simulation scape.


Our neuroevolutionary approach would take the following set of steps to pro- duce a solution:


2. We develop a set of general mutation operators, whose names and functionali- ties are as follows:

add_Neuron

Generates a new neuron, and connects it to a random postsynaptic neuron in the NN, and a random presynaptic neuron in the NN. add_SynapticConnection

Selects a random neuron, which then randomly selects and adds either an input or output synaptic link to another randomly selected element (neu- ron, sensor, or actuator) in the NN system.

splice

Chooses a random neuron, then a random element that the neuron is con- nected to, disconnects the two, and then reconnect them through a newly created neuron.

add_Bias

Chooses a random neuron without a bias, and adds a bias to its weights list.

3. We set up a fitness function for this problem. In this case, the longer a NN sys- tem is able to keep the poles balanced by pushing the cart with its actuators, the higher its fitness. Thus : Fitness = Simulated_Time_Balanced .

4. We create an initial or seed population of very simple neural networks, ran- domly generated, with random synaptic weights.

5. We convert each genotype in the population to its phenotype, and then calcu- late the fitness of each individual. Each phenotype interfaces with its own pri- vate scape, a private simulation of the track-cart-poles system. Each tries to balance the poles for as long as it can by pushing the cart back and forth. As soon as any of the two poles deviates more than 35 degrees from the vertical, or the cart goes outside the 2 meter track, the simulation is over. At this point the NN system is given its fitness score based on the time it balanced the two poles. This is done for every individual/NN in the population.

6. We choose 50% of the most fit individuals in the population for the next step, and remove the least fit in the population by deleting their genotypes from the database.


7. We then use the genotype of these fit individuals to create offspring. We clone the fit genotypes, and then apply X number of mutation operators to each of the clones to produce the mutant clones which we designate as the offspring. For each clone, X is chosen randomly to be between 1 and the square root of total number of neurons making up the clone's NN. Thus, the larger the NN, the greater the range from which we choose X (we might apply just 1 mutation op- erator, or as many as sqrt(Tot_Neurons) to produce a mutant-clone/offspring).

8. We compose the new population from the fit parents, and their offspring. This new population is the new generation.

9. Go to step 5, until one of the NN systems achieves a fitness of at least 30 simu- lated minutes balanced. This resulting NN system is the sought after neurocontroller.

Once such a NN system is generated, we extract it from the population, and embed the phenotype into the hardware that supplies the sensors and actuators. The NN system is then connected and embedded in this robot system, the actual piece of hardware that interfaces with the physical track-cart-poles system. The robot's sensors then feed the NN with the signals it gathers from the track-cart- poles system (like our eyes that gather the visual information, propagating it to our brain), and the NN's output controls the robot's actuator, which then based on those signals (like our muscles based on the nerve signals coming from our brain) pushes the physical cart on the track.

Thus we have taken this problem all the way from formulating it in neuroevolutionary terms, to evolving the solution, and then using the evolved

solution to solve the actual problem. In the next section we try to see how we would use the same principles to solve a slightly more complex problem.

4.2.3 Example 3: End Game; Evolving Intelligence

What if the goal is to evolve intelligence? There is no better approach than the one through neuroevolution. Biological evolution had to evolve both: The mor- phology of the organism, and the neurocognitive computer to control it. It did this protein by protein, taking billions of years and trillions of permutations, but we can cut a few corners. And where the biologically evolved intelligence was simply a side effect of evolving a survival machine which can replicate in a hostile and uncertain world, where learning and adapting to the ever capricious environment is a must, for our problem we can ensure that adaptability and intelligence are di- rectly tied in with the fitness function.

As we previously discussed though, there is a certain problem with the granu- larity of simulation, and the dynamic and complexifying environment, as well as the organisms. For there to be a smooth evolutionary path from a simple organism to an intelligent and complex one, not only the organisms must smoothly increase


in complexity as they evolve, but they must also be able to affect the environment, and the environment must have a granularity fine enough that it too can slowly be- come complex, such that the complexifying organisms and the environment com- pose a positive feedback loop, slowly ratcheting their mutual complexity upwards.

For the environment to provide enough flexibility for the organisms to interact with, it must be simulated at a very high level of granularity, perhaps even atomic. The population size must also be vast, after all, it took billions of years of evolu- tion with the population of various species and organisms inhabiting and spread out across the entire planet. All of this will require an enormous amount of computational power.

The place where we can cut corners and speed things up is in the actual geno- type representation, morphology, and neural network systems. With regards to ge- nome, we do not need for the evolutionary process to discover RNA and evolve it into DNA, we can start off with systems which already have this genome encoding mechanism. We also don't need for the evolutionary process to discover from scratch how to create a biological information-processing element, the neuron. We can start off using agents which can already use neurons from the very start. We also can cut a few corners with regards to evolving viable morphologies. We could provide the NN with the various sensors and actuators that it can integrate into it- self through evolution. The sensors and actuators would in effect represent the morphology of the organism, and so we could allow the evolution, through this approach, to evolve agents from the morphological form which uses a simple fla- gella to move, to an agent with a sophisticated neurocognitive computer and bi- pedal morphology. We need not rediscover bipedal organisms, legs, fins... and we do not need to rediscover the various chemical pathways to use energy, or support- ive structures like bones... thus our evolutionary system just needs to primarily concentrate on the evolution of the neurocognitive computer, the rest is taken care of by science and engineering, rather than chance.

What about the fitness function? We could of course create organisms which are capable of producing offspring once they have gathered enough energy from the artificial environment, thus letting them control their own reproductive cycles. Or we could create an outside system that chooses the genotypes from which to create the offspring. It is interesting to consider what would be the difference in evolutionary paths taken by such two disparate approaches. The organisms capa- ble of initiating their own reproductive cycle need no fitness function because only those will survive whom can create offspring, since only they will be able to pass their genetic material to the next generation, and their offspring too will try to cre- ate offspring of their own, mutating and evolving over time. The ability of an agent to control its reproductive cycle, and having to compete for resources, will drive the system towards complexity. But since we start the seed population with random and very simple NN systems, initially no one would be able to create their own offspring, or have capabilities to effectively gather the needed resources to do


so. So we would actually have to bootstrap the system until it has achieved a steady-state. To do so, we would run the simulation, restarting populations and generating new offspring, until finally one of the individuals in the population acquires the ability to reproduce, at whic h point the agents will take over when it comes to creating their offspring. This is similar to how it took a few billion years of random collisions and molecular permutations on our planet, before by chance alone, one of the combinations of those molecules formed a chemical replicator.

If we were to not allow self initiated reproduction, and instead used a program which chose whom of the individuals had the most fit and diverse genotype, and whom should be chosen as the base for offspring creation, then we would have to think which of the traits and accomplishments of the organism during its lifetime that we should reward? Where to place the new offspring in the physical environ- ment? In the case of self initiated reproduction, the offspring can be placed right next to the parent, which might also evolve the behavior of teaching, after all, those offspring that have the neural plasticity to learn, and a parent that can teach it the already learned tricks of survival, would have a much greater chance of sur- viving, and thus passing onwards its genotype which encoded for teaching its off- spring in the first place... To solve this problem we could set up spawning pools, certain designated areas where we would put offspring. Or we could randomly po- sition them somewhere in the environment. Or perhaps we could randomly choose an organism in the environment that most closely resembles the genome of the new offspring, and place that offspring next to that currently living individual. All of these are viable approaches, but for this example, we will choose the bootstrap- ping self initiated reproduction approach.

Let us then think how, if computational power was limitless, and high precision physics simulations at the level of atomic particles was also available, would we then set up a neuroevolutionary system to evolve intelligence? We will follow the same steps and approaches we did in the previous section, we will define the prob- lem, the possible genotype encoding, the interface, and the setup.

The Problem: Evolve a population of organisms capable of intelligence, learn- ing, adaptation, and inhabiting/controlling a body similar to the form of a sapient organism.

The Genotype Encoding: The NN system genotype encoding is the same as in the previous two sections. But because we need for there to be a whole different set of morphologies, all the way from a bacteria to that of a bipedal robot, we need to somehow include the evolution of morphology, integrated with the evolution of the NN, all in a single system. We can do this through the use of sensors, actua- tors, and modular robotics, as shown in Fig-4.14 .


Fig. 4.14 Evolving morphology through sensor and actuator modules of the NN system.

The above figure demonstrates how to encode morphological properties and in- tegrate them into the same encoding as the NN, and thus evolve morphology flu- ently and in synchrony with the neural network. A sensor is a program used by the NN to gather data from the scape it interfaces with. A sensor can be anything, and a sensor program can also define, when simulated in some physical environment, a shape, or morphological property, location on a body... An actuator is a program that the NN uses to act upon the world. There can be many different kinds of actu- ators, each possessing some particular morphological property, look, and defining some particular body part. But also, just like the case with the sensor which might read sensory signals coming from inside the NN, the actuator might be a system that affects the agent's own NN topology. Again, a sensor and an actuator is any- thing we can come up with to feed the NN signals, and used by the NN to perform some function, respectively.

We can further tie-in the sensor and actuator combinations with particular body structures when simulated inside environments. For example there could be a fla- gella actuator, and if some NN uses a chemical receptor sensor, and has a flagella actuator, meaning the NN gathers data from the environment through its chemical receptor, and can only move through the environment using its flagella, then the physical simulation can represent this individual as a bacteria, with the appropriate physical properties and size. The sensor and the actuator both have a particular way they look, and their own physical properties. When for example the noted sensors and actuators occur together in the NN, the scape would set the morphology of the NN's avatar to a bacterium, representing the actuator as the flagella, and the sensor as a small attachment capable of sensing the chemical properties of the avatar's immediate surroundings.


Thus, on top of the tuple encoded NN genotype, we also want to have a very large list of sensors and actuators that the NN systems can integrate over time into themselves through the application of the add_sensor, add_actuator, and swap_sensor/swap_actuator mutation operators. The swap_sensor and swap_actuator mutation operators choose an already integrated sensor or actuator, and swap it with a new randomly chosen sensor or actuator respectively. The dif- ferent sensors could be as follows: [chemo_sensor, photo_sensor, pressure_sensor, monoscopic_camera, telescopic_camera...]. The different actuators could be as follows: [flagella, differential_drive_wheels, 3_degfree_leg, 4_degfree_leg, 5_degfree_leg, 3_degfree_arm, 4_degfree_arm, 5_degfree_arm, 3_fingeredhand, 4_fingeredhand, 5_fingeredhand, 6_fingeredhand, pivot_tilt_sensor_mount, 3part_torso, 4part_torso...]. The sensor and actuator tags are names of functions, which when integrated into a NN existing in a simulation, also have simulated physical representations. The sensors and actuators do not need to be evolved from atomic or molecular building blocks, the NN needs only to learn how to con- trol these mountable interfaces.

Furthermore, the type of actuators used and integrated could define the general body type, and we could set it up such that the increase in size of the neural network automatically increases the general size of the body and energy drain. Or we could use a finer level of granularity when adding new sensors and actuators, and simply add one joint at a time, thus building up the gripper and other types of actuators. This could then possibly make the add_sensor and add_actuator mutations more flexible, allow them to randomly also specify the position where to attach the sim- ple new morphological pieces (joints, connectors …), but at the same time this would provide a few orders of magnitude more permutations resulting in unusable, unstable, and unfit morphologies.

Finally, we could also add a sensor list as follows: [read_self_genotype, read_other_genotype] and actuator list as follows: [modify_node, next_node, swap_node, move_node...]. These functions would allow the organisms to evolve

capabilities to self augment, and to have the ability to read the genotypes belong- ing to others, which would perhaps require the simulation environment to set up certain rules, like having the other organism be willing, or be subdued or killed first, before its genotype can be examined.

These are all possibilities, and there is really no reason why abilities to use the- se types of skills should not be possible to evolve. As long as the environment and the mutation operators available are flexible and robust enough, there will be an evolutionary path to integrate these capabilities. In this book, when we begin dis- cussing applications, we will actually build a small 2d ALife system, where prey organisms could evolve “teeth ”, and learn how to attack other prey. In fact I have implemented such a system, and indeed simple prey and predators did evolve to hunt and evade each other, to use plants (simulated food elements) as bait, and even evolve from prey to predator by evolving (through the use of add_sensor and add_actuator mutation operators) new morphological parts, integrating them, and becoming more effective at hunting, evading, and navigating.


The Interface: Somewhat of a continuation of the above, the interface of the NN to the physical simulated environment is done through the sensors and actua- tors. The NN receives through the sensors all the information that its avatar sens- es, and whatever signals it sends to the actuators, the avatar performs. When the avatar dies, the NN dies, at which point the phenotype's, and therefore the NN system's performance in the environment, is given a fitness score depending on the organism's achievements during its lifetime. The achievements could be the following: the amount of food gathered during lifetime, the number of inventions made, the amount of world explored, or simply the number of offspring created if the system is a bootstrapped one where agents can replicate of their own accord.

The Setup: In this hypothetical example we will choose to use a bootstrapped approach, bootstrapping the initial seed population of organisms, until one emerg- es that can effectively use the create_offspring actuator to produce an offspring when it has enough energy to do so. The environment, the world, is a simulated 3d environment. It is simulated all the way at the atomic level, since we assume that we have access to unlimited computational power, and thus the processing power required to simulate an entire planet at an atomic level is not an issue. Each NN interfaces with the simulated world through its avatar, a simulated body whose sensors feed sensory signals to the NN, and whose actuators are controlled by the NN. We could even go as far as simulate the actual NN topology in physical form, and position that NN physical representation inside the avatar.

The physical form the organism takes is determined by the sensors and actua- tors that belong to the NN, the sensors and actuators that the NN evolved over time. The size and shape of the avatar is further dependent on the size of the NN system, and the energy requirements, the amount of energy the organism burns per simulated second, also depends on the sensors, actuators, the NN size, and the or- ganism's size. Each organism will also start with a create_offspring actuator, which requires a certain amount of energy and time to be used, after which there is

a delay time during which an offspring is created. The offspring is created in the same fashion as in the previous section, it is a mutated clone of the organism, and the number of mutation operators applied to produce it depends, with random in- tensity, on the complexity of the parent's NN. Unlike in the previous problem, each NN system does not interface with their own private scape , but instead all the NN systems (the avatars/agents), inhabit the same large simulated world, the same public scape . Because each organism creates its own offspring based on the energy it has gathered by eating other organisms in the environment, we do not need to set up a fitness function. Instead only the create_offspring function needs to be set up in such a way that it has a cost to the organism, and that perhaps dur- ing the first few simulated years of the offspring, the offspring does not function at full capacity, and neither does it function at full capacity when it is in its more ad- vanced age. Its power output, efficiency, speed, could follow a Gaussian curve


proportional to its age. Finally, the organisms are set to die when they get eaten, or when reaching some specific age, which might itself be based on the total number of sense-think-act cycles the NN has performed. This could have an interesting ef- fect, since larger NNs will not react as quickly as smaller ones, due to it taking a longer time to process a signal with 100 billion neurons than it does with a single neuron. This will mean that more complex organisms, though slower, will live longer (same number of sense-think-act cycles, but longer period of time), and be able to do much more complex computations per cycle than simpler organisms. So age could also be based on the neural network complexity of the organism. With this in mind, the evolutionary system might follow the following steps:


add_Neuron

Generates a new neuron, and connects it to a random postsynaptic neuron in the NN, and a random presynaptic neuron in the NN. add_SynapticConnection

Selects a random neuron, which then randomly selects and adds either an input or output synaptic link to another randomly selected element (neu- ron, sensor, or actuator) in the NN system.

splice

Chooses a random neuron, then a random element that the neuron is con- nected to, disconnects the two, and then reconnect them through a newly created neuron.

add_Bias

Chooses a random neuron without a bias, and adds a bias to its weights list. add_sensor

This mutation operator chooses a random sensor from the available list of sensors, and connects it to a randomly chosen neuron in the NN. swap_sensor

This mutation operator randomly chooses a currently used sensor, and swaps it for another available sensor.

add_actuator

This mutation operator chooses a random actuator from the available list of actuators, and connects to it a random neuron in the NN.


swap_actuator

This mutation operator randomly chooses a currently used actuator, and swaps it for another available actuator.

3. We set up the physical simulation of the environment, and all non biological properties to be ran by the simulation. We also set up the metabolism simula- tion within the environment, such that agents are drained of energy proportion- al to the size of their avatars, and NN size.

4. We create an initial/seed population of very simple neural network systems, each with its own set of sensors and actuators that defines the morphological properties of their avatars.

5. We convert each genotype in the population to its phenotype, and then let the organisms live out their lives. Some will starve to death, some will flourish.

6. If all organisms die, and non are able to create an offspring, we supplement the population by creating offspring from the genotypes that were most fit, exhibit- ed most intelligence, survived the longest... (this is the population bootstrap- ping part, used until one of the generated agents can control its create_offspring actuator, and is able to create an offspring that can continue the cycle).

7. Once the organisms begin to emerge which know how to use create_offspring actuators, and the population stabilizes and begins to grow, the bootstrapping and population supplementation stops. Whereas until this point we in a sense simulated the stochastic collisions and interactions and the creation of various invalid replicators, after this point a replicator has been generated through ran- dom processes, and evolution takes over where stochastic search left off. At this point the properties of the simulated world and evolution take over.

8. Due to the flexibility and fine granularity of the environment, it is morphed and shaped by the evolving organisms, which are morphed and shaped by co- evolution, arms race, competition, and the environment that they shape and morph.

9. As organisms compete with each other in an environment that is growing ever more complex, different species will begin to emerge as various agents find niches in the environment to exploit, separating from others to occupy and spe- cialize in those niches. Different sized NNs and avatars will begin to emerge...

10. The simulation is allowed to run indefinitely, so that the increasing complex- ity of the environment and agent interaction ratchets the increase in complexity of the system... setting up a positive feedback loop. If at any moment complexi- ty stops increasing, if it stabilizes, or evolution takes the wrong turn some- where, and becomes unable through mutation to jump out of a local intelligence optima, the researcher can then modify the environment to try to set up scenar- ios that require an increase in intelligence. This can be done by for example set- ting up famine scenarios, increasing the difficulty of reaching certain foods, adding new elements to the environment that require cleverness to be exploited, or making the environment more fine grained, more realistic.

4.3 Neuroevolution Through Memetic Algorithms

As you can see, the setup is pretty much the same as in the previous examples. As long as we can formulate the problem in evolutionary terms, and any problem can be, we can then evolve a solution for it. In the next section we will briefly ex- plore a variation on the standard evolutionary algorithm based approach. In the next section we will discuss a memetic algorithm based neuroevolution.


A memetic algorithm subdivides the search algorithm into two phases, global search and local search. In neural networks, this separation into two phases might mean the separation of the topological mutation operators from the synaptic weight mutation operators. When we mutate the topology of the neural network, we are exploring NN systems that differ from each other significantly. On the oth- er hand, when we tune and perturb synaptic weights, we are exploring the local so- lution space of the NN system of a particular topology, tuning the synaptic weights and guiding them towards the local optima that the topology can achieve. The following figure shows and compares the steps of the genetic and memetic al- gorithm based neuroevolutionary systems.

Fig. 4.15 The comparison of genetic and memetic algorithm based neuroevolution.

There are both, advantages and at times disadvantages when using memetic computing instead of genetic computing. One of the main advantages is that when we evolve a new topology for an organism, we do not immediately discard it if it


does not function better than its parent. Instead we give it time to tune in the syn- aptic weights for its neural topology, giving it time to reach its full potential (giv- en a reasonable amount of computational time), and only then judge it. In a genet- ic algorithm type system, we would randomly mutate a fit topology, giving the new features random synaptic weights, and then expect it to outperform its parent. That is of course highly unlikely, after all, when we perturb a functional topology with random features which can technically be considered garbage DNA until proven otherwise, there is very little chance that the new synaptic weights will be in tune with the whole system and make it superior to its parent. At the same time,

when we mutate a topology, there are usually only a few new weights added (when adding a new neuron, making new synaptic connections, or adding a bi- as...), and so it would not be difficult or costly, to try out a few different weight setups before discarding the topological innovation. Thus, where memetic algo- rithm keeps topology as a constant while searching through the synaptic weights that make this new topology functional (before trying out a new topology), the ge- netic algorithm system hopes to hit the perfect combination with regards to the NN topology and its synaptic weights in one throw. But getting both at the same time, from the sea of such combinations, is not likely to occur, and it is for this reason why evolution takes so much time.

But there are a few disadvantages to memetic algorithm based systems as well, particularly in ALife simulations where organisms can create offspring by their own initiative. For example, how do we do weight tuning in such a scenario? If to perform weight tuning we use a stochastic hill-climbing (SHC) algorithm, how can we allow the organism to create offspring during its lifetime? After all, the whole point of SCH is that we see if the new locally found solution is better than previous one, and if not we destroy it and recreate a new one from the originally fit system. To weight tune, we need to have a fitness score for the organism, which is usually given once it dies. In a genetic algorithm based system, an organism or agent simply creates an offspring. But in a memetic algorithm driven system, at what point do we perform global search? At what point do we perform local search? Also, during the local search we would try one organism at a time with a different set of local parameters, to see if it's better or worse than its parent, but that is not possible in an ALife scenario. The parent creates an offspring and now both are alive, and we won't know which is better until both are dead. If both are dead, what will create a new offspring to continue local search?...

Thus, when using a memetic algorithm based neuroevolutionary approach, it is no longer as trivial as giving an organism an actuator that allows it to create an offspring. Thus if we are willing to have a program external to the ALife simula- tion create the offspring of the agents within the scape, then memetic algorithm


based neuroevolution can also be used in ALife. To do this, we remove the ability of each organism to control its own reproduction, and use an external program that

calculates each organism's fitness during its lifetime. When some agent within the environment dies, this program creates an offspring based on some already dead but fit genotype. Finally, this selection algorithm program then randomly positions this newly created offspring in some kind of spawning pool in the ALife environ- ment.

Fig. 4.16 Using spawning pools in ALife for offspring creation.

A spawning pool is basically a designated location where the offspring will be spawned. We might set up as many spawning pools as there are different species in the population, and position the spawning pools in the area which has the high- est density of its species around. Then, when an offspring is created, it is not cre- ated near the parent, but in the species communal spawning pool. To perform syn- aptic weight tuning in ALife when using the memetic algorithm approach, we would then wait for the parent to die, and then respawn it with a new set of synap- tic weights. The organism would be spawned in the spawning pool location. We would do this multiple times, as appropriate for a memetic algorithm. Once we've given enough processing time to tune the synaptic weights of some given geno- type, we would give that agent's genotype its final fitness score. At this point, when a new offspring is created from some set of fit genotypes, that list of fit gen- otypes (to which I also refer to as dead_pool at times) has the newly scored geno- type added to it, with its true fitness score.


4.4 Neural Networks as Graph Based Genetic Programming Systems

We discussed in Chapter-3 four flavors of evolutionary computation: genetic algorithms, genetic programming, evolutionary algorithms, and evolutionary pro- gramming. We also noted that they are all virtually the same thing, just with a

slightly different encoding method. Neuroevolution can be added to this list as just another side of the coin, and which is basically a specialization of a graph based genetic programming system, as shown in Fig-4.17 .

Fig. 4.17 Tree encoding to graph encoding to neural networks.

Where in Chapter-3 we saw how genetic programming can be used to evolve circuits, here instead of letting each node be a logic gate, it is a function that

weighs incoming signals with synaptic weights, sums up the weighted signals in some fashion, and then applies some activation function to this sum to produce the final output. A neuron is just a particular set of functions, it can be considered as a program, and thus a genetic programming approach, whether it be tree encoded (single output) or graph encoded (multi-input and multi-output), is effectively a neural network.


Just as you may create a genetic programming framework, and provide a num- ber of different programs to be used as nodes in your system, so may you create a neuroevolutionary framework, and provide a number of different activation func- tions to be used by the neurons. The more advanced these two types of frame- works become, the more the same they become. If we are to take the position of stating that neuroevolution is a specialized form of genetic programming which concentrates on using smooth functions in its nodes, we have to admit that a neu- ral circuit whose neurons use tanh as activation functions, is a universal function approximator... and since a neural network is a graph of interconnected neural cir- cuits, then a neural network is a graph of any type of functions as well, as shown in Fig-4.18 .

Fig. 4.18 Neural network as a graph of neural circuits using tanh activation functions.

Thus, an advanced enough neuroevolutionary system, and an advanced enough genetic programming system, would be equivalent. Whatever nodes a genetic pro- gramming framework would have access to, a neuroevolutionary system could al- so have access to. Sure, it would feel less like a neural network if we were to start using activation functions: “ while ” and “ if ” from the very start inside the neu- rons... But why? After all, a threshold neuron is basically an if , and a recurrent connection is in some sense a while loop. Why not allow access to these programs directly, rather than having to re-evolve them through combinations of activation functions and particular topologies, neuron by neuron? In the same way, a Hop- field network circuit, or a small self organizing map, can be evolved through ge- netic programming by letting the programs be neurons...


Though I know I've already made a comment to the effect that all these sys- tems, genetic algorithms, genetic programming, evolutionary strategies, evolu- tionary programming, neural networks, universal networks... are all the same, it is important that this is seen. A genetic algorithm works on string based genotypes, genetic programming is just a genetic algorithm that is applied to tree based en- coding, and a graph encoded genetic programming system is a standard genetic programming algorithm applied to multi-rooted trees. Evolutionary strategies is just a standard genetic algorithm which also allows for the parameters dictating the various evolutionary features to mutate, and evolutionary programming is just genetic programming applied to finite state machines instead of tree, or graph encoded systems... Nevertheless, as per the standard, in this book we will view a graph as a neural network if the majority of the nodes are biologically inspired.

4.5 References

[1] Stanley KO, Risto M (2002) Efficient Reinforcement Learning through Evolving Neural Network Topologies. In Proceedings of the Genetic and Evolutionary Computation Confer- ence.

[2] Sher GI (2012) Evolving Chart Pattern Sensitive Neural Network Based Forex TradingAgents. Available at: http://arxiv.org/abs/1111.5892 .

[3] Cormen TH, Leiserson CE, Rivest RL, Stein C (2001) Introduction to Algorithms T. H. Cormen, C. E. Leiserson, R. L. Rivest, and C. Stein, eds. (MIT Press).

[4] Stanley KO, Miikkulainen R (2003) A Taxonomy for Artificial Embryogeny. Artificial Life 9, 93-130.

[5] Cangelosi A, Parisi D, Nolfi S (1994) Cell Division and Migration in a “Genotype ” for Neural Networks. Network Computation in Neural Systems 5, 497-515.

[6] Turing AM (1952) The Chemical Basis of Morphogenesis. Philosophical Transactions of the Royal Society B Biological Sciences 237, 37-72.

[7] Fleischer K, Barr AH (1993) A Simulation Testbed for The Study of Multicellular Develop- ment: The Multiple Mechanisms of Morphogenesis. In C. G. Langton (Ed.), Artificial life III, 389-416.

[8] De M, Suzuki R, Arita T (2007) Heterochrony and Evolvability in Neural Network Devel- opment. Artificial Life and Robotics 11, 175-182.

[9] Matos A, Suzuki R, Arita T (2009) Heterochrony and Artificial Embryogeny: a Method for Analyzing Artificial Embryogenies Based on Developmental Dynamics. Artificial Life 15, 131-160.

[10] Thiran P, Peiris V, Heim P, Hochet B (1994) Quantization Effects in Digitally Behaving Circuit Implementations of Kohonen Networks. IEEE Transactions on Neural Networks 5, 450-458.

[11] Glesner M, Pochmuller W, (1994) An Overview of Neural Networks in VLSI. Chapman & Hall, London.

[12] Schwartz TJ (1990) A Neural Chips Survey. AI Expert 5, 34-38.

[13] Heemskerk JNH (1995) Overview of Neural Hardware. Neurocomputers for BrainStyle Processing Design Implementation and Application, 1-23.


[14] Matsuzawa M, Potember RS, Stenger DA, et al (1993) GABA-Activated Whole-Cell Cur- rents in Containment and Growth of Neuroblastoma Cells on Chemically Patterned Sub- strates. J. Neurosci. Meth. 50, 253-260.

[15] Matsuzawa M, Kobayashi K, Sugioka K, Knoll W (1998) A Biocompatible Interface for The Geometrical Guidance of Central Neurons in Vitro. Journal of Colloid and Interface Sci- ence 202, 213-221.

[16] Matsuzawa M, Krauthamer V, Richard S (1999) Fabrication of Biological Neuronal Net- works for the Study of Physiological Information Processing. Johns Hopkins APL Tech. Dig. 20(3), 262-270

Chapter 5 The Unintentional Neural Network Programming Language

Abstract The programming language Erlang has a perfect 1:1 mapping to the problem domain of developing neural network computational intelligence based systems. Erlang was created to develop distributed, process based, message pass- ing paradigm oriented, robust, fault tolerant, concurrent systems. All of these fea- tures are exactly what a programming language created specifically for developing neural network based systems would have. In this chapter I make claims to why Er- lang is such a perfect choice for the development of distributed computational in- telligence systems, and how the features of this programming language map per- fectly to the features needed by a neural network programming language. In this chapter I briefly discuss my reasons for considering Erlang to be, though uninten- tionally so, the quintessential neural network programming language.

We have discussed what neural networks are, how they process data, and one of the goals in computational intelligence, the creation of one that rivals our own. I think that neural networks are really the best way to achieve singularity, and to create general computational intelligence, a truly intelligent neurocognitive sys- tem. But there is a problem, the way neural networks function, their architecture, the way they process information, the amount of processing they require, the con- currency the NN system needs, and how such neural networks need to be distrib- uted among available computing systems, is not easily mapped-to by the standard programming language architectures like C/C++/C#/Java/Lisp...

If you're wondering “Why should it matter? ”, here is the thing: If you wanted to build a neural network system in C++, and there have been a number of such systems built, you have to think in C++ and its architecture, and translate those ideas into the NN architecture. Though it does not sound like it should matter, it does, and it has to do with linguistic determinism [1], which roughly states that it is difficult to think in, and create new ideas in, languages which have not been de- signed for, or have the elements required for, such ideas. For example, if your nat- ural language does not support the concept of mathematics, thinking about math- ematics would be very difficult, it would require a revolution within the language, making it support such concepts. When you are programming in C++, you are thinking about systems in C++, instead of thinking and visualizing neural network systems, which is the real goal of your project. You are thinking about C++ and worrying about how you can represent NNs through it. Neural Networks are very different in their architecture from the programming languages used today. To de- velop robust NN systems, to advance them, to be able to concentrate just on com- putational intelligence (CI) without having to worry about the programming lan- guage, and to have to the tools which can concisely map to the NN based CI

DOI 10.1007/978-1-4614- 4463 - 3_5 , © Springer Science+Business Media New York 2013


architecture is important. It not only makes things easier, it makes it possible to consider such things, and thus it makes it possible to create and develop such things. This is what Erlang [4,5,6] offers.

5.1 The Necessary Features

If you had the chance to make a programming language from scratch, with any features you wanted, and you wanted to create a language with the architecture that maps perfectly to neural networks, and for the development of true computa- tional intelligence, what features would it need?

We would of course first want the programming language to make it architec- turally mirror neural networks, so that all ideas and inventions in the field of neu- ral computation could exactly and directly be represented by this new program- ming language. This means that this programming language must have structures similar to neural networks. To do this, the programming language would need the following:

1. Neural networks are composed of independent, concurrent, distributed pro- cessing units called neurons. Thus the programming language architecture would need to support having such elements, having independently acting pro- cesses which can all function in parallel, and which can be easily distributed throughout the modern parallel hardware.

2. The neurons in NNs communicate with each other through signals. Thus the programming language architecture needs to allow the processes to communi- cate with each other through messages or signals too.

Having the programming language's architecture that mirrors the architecture of neural networks is not enough. Our brains are robust, we usually don't encoun- ter situations where we get a “bug ”, and suddenly we crash. In other words, our brains, our neural systems, our biological makeup is robust, fault tolerant, and self recovering/healing. For the language architecture to support the creation of a true computational intelligence, it needs to allow for a similar level of robustness. To accommodate this it needs the following features:

1. Allow for an easy way to recover from errors.

2. If one of the elements of the computational intelligence system crashes or goes down, the CI system must have features that can recover and restart the crashed elements automatically. There must be multiple levels of security, such that the processes are able to watch each other's performance, monitoring for crashes and assisting in recovering the crashed elements.

But simply being able to recover from crashes is not enough, simply being ro- bust is not enough. Though the neural network itself takes care of the learning part, performs the incorporation of new ideas, the growth and experience gaining,


there is one thing that biological organisms do not have the ability to do with re- gards to their intelligence. Biological organisms do not have the ability to modify their our own neural structures, we do not have the ability to rewrite our neural networks at will, the ability to update the very manner in which our biological neural networks process information... but that is the limitation of biological sys- tems only, and non biological systems need not have such limitations. Thus a pro- gramming language architecture must also provide the following features:

1. The programming language must allow for code hot-swapping. For the ability for the CI system to rewrite the code that defines its own structure, its own neu- ral network, or fix errors and then update itself, its own source code without taking anything offline.

2. The programming language architecture must allow for the CI system to be able to run forever, crashes should be local in nature, which should be fixable by the CI itself.

Finally, taking into account that the neural network based CI systems should be able to interface and control robotic systems, be used in Unmanned Ariel Vehicles (UAVs), or in humanoid robots, the programming language should from the start make it easy to develop and allow for control of a lot of different types of hard- ware. It should allow: for an easy ability to develop different and numerous hard- ware drivers.

In short, the programming language architecture that we are looking for must process information through the use of independent concurrent and distributed processes. It should allow for code hot-swapping. It should allow for fault toler- ance, and error fixing, and self healing and recovery. And finally, it should be made with ability to interface with large number of hardware parts, it should allow for an easy way to develop hardware drivers, so that not only the software part of the CI be allowed to grow and self modify and update, but it should also be able to incorporate and add new hardware parts, whatever those new parts may be.

A list of features that a neural network based computational intelligence system needs, as quoted from the list made by Bjarne Dacker [2], is as follows:

1. The system must be able to handle very large numbers of concurrent activities. 2. Actions must be performed at a certain point in time or within a certain time. 3. Systems may be distributed over several computers.

4. The system is used to control hardware.

5. The software systems are very large.

6. The system exhibits complex functionality such as, feature interaction.

7. The systems should be in continuous operation for many years.

8. Software maintenance (reconfiguration, etc) should be performed without stop- ping the system.

9. There are stringent quality, and reliability requirements.

10.Fault tolerance

Surprisingly enough, Dacker was not talking about neural network based general computational intelligence systems when he made this list, he was talking about telecom switching systems.

5.2 Erlang: From Telecommunications Networks To Neural Networks

Erlang is a concurrency oriented (CO) programming language. It was devel- oped at Ericsson, a project lead by Dr. Joe Armstrong. Erlang was created for the purpose of developing telecom switching systems. Telecom switching systems have a number of demanding requirements, such systems are required to be highly reliable, fault tolerant, they should be able to operate forever, and act reasonably in the presence of hardware and software errors. And these features are so close to those needed by NN based systems, that the resulting language's features are ex- actly those of a neural network programming language. Quoting these necessary features of the programming language from [3]:

1.

2.

3.

4.

5.

“Encapsulation primitives — there must be a number of mechanisms for limit- ing the consequences of an error. It should be possible to isolate processes so that they cannot damage each other.

Concurrency — the language must support a lightweight mechanism to create parallel process, and to send messages between the processes. Context switch- ing between process, and message passing, should be efficient. Concurrent processes must also time-share the CPU in some reasonable manner, so that CPU bound processes do not monopolize the CPU, and prevent progress of other processes which are “ready to run. ”

Fault detection primitives — which allow one process to observe another pro- cess, and to detect if the observed process has terminated for any reason. Location transparency — If we know the PId of a process then we should be able to send a message to the process.

Dynamic code upgrade — It should be possible to dynamically change code in a running system. Note that since many processes will be running the same code, we need a mechanism to allow existing processes to run “old ” code, and for “new ” processes to run the modified code at the same time.

With a set of libraries to provide:

6. Stable storage — this is storage which survives a crash.

7. Device drivers — these must provide a mechanism for communication with the outside world.

8. Code upgrade — this allows us to upgrade code in a running system.

9. Infrastructure — for starting, and stopping the system, logging errors , etc. “

5.3 The Conceptual Mapping of a NN to Erlang's Architecture

It is this that Erlang provides, and it is for this reason why we use it for the de- velopment of neural network based systems in this book. I have found this lan- guage to be so perfect for the task, that I must admit to be unable to see myself us- ing anything else in future research within this field.

Whereas before I would need to first create the NN algorithms, topologies, and architectures separately, and then try to figure out how to map the programming language like C++ to the task, or even worse, think in C++, and thus create a sub- par and compromised NN based system, or still worse think in C++ and not be able to see the forest for the trees when it comes to CI and NN... With Erlang, the ideas, the algorithms and NN structures are mapped to Erlang perfectly, and vice versa. The ideas that would otherwise be impossible to implement, or even con- sider when one thinks in one of the more commonly used languages, are easily and clearly mapped to Erlang. You do not need to switch from thinking about neu- ral network systems, algorithms, and architecture of a NN based CI when develop- ing in Erlang. The conciseness of the language, the clarity of the code and the pro- gramming language's architecture... make even the most complex problems which would otherwise not be possible to solve, effortless .


In Erlang, concurrency is achieved through processes. Processes are self con- tained, independent, concurrently running micro server/clients, only able to inter- act with each other through message passing. Already you can visualize that these processes are basically neurons, independent, distributed, concurrent... only able to communicate with each other by sending signals, action potentials.

Once again taking a quote from Armstrong's thesis, where he notes the im- portance of there being a one to one mapping between the problem and the pro- gram, the architecture of the programming language and that which is being de- veloped with it: “It is extremely important that the mapping is exactly 1:1. The reason for this is that it minimizes the conceptual gap between the problem and the solution. If this mapping is not 1:1 the program will quickly degenerate, and become difficult to understand. This degeneration is often observed when non-CO languages are used to solve concurrent problems. Often the only way to get the program to work is to force several independent activities to be controlled by the same language thread or process. This leads to an inevitable loss of clarity, and makes the programs subject to complex and irreproducible interference errors. “ We see that the mapping from Erlang's architecture to neural networks is 1:1, as shown in Fig-5.1 .

From the figure it becomes obvious that indeed, there is a perfect correlation between the architecture of this programming language, and the NN problem do- main. In the figure each neuron is directly mapped to a process, each connection


between the neurons is a connection between processes. Every signal, simulated action potential that is sent from one neuron to another is a signal, a message in vector/list or tuple form, from one process to another. We could not have hoped for a better mapping.

Erlang was also created with an eye towards scaling to millions of processes working in parallel, so even here we are in great luck, for the future in this field will require vast neural network based systems, w ith millions or even billions of neurons on every computing node. Also, because robotics is such a close field to computa- tional intelligence, the evolved NN based systems will need to be able to interface with the sensors and actuators, with the hardware in which the CI is embedded and which it inhabits; again Erlang is perfect for this, it was made for this, it was cre- ated to interface and interact with varied types of hardware, and it was created such that developing drivers is easy.

Erlang was created to be not just concurrent, but distributed, over the web or any other medium. Thus again, the CI system that needs to be distributed over a number of machines, or over Internet, is achievable through Erlang, in fact it is a natural quality when being written in Erlang.

Fig. 5.1 The mapping from Erlang's architecture to the domain of neural network based CI.

5.5 I've Done This Once Before; And We Are On Our Way

But of course there are other important features, beyond that of scaling and the perfect mapping from the problem to the solution domain. There is also the issue of fault tolerance, the issue of robustness...

5.4 Robustness and Fault Tolerance in Computational Intelligence

It would be rather comical if it were possible for an advanced CI system to be brought down by a single bug. Here again Erlang saves us. This programming language was designed to develop systems that must run forever, that can not be taken offline, even when there is a bug, and even when it absolutely must be fixed. Through supervision trees Erlang allows for processes to monitor each other, and to restore each other. Thus if any element of the neural network crashes, another part, an exoself of the CI system can restore it to the previously functional form. But not only can it restore the CI to a previously functional form in the case of emergency, but it can also allow for the bug to be fixed, and the new updated source to be ran without going offline. It allows for the system to fix itself, to self heal, to recover, to upgrade and evolve. What other programming language offers such features so easily and so effortlessly?


In the following chapters we will develop a neural network based computation- al intelligence system. All these features that are offered by Erlang will make it easy for you and I to create it. If you think that the thus far described type of NN based CI system is unreachable, or impossible... you will be surprised, because we will build it by the end of this book. We will create ALife simulations, we will al- low the NN systems to control the simulated organisms, evolving ability to hunt in a virtual 2d environment, to find food, to trick and bait prey... We will create a system that can evolve NNs which recognize visual patterns in financial data, ana- lyzing the actual charts and plots, rather than simply lists of prices. We will create a universal learning network, a system that can be used to evolve and optimize digital circuits. And all of this will be easily achieved, to a great extent thanks to Erlang.

It is incredible that a single individual can create a system of such complexity, or more precisely, create a system that can evolve intelligent systems of such complexity, all by himself. And you will be that individual, by the end of this vol- ume you will have created such a system, and you will know exactly how to de- velop a neuroevolutionary platform that can evolve general neural networks,


evolved for intelligence, for pattern recognition, for anything you can imagine and apply them to.

There are no requirements for this book, you need only have the most basic ex- perience with Erlang. Everything about neural networks and neuroevolution I'll show you. We'll build this first NN CI system together, I've built one before and so I can guide you. By the time we're done, we'll have built one of the most ad- vanced neuroevolutionary systems currently available in the world. Afterwards, you'll continue your travels on your own, and you'll use what you've learned here to build something that even I can't foretell. So common, what are you waiting for, let's go!

5.6 References

[1] Everett DL (2005). Cultural Constraints on Grammar and Cognition in Piraha Another Look at the Design Features of Human Language. Current Anthropology 46, 621-646.

[2] Dacker B (2000) Concurrent Functional Programming for Telecommunications: A Case Study of Technology Introduction. Masters thesis KTH Royal Institute of Technology Stock- holm.

[3] Joe Armstrong (2003) Making Reliable Distributed Systems in The Presence of Software Er- rors. The Royal Institute of Technology Stockholm, Sweden, (PhD thesis).

[4] Armstrong, J. (2007). Programming Erlang Software for a Concurrent World. Pragmatic Bookshelf. ISBN 9781934356005.

[5] Thompson SJ; Cesarini F (2009) Erlang Programming: A Concurrent Approach to Software Development. Sebastopol, California: O'Reilly Media, Inc. ISBN 978059651818.

[6] Logan M, Merritt E, Carlsson R (2010) Erlang and OTP in Action. Greenwich, CT: Manning Publications. ISBN 9781933988788.

Part II

NEUROEVOLUTION: TAKING THE FIRST STEP

In this part of the book we will develop a simple yet powerful Topology and Weight Evolving Artificial Neural Network (TWEANN) platform. We will devel- op it in Erlang, and I will present all the source code within the chapters. Most functions that we will develop will require some function by function explanation and elaboration, and I will add the description and elaboration of the functions de- veloped through the use of comments within the presented source code. In this manner, you will be able to read the source code and the comments in the same flow. It is important to read the comments, as they make up part of the text, and will assist in the explanation of how the particular functions work, and what it is that they do.

In Chapter-6 we will develop the genotype encoding, the phenotype representa- tion, and the mapping between the two for our NN system. In Chapter-7 we will add a local search algorithm, the stochastic hill climber, and the random restart stochastic hill climbing optimization algorithm, and test our simple optimizable NN on the XOR problem. In Chapter-8 we will expand our system further, devel- op the population_monitor and the genome_mutator, thus transforming our system into a simple, yet already very powerful TWEANN. Finally, in Chapter-9 we will analyze our system, and find a few difficult to notice bugs. This chapter will show how to look for errors in a system such as this, and how easy it is to miss small logical based errors. The evolutionary algorithm tends to route around such small errors, allowing the system to still function and find solutions even in the presence of such errors, and thus hiding them and making them difficult to spot.

All the code developed in the following chapters is available at: https: //github.com/CorticalComputer/Book_NeuroevolutionThroughErlang. This provides the folders containing the source code for each chapter, so that you can follow along and perform the testing, code exploration, and try out examples if you wish so. Or if you wish, you can just look over the code within the text as you read, without performing the demonstrated tests in the console yourself.

Chapter 6 Developing a Feed Forward Neural Network

Abstract In this chapter we discuss how a single artificial neuron processes sig- nals, and how to simulate it. We then develop a single artificial neuron and test its functionality. Having discussed and developed a single neuron, we decide on the NN architecture we will implement, and then develop a genotype constructor, and a mapper from genotype to phenotype. Finally, we then ensure that that our simple NN system works by using a simple sensor and actuator attached to the NN to test its sense-think-act ability.

As we discussed in an earlier chapter, Neural Networks (NN) are directed graphs composed of simple processing elements as shown in Figure-6.1 . Every vertex in such a directed graph is a Neuron, every edge is an outgoing axon and a path along which the neuron sends information to other Neurons. A NN has an in- put layer which is a set of neurons that receive signals from sensors, and an output layer which is a set of neurons that connect to actuators. In a general NN system the sensors can be anything, from cameras, to programs that read from a database and pass that data to the neurons. The Actuators too can range from functions which control motors, to simple programs which print the output signals to the screen. Every neuron processes its incoming signals, produces an output signal, and passes it on to other neurons.

Fig. 6.1 A simple Neural Network.

Whether the NN does something intelligent or useful is based on its topology and parameters. The method of modifying the NN topology and parameters to make it do something useful, is the task of its learning algorithm. A learning algo- rithm can be supervised, like in the case of the error back propagation learning al- gorithm, or it can be unsupervised like in the evolutionary or reinforcement learn- ing algorithms. In a supervised learning algorithm the outputs of the NN need to

DOI 10.1007/978-1-4614- 4463 - 3_6 , © Springer Science+Business Media New York 2013


be known in advance, such that corrections can be given to the NN based on the differences in its produced outputs and the correct outputs. Once we have mini- mized the differences between the answers we want and the answers the NN gives, we apply the NN to a new set of data, to another problem in the same field but one which the NN has not encountered during its training. In the case of unsupervised learning, it is only important to be able to tell whether one NN system performs better than another. There is no need to know exactly how the problem should be solved, the NNs will try to figure that out for themselves; the researcher only needs to choose the neural networks that produce better results over those that do not. We will develop these types of systems in future sections.

In this chapter we will learn how to program a static neural network system whose topological and parametric properties are specified during its creation, and are not changed during training. We will develop a genotype encoding for a sim- ple monolithic Neural Network, and then we'll create a mapper program which converts the NN genotype to its phenotypic representation. The process of modify- ing these weights, parameters, and the NN topology is the job of a learning algo- rithm, the subject that we will cover in the chapters that follow.

In the following sections when we discuss genotypes and phenotypes, we mean their standard definitions: a genotype is the organism's full hereditary information, which is passed to offspring in mutated or unchanged form, and the phenotype is the organism's actual observed properties, its morphology and behavior. The pro- cess of mapping a genotypical representation of the organism to the phenotypical one is done through a process called development, to which we also will refer to as: mapping. A genotype of the organism is the form in which we store it in our database, on the other hand its phenotype is its representation and behavior when the organism, a NN in our case, is live and functioning. In the NN system that we build in this chapter, the genotype will be a list of tuples, and the phenotype a graph of interconnected processes sending and receiving messages from one an- other.

                • Note********

The encoding of a genotype itself can either be direct, or indirect. A direct encoding is one in which the genotype encodes every topological and parametric aspect of the NN phenotype in a one to one manner, the genotype and the phenotype can be considered one and the same. An in- direct encoding applies a set of programs, or functions to the genotype, through which the phe- notype is developed. This development process can be highly complex and stochastic in nature which takes into consideration the environmental factors during the time of development, and producing a one too many mapping from a genotype to the phenotype. An example of a direct encoding is that of a bit string which maps to a colored strip in which the 0s are directly con- verted to white sections and 1s to black. An example of an indirect encoding is the case of DNA, where the development from the genotype to a phenotype is a multi-stage process, with complex interactions between the developing organism and the environment it is in. ********************

6.1 Simulating A Neuron

We will now slowly build up a NN system, from a single neuron, to a fully functional feed forward neural network. In the next section we take our first step and develop an artificial neuron using Erlang.


Let us again briefly review the representation and functionality of a single arti- ficial neuron, as shown in Figure-6.2 . A neuron is but a simple processing element which accepts input signals, weighs the importance of each signal by multiplying it by a weight associated with it, adds a bias to the result, applies an activation function to this sum, and then forwards the result to other elements it is connected to. As an example, assume we have a list of input signals to the neuron: [I1,I2,I3,I4], this input is represented as a vector composed of 4 elements. The neuron then must have a list of weights, one weight for every incoming signal: [W1,W2,W3,W4]. We weigh each signal with its weight by taking a dot product of the input vector and the weight vector as follows: Dot_Product = I1*W1 + I2*W2 + I3*W3 + I4*W4. If the neuron also has a threshold value or bias, we simply add this bias value to the Dot_Product. Finally, we apply the activation

Fig. 6.2 An artificial Neuron.


function to the dot product to produce the final output of the neuron: Output = Ac- tivation_Function(Dot_Product), and for a neuron that also has a bias: Output = Activation_Function(Dot_Product + Bias). A bias is an extra floating point pa- rameter not associated with any particular incoming signal, and it adds a level of tunable asymmetry to the activation function.

Mathematically, the neuron that uses a set of weights and a bias is equivalent to a neuron that accepts an “extended input vector ” and uses an “extended weight vector ” to weigh the signals. An extended input vector has “1 ” appended to the in- put vector and an extended weight vector has the bias appended to the weight vec- tor. Using the extended vectors, we then take a single dot product as follows: [I1,I2,I3,I4,1]dot[W1,W2,W3,W4,Bias]= (I1*W1) +(I2*W2) +(I3*W3) +(I4*W4) +(1*Bias), which is equal to the dot product of the input and weight vector, plus the bias as before. Neurons that do not use a bias would simply not append the ex- tension to the input, and thus produce the dot product without a bias value.

Lets simulate and test a very simple neuron, which we will represent using a process. The neuron will have a predetermined number of weights, 2, and it will include a bias. With 2 wights, this neuron can process input vectors of length 2. The activation function will be the standard sigmoid function, in our neuron it's approximated by the hyperbolic tangent (tanh) function included in the math mod- ule. The architecture of this neuron will be the same as in Figure-6.2 .

In the following algorithm, we spawn a process to represent our Neuron, and register it so that we can send and receive signals from it. We use a simple remote procedure call function called ‘sense' to send signals to the registered neuron, and then receive the neuron's output.

simple_neuron.erl

-module(simple_neuron).

-compile(export_all).

create()->

Weights = [random:uniform()-0.5,random:uniform()-0.5,random:uniform()-0.5],

register(neuron, spawn(?MODULE,loop,[Weights])).

%The create function spawns a single neuron, where the weights and the bias are generated randomly to be between -0.5 and 0.5.

loop(Weights) ->

receive

{From, Input} ->

io:format( “****Processing****~n Input:~p~n Using

Weights:~p~n ”,[Input,Weights]),

Dot_Product = dot(Input,Weights,0),

Output = [math:tanh(Dot_Product)],


From ! {result,Output},

loop(Weights)

end.

%The spawned neuron process accepts an input vector, prints it and the weight vector to the screen, calculates the output, and then sends the output to the contacting process. The output is also a vector of length one.

dot([I|Input],[W|Weights],Acc) ->

dot(Input,Weights,I*W+Acc);

dot([],[Bias],Acc)->

Acc + Bias.

%The dot product function that we use works on the assumption that the bias is incorporated in- to the weight list as the last value in that list. After calculating the dot product, the input list will empty out while the weight list will still have the single bias value remaining, which we then add to the accumulator.

sense(Signal)->

case is_list(Signal) and (length(Signal) == 2) of

true->

neuron ! {self(),Signal},

receive

{result,Output}->

io:format( “ Output: ~p~n ”,[Output])

end;

false->

io:format( “The Signal must be a list of length 2~n ”)

end.

%We use the sense function to contact the neuron and send it an input vector. The sense func- tion ensures that the signal we are sending is a vector of length 2.

Now let's compile and test our module:

1> c(simple_neuron).

{ok,simple_neuron}

2> simple_neuron:create().

true.

3> simple_neuron:sense([1,2]).

        • Processing****

Input:[1,2]

Using Weights:[0.44581636451986995,0.0014907142064750634, -0.18867324519560702]

Output: [0.25441202264242263]


It works! We can expand this neuron further by letting it accept signals only from certain predetermined list of PIds, and then output the result not back to those same processes, but instead to another set of PIds. With such modifications this neuron could then be used as a fully functional processing element in a NN. In the next section we will build a single neuron neural network that uses such pro- cessing element.

6.2 A One Neuron Neural Network

Next we will create the simplest possible NN. Our NN topology will be com- posed of a single Neuron which receives a signal from a Sensor, calculates an out- put based on its weights and activation function, and then passes that output signal to the Actuator. This topology and architecture is shown in Figure-6.3 . You will also notice that there is a 4 element called Cortex. This element is used to trigger the sensor to start producing sensory data, and it also contains the PIds of all the processes in the system so that it can be used to shut down the NN when we are done with it. Finally, this type of element can also be used as a supervisor of the NN, and play a role in the NN's synchronization with the learning algorithm. The- se features will become important when we start developing the more complex NN systems in the chapters that follow.

Fig. 6.3 One Neuron Neural Network.

th


To create this system, we will need to significantly modify the functions in our simple_neuron module, and add new features as shown in the following source code:

simplest_nn.erl

-module(simplest_nn).

-compile(export_all).

create() ->

Weights = [random:uniform()-0.5,random:uniform()-0.5,random:uniform()-0.5],

N_PId = spawn(?MODULE,neuron,[Weights,undefined,undefined]),

S_PId = spawn(?MODULE,sensor,[N_PId]),

A_PId = spawn(?MODULE,actuator,[N_PId]),

N_PId ! {init,S_PId,A_PId},

register(cortex,spawn(?MODULE,cortex,[S_PId,N_PId,A_PId])).

%The create function first generates 3 weights, with the 3rd weight being the Bias. The Neuron is spawned first, and is then sent the PIds of the Sensor and Actuator that it's connected with. Then the Cortex element is registered and provided with the PIds of all the elements in the NN system.

neuron(Weights,S_PId,A_PId) ->

receive

{S_PId, forward, Input} ->

io:format( “****Thinking****~n Input:~p~n with

Weights:~p~n ”,[Input,Weights]),

Dot_Product = dot(Input,Weights,0),

Output = [math:tanh(Dot_Product)],

A_PId ! {self(), forward, Output},

neuron(Weights,S_PId,A_PId);

{init, New_SPId, New_APId} ->

neuron(Weights,New_SPId,New_APId);

terminate ->

ok

end.

%After the neuron finishes setting its SPId and APId to that of the Sensor and Actuator respec- tively, it starts waiting for the incoming signals. The neuron expects a vector of length 2 as in-

put, and as soon as the input arrives, the neuron processes the signal and passes the output vec-

tor to the outgoing APId.

dot([I|Input],[W|Weights],Acc) ->

dot(Input,Weights,I*W+Acc);

dot([],[],Acc)->


dot([],[Bias],Acc)->

Acc + Bias.

%The dot function takes a dot product of two vectors, it can operate on a weight vector with and without a bias. When there is no bias in the weight list, both the Input vector and the Weight vector are of the same length. When Bias is present, then when the Input list empties out, the Weights list still has 1 value remaining, its Bias.

sensor(N_PId) ->

receive

sync ->

Sensory_Signal = [random:uniform(),random:uniform()],

io:format( “****Sensing****:~n Signal from the environment

~p~n ”,[Sensory_Signal]),

N_PId ! {self(),forward,Sensory_Signal},

sensor(N_PId);

terminate ->

ok

end.

%The Sensor function waits to be triggered by the Cortex element, and then produces a random vector of length 2, which it passes to the connected neuron. In a proper system the sensory sig- nal would not be a random vector but instead would be produced by a function associated with the sensor, a function that for example reads and vector-encodes a signal coming from a GPS attached to a robot.

actuator(N_PId) ->

receive

{N_PId,forward,Control_Signal}->

pts(Control_Signal),

actuator(N_PId);

terminate ->

ok

end.

pts(Control_Signal)->

io:format( “****Acting****:~n Using:~p to act on environ-

ment.~n ”,[Control_Signal]).

%The Actuator function waits for a control signal coming from a Neuron. As soon as the signal arrives, the actuator executes its function, pts/1, which prints the value to the screen.


cortex(Sensor_PId,Neuron_PId,Actuator_PId)->

receive

sense_think_act ->

Sensor_PId ! sync,

cortex(Sensor_PId,Neuron_PId,Actuator_PId);

terminate ->

Sensor_PId ! terminate,

Neuron_PId ! terminate,

Actuator_PId ! terminate,

ok

end.

%The Cortex function triggers the sensor to action when commanded by the user. This process also has all the PIds of the elements in the NN system, so that it can terminate the whole system when requested.

Lets compile and try out this system:

1>c(simplest_nn).

{ok,simplest_nn}

2>simplest_nn:create().

true

3> cortex ! sense_think_act.

        • Sensing****:

Signal from the environment [0.09230089279334841,0.4435846174457203]

sense_think_act

        • Thinking****

Input:[0.09230089279334841,0.4435846174457203]

with Weights:[-0.4076991072066516,-0.05641538255427969,0.2230402056221108]

        • Acting****:

Using:[0.15902302907693572] to act on environment.

It works! But though this system does embody many important features of a re- al NN, it is still rather useless since it's composed of a single neuron, the sensor produces random data, and the NN has no learning algorithm so we can not teach it to do something useful. In the following sections we are going to design a NN system for which we can specify different starting topologies, for which we can specify sensors and actuators, and which will have the ability to learn to accom- plish useful tasks.


6.3 Planning Our Neural Network System's Architecture

A standard Neural Network (NN) is a graph of interconnected Neurons, where every neuron can send and receive signals from other neurons and/or sensors and actuators. The simplest of NN architectures is that of a monolithic feed forward neural network (FFNN), as shown in Figure-6.4 . In a FFNN, the signals only propagate in the forward direction, from sensors, through the neural layers, and fi- nally reaching the actuators which use the output signals to act on the environ- ment. In such a NN system there are no recursive or cyclical connections. After the Actuators have acted upon the environment, the sensors once again produce and send sensory signals to the neurons in the first layer, and the “Sense-Think- Act ” cycle repeats.

Fig. 6.4 A Feed Forward Neural Network.

Every neuron must be able to accept a vector input of length 1+, and produce a vector output of length 1. Since all neural inputs and outputs are in vector form, and the sensory signals sent from the sensors are also in vector form, the neurons neither need to know nor care whether the incoming signals are coming from other neurons or sensors. Let's take a closer look at the two types of connections that occur in a NN, the [neuron|sensor]-to-neuron and the neuron-to-actuator connec- tion as shown in Figure-6.5 .


Fig. 6.5 Neuron/Sensor-To-Neuron & Neuron-To-Actuator connections.

Every input signal to a neuron is a list of values [I1...In], a vector of length 1 or greater. The neuron's output signal is also a vector, a list of length 1, [O]. Because each Neuron outputs a vector of length 1, the actuators accumulate the signals coming from the Neurons into properly ordered vectors of length 1+. The order of values in the vector is the same as the order of PIds in its fanin pid list. Once the actuator has finished gathering the signals coming from all the neurons connected to it, it uses the accumulated vector as a parameter to its actuation function.

Once all the neurons in the output layer have produced and forwarded their sig- nals to actuators, the NN can start accepting new sensory inputs again (*Note* It is possible for a NN to process multiple sensory input vectors, one after the other, rather than one at a time and waiting until an output vector is produced before ac- cepting a new wave of sensory vectors. This would be somewhat similar to the way a multi-stage pipeline in a CPU works, with every neural layer in the NN pro- cessing signals at the same time, as opposed to the processing of sensory vectors


propagating from first to last layer, one set of sensory input vectors at a time.) This Sense-Think-Act cycle requires some synchronization, especially if a learning algorithm is also present. This synchronization will be done using the Cortex ele- ment we've briefly discussed earlier. We will recreate a more complex version of the Cortex program which will synchronize the sensors producing sensory signals, the actuators gathering the output vectors from the NN's output layer, and the learning algorithm modifying the weight parameters of the NN and allowing the system to learn.

Putting all this information and elements together, our Neural Network will function as follows: The sensor programs poll/request input signals from the envi- ronment, and then preprocess and fan out these sensory signals to the neurons in the first layer. Eventually the neurons in the output layer produce signals that are passed to the actuator program(s). Once an actuator program receives the signals from all the neurons it is connected from, it post-processes these signals and then acts upon the environment. A sensor program can be anything that produces sig- nals, either by itself (random number generator) or as a result of interacting with the environment, like a camera, an intrusion detection system, or a program that simply reads from a database and passes those values to the NN for example. An

Fig. 6.6 All the elements of a NN system.


actuator program is any program that accepts signals and then acts upon the envi- ronment based on those signals. For example, a robot actuator steering program can accept a single floating point value, post process the value so that its range is from -1 to 1, and then execute the motor driver using this value as the parameter, where the sign and magnitude of the parameter designates which way to steer and how hard. Another example actuator is one that accepts signals from the NN, and then buys or sells a stock based on that signal, with a complementary sensor which reads the earlier price values of the same stock. This type of NN system architec- ture is visually represented in Figure-6.6 .

A sensor, actuator, neuron, and the cortex are just 4 different types of processes that accept signals, process them, and execute some kind of element specific func- tion. Lets discuss every one of these processes in detail, to see what information we might need to create them in their genotypic and phenotypic form.

Sensor: A sensor is any process that produces a vector signal that the NN then processes. This signal can be produced from the sensor interacting with the envi- ronment, for example the data coming from a camera, or from the sensor some- how generating the signal internally.

Actuator: An actuator is a process that accepts signals from the Neurons in the output layer, orders them into a vector, and then uses this vector to control some function that acts on the environment or even the NN itself. An actuator might have incoming connections from 3 Neurons, in which case it would then have to wait until all 3 of the neurons have sent it their output signals, accumulate these signals into a vector, and then use this vector as a parameter to its actuation func- tion. The function could for example dictate the voltage signal to be sent to a servo that controls a robot's gripper.

Neuron: The neuron is a signal processing element. It accepts signals, accumu- lates them into an ordered vector, then processes this input vector to produce an output, and finally passes the output to other elements it is connected to. The Neu- ron never interacts with the environment directly, and even when it does receive signals and produces output signals, it does not know whether these input signals are coming from sensors or neurons, or whether it is sending its output signals to other neurons or actuators. All the neuron does is have a list of input PIds from which it expects to receive signals, a list of output PIds to which the neuron sends its output, a weight list correlated with the input PIds, and an activation function it applies to the dot product of the input vector and its weight vector. The neuron waits until it receives all the input signals, processes those signals, and then passes the output onwards.

Cortex: The cortex is a NN synchronizing element. It needs to know the PId of every sensor and actuator, so that it will know when all the actuators have re- ceived their control inputs, and that it's time for the sensors to again gather and


fanout sensory data to the neurons in the input layer. At the same time, the Cortex element can also act as a supervisor of all the Neuron, Sensor, and Actuator ele- ments in the NN system.

Now that we know how these elements should work and process signals, we need to come up with an encoding which can be used to store any type of NN to- pology in a database, or a flat file. This stored representation of the NN is its geno- type. We should be able to specify the topology and the parameters of the NN within the genotype, and then generate from it a process based NN system, the phenotype. Using a genotype also allows us to train a NN to do something useful, and then save the updated and trained NN to a file for later use. Finally, once we decide to use an evolutionary learning algorithm, the NN genotypes are what the mutation operators will be applied to, and from what the mutated offspring will be generated.

In the next section we will develop a simple, human readable, and tuple based genotype encoding for our NN system. This type of encoding will be easy to un- derstand, work with, and easy to encode and operate on using standard directional graph based functions. The use of such a direct way to store the genotype will also make it easy to think about it, and thus to advance, scale, and utilize it in the more advanced systems we'll develop in the future.

6.4 Developing a Genotype Representation

There are a number of ways to encode the genotype of a monolithic Neural Network (NN). Since NNs are directed graphs, we could simply use Erlang's di- graph module. The digraph module in particular has functions with which to cre- ate Nodes/Neurons, Edges/Connections between the nodes, and even sub graphs, thus easily allowing us to develop modular topologies. Another simple way to en- code the genotype is by representing the NN as a list of tuples, where every tuple is a record representing either a Neuron, Sensor, Actuator, or the Cortex element. Finally, we could also use a hash table, ets for example, instead of a simple list to store the tuples.

In every one of these cases, every element in the genotype is encoded as a hu- man readable tuple. Our records will directly reflect the information that would be included and needed by every process in the phenotype. The 4 elements can be represented using the following records:

Sensor: -record(sensor, {id, cx_id, name, vl, fanout_ids}).


The sensor id has the following format: {sensor, UniqueVal} . cx_id is the Id of the Cortex element. ‘name' is the name of the function the sensor executes to gen- erate or acquire the sensory data, and vl is the vector length of the produced senso- ry signal. Finally, fanout_ids is a list of neuron ids to which the sensory data will be fanned out.

Actuator: -record(actuator, {id, cx_id, name, vl, fanin_ids}).

The actuator id has the following format: {actuator, UniqueVal} . cx_id is the the Id of the Cortex element. ‘ name' is the name of the function the actuator exe- cutes to act upon the environment, with the function parameter being the vector it accumulates from the incoming neural signals. ‘ vl' is the vector length of the ac- cumulated actuation vector. Finally, the fanin_ids is a list of neuron ids which are connected to the actuator.

Neuron: -record(neuron, {id, cx_id, af, input_idps, output_ids}).

A neuron id uses the following format: {neuron,{LayerIndex, UniqueVal}} . cx_id is the the Id of the Cortex element. The activation function, af , is the name of the function the neuron uses on the extended dot product (dot product plus bi- as). The activation function that we will use in the simple NN we design in this chapter will be ‘tanh', later we will extend the list of available activation functions our NNs can use. ‘ input_idps' stands for Input Ids “Plus ”, which is a list of tuples as follows: [{Id1,Weights1} … {IdN,WeightsN},{bias,Val}] . Each tuple is com- posed of the Id of the element that is connected to the neuron, and weights corre- lated with the input vector coming from the neuron with the listed Id. The last tu- ple in the input_idps is {bias,Val}, which is not associated with any incoming signal, and represents the Bias value. Finally, output_ids is a list of Ids to which the neuron will fanout its output signal.

Cortex: -record(cortex, {id, sensor_ids, actuator_ids, nids}).

The cortex Id has the following format: {cortex, UniqueVal} . ‘ sensor_ids' is a list of sensor ids that produce and pass the sensory signals to the neurons in the in- put layer. ‘ actuator_ids' is a list of actuator ids that the neural output layer is con- nected to. When the actuator is done affecting the environment, it sends the cortex a synchronization signal. After the cortex receives the sync signal from all the ids in its actuator_ids list, it triggers all the sensors in the sensor_ids list. Finally, nids is the list of all neuron ids in the NN.

Figure-6.7 shows the correlation between the tuples/records and the process based phenotypic representations to which they map. Using this record representa- tion in our genotype allows us to easily and safely store all the information of our NN. We need only decide whether to use a digraph, a hash table, or a simple list to


store the Genotype of a NN. Because we will be building a very simple Feed For- ward Neural Network in this chapter, let us start by using a simple list. For the more advanced evolutionary NN systems that we'll build in the later chapters, we will switch to an ETS or a Digraph representation.

Fig. 6.7 Record to process correlation.

In the next section we will develop a program which accepts high level specifi- cation parameters of the NN genotype we wish to construct, and which outputs the genotype represented as a list of tuples. We will then develop a mapping function which will use our NN genotype to create a process based phenotype, which is the actual NN system that senses, thinks, and takes action based on its sensory signals and neural processing.

6.5 Programming the Genotype Constructor

Now that we've decided on the necessary elements and their genotypic repre- sentation in our NN system, we need to create a program that accepts as input the high level NN specification parameters, and produces the genotype as output.


When creating a NN, we need to be able to specify the sensors it will use, the ac- tuators it will use, and the general NN topology. The NN topology specification should state how many layers and how many neurons per layer the feed forward NN will have. Because we wish to keep this particular NN system very simple, we will only require that the genotype constructor is able to generate NNs with a sin- gle sensor and actuator. For the number of layers and layer densities of the NN, all the information can be contained in a single LayerDensities list as shown in Fig- ure-6.8 . Thus, our genotype constructor should be able to construct everything from a parameter list composed of a sensor name, an actuator name, and a LayerDensities list. The LayerDensities parameter will actually only specify the hidden layer densities, where the hidden LayerDensities are all the non output lay- er densities. The output layer density will be calculated from the vector length of the actuator. An empty HiddenLayerDensities list implies that the NN will only have a single neural layer, whose density is equal to the actuator's vector length.

Fig. 6.8 A NN composed of 3 layers, with a [3, 2, 2] layer density pattern.

For example, a genotype creating program which accepts (SensorName,ActuatorName,[1,3]) as input, where the sensor vector length is 3 and the actuator vector length is 1, should produce a NN with 3 layers, whose out- put layer has 1 neuron, as shown in Figure-6.9 . The input layer will have a single neuron which has 3 weights and a bias, so that the neurons in the first layer can process input vectors of length 3 coming from the sensor. The output layer has a single neuron, due to actuator's vl equaling 1.


Fig. 6.9 Genotype with: LayerDensities == [1,3,1], and HiddenLayerDensities == [1,3].

We first create a file to contain the records representing each element we'll use: records.hrl

-record(sensor, {id, cx_id, name, vl, fanout_ids}).

-record(actuator,{id, cx_id, name, vl, fanin_ids}).

-record(neuron, {id, cx_id, af, input_idps, output_ids}).

-record(cortex, {id, sensor_ids, actuator_ids, nids}).

Now we develop an algorithm that constructs the genotype of a general feed forward NN based on the provided sensor name, actuator name, and the hidden layer densities parameter:

constructor.erl

-module(constructor).

-compile(export_all).

-include( “records.hrl ”).

construct_Genotype(SensorName,ActuatorName,HiddenLayerDensities)->

construct_Genotype(ffnn,SensorName,ActuatorName,HiddenLayerDensities).

construct_Genotype(FileName,SensorName,ActuatorName,HiddenLayerDensities)->

S = create_Sensor(SensorName),

A = create_Actuator(ActuatorName),

Output_VL = A#actuator.vl,

LayerDensities = lists:append(HiddenLayerDensities,[Output_VL]),

Cx_Id = {cortex,generate_id()},

Neurons = create_NeuroLayers(Cx_Id,S,A,LayerDensities),

[Input_Layer|_] = Neurons,

[Output_Layer|_] = lists:reverse(Neurons),

FL_NIds = [N#neuron.id || N <- Input_Layer],


LL_NIds = [N#neuron.id || N <- Output_Layer],

NIds = [N#neuron.id || N <- lists:flatten(Neurons)],

Sensor = S#sensor{cx_id = Cx_Id, fanout_ids = FL_NIds},

Actuator = A#actuator{cx_id=Cx_Id,fanin_ids = LL_NIds},

Cortex = create_Cortex(Cx_Id,[S#sensor.id],[A#actuator.id],NIds),

Genotype = lists:flatten([Cortex,Sensor,Actuator|Neurons]),

{ok, File} = file:open(FileName, write),

lists:foreach(fun(X) -> io:format(File, “~p.~n ”,[X]) end, Genotype),

file:close(File).

%The construct_Genotype function accepts the name of the file to which we'll save the geno- type, sensor name, actuator name, and the hidden layer density parameters. We have to generate unique Ids for every sensor and actuator. The sensor and actuator names are used as input to the create_Sensor and create_Actuator functions, which in turn generate the actual Sensor and Ac- tuator representing tuples. We create unique Ids for sensors and actuators so that when in the future a NN uses 2 or more sensors or actuators of the same type, we will be able to differenti- ate between them using their Ids. After the Sensor and Actuator tuples are generated, we extract the NN's input and output vector lengths from the sensor and actuator used by the system. The Input_VL is then used to specify how many weights the neurons in the input layer will need, and the Output_VL specifies how many neurons are in the output layer of the NN. After ap- pending the HiddenLayerDensites to the now known number of neurons in the last layer to gen- erate the full LayerDensities list, we use the create_NeuroLayers function to generate the Neu- ron representing tuples. We then update the Sensor and Actuator records with proper fanin and fanout ids from the freshly created Neuron tuples, compose the Cortex, and write the genotype to file.

create_Sensor(SensorName) ->

case SensorName of

rng ->

  1. sensor{id={sensor,generate_id()},name=rng,vl=2};

_ ->

exit( “System does not yet support a sensor by the

name:~p. ”,[SensorName])

end.

create_Actuator(ActuatorName) ->

case ActuatorName of

pts ->

  1. actuator{id={actuator,generate_id()},name=pts,vl=1};

_ ->

exit( “System does not yet support an actuator by the

name:~p. ”,[ActuatorName])

end.

%Every sensor and actuator uses some kind of function associated with it, a function that either polls the environment for sensory signals (in the case of a sensor) or acts upon the environment (in the case of an actuator). It is the function that we need to define and program before it is


used, and the name of the function is the same as the name of the sensor or actuator itself. For example, the create_Sensor/1 has specified only the rng sensor, because that is the only sensor function we've finished developing. The rng function has its own vl specification, which will determine the number of weights that a neuron will need to allocate if it is to accept this sen- sor's output vector. The same principles apply to the create_Actuator function. Both, cre- ate_Sensor and create_Actuator function, given the name of the sensor or actuator, will return a record with all the specifications of that element, each with its own unique Id.

create_NeuroLayers(Cx_Id,Sensor,Actuator,LayerDensities) ->

Input_IdPs = [{Sensor#sensor.id,Sensor#sensor.vl}],

Tot_Layers = length(LayerDensities),

[FL_Neurons|Next_LDs] = LayerDensities,

NIds = [{neuron,{1,Id}}|| Id <- generate_ids(FL_Neurons,[])],

cre-

ate_NeuroLayers(Cx_Id,Actuator#actuator.id,1,Tot_Layers,Input_IdPs,NIds,Next_LDs,[]). %The function create_NeuroLayers/3 prepares the initial step before starting the recursive cre- ate_NeuroLayers/7 function which will create all the Neuron records. We first generate the place holder Input Ids “Plus ”(Input_IdPs), which are tuples composed of Ids and the vector lengths of the incoming signals associated with them. The proper input_idps will have a weight list in the tuple instead of the vector length. Because we are only building NNs each with only a single Sensor and Actuator, the IdP to the first layer is composed of the single Sensor Id with the vector length of its sensory signal, likewise in the case of the Actuator. We then generate unique ids for the neurons in the first layer, and drop into the recursive create_NeuroLayers/7 function.

cre-

ate_NeuroLayers(Cx_Id,Actuator_Id,LayerIndex,Tot_Layers,Input_IdPs,NIds,[Next_LD|LDs],

Acc) ->

Output_NIds = [{neuron,{LayerIndex+1,Id}} || Id <- generate_ids(Next_LD,[])], Layer_Neurons = create_NeuroLayer(Cx_Id,Input_IdPs,NIds,Output_NIds,[]), Next_InputIdPs = [{NId,1}|| NId <- NIds],

cre-

ate_NeuroLayers(Cx_Id,Actuator_Id,LayerIndex+1,Tot_Layers,Next_InputIdPs,Output_NIds,

LDs,[Layer_Neurons|Acc]);

create_NeuroLayers(Cx_Id,Actuator_Id,Tot_Layers,Tot_Layers,Input_IdPs,NIds,[],Acc) ->

Output_Ids = [Actuator_Id],

Layer_Neurons = create_NeuroLayer(Cx_Id,Input_IdPs,NIds,Output_Ids,[]),

lists:reverse([Layer_Neurons|Acc]).

%During the first iteration, the first layer neuron ids constructed in create_NeuroLayers/3 are held in the NIds variable. In create_NeuroLayers/7, with every iteration we generate the Out- put_NIds, which are the Ids of the neurons in the next layer. The last layer is a special case which occurs when LayerIndex == Tot_Layers. Having the Input_IdPs, and the Output_NIds, we are able to construct a neuron record for every Id in NIds using the function create_layer/4. The Ids of the constructed Output_NIds will become the NIds variable of the next iteration, and the Ids of the neurons in the current layer will be extended and become Next_InputIdPs. We


then drop into the next iteration with the newly prepared Next_InputIdPs and Output_NIds. Fi- nally, when we reach the last layer, the Output_Ids is the list containing a single Id of the Actu- ator element. We use the same function, create_NeuroLayer/4, to construct the last layer and re- turn the result.

create_NeuroLayer(Cx_Id,Input_IdPs,[Id|NIds],Output_Ids,Acc) ->

Neuron = create_Neuron(Input_IdPs,Id,Cx_Id,Output_Ids),

create_NeuroLayer(Cx_Id,Input_IdPs,NIds,Output_Ids,[Neuron|Acc]);

create_NeuroLayer(_Cx_Id,_Input_IdPs,[],_Output_Ids,Acc) ->

Acc.

%To create neurons from the same layer, all that is needed are the Ids for those neurons, a list of Input_IdPs for every neuron so that we can create the proper number of weights, and a list of Output_Ids. Since in our simple feed forward neural network all neurons are fully connected to the neurons in the next layer, the Input_IdPs and Output_Ids are the same for every neuron be- longing to the same layer.

create_Neuron(Input_IdPs,Id,Cx_Id,Output_Ids)->

Proper_InputIdPs = create_NeuralInput(Input_IdPs,[]),

  1. neuron{id=Id,cx_id =

Cx_Id,af=tanh,input_idps=Proper_InputIdPs,output_ids=Output_Ids}.

create_NeuralInput([{Input_Id,Input_VL}|Input_IdPs],Acc) ->

Weights = create_NeuralWeights(Input_VL,[]),

create_NeuralInput(Input_IdPs,[{Input_Id,Weights}|Acc]);

create_NeuralInput([],Acc)->

lists:reverse([{bias,random:uniform()-0.5}|Acc]).

create_NeuralWeights(0,Acc) ->

Acc;

create_NeuralWeights(Index,Acc) ->

W = random:uniform()-0.5,

create_NeuralWeights(Index-1,[W|Acc]).

%Each neuron record is composed by the create_Neuron/3 function. The create_Neuron/3 func- tion creates the Input list from the tuples [{Id,Weights}...] using the vector lengths specified in the place holder Input_IdPs. The create_NeuralInput/2 function uses create_NeuralWeights/2 to generate the random weights in the range of -0.5 to 0.5, adding the bias to the end of the list.

generate_ids(0,Acc) ->

Acc;

generate_ids(Index,Acc)->

Id = generate_id(),

generate_ids(Index-1,[Id|Acc]).

generate_id() ->

{MegaSeconds,Seconds,MicroSeconds} = now(),


1/(MegaSeconds*1000000 + Seconds + MicroSeconds/1000000).

%The generate_id/0 creates a unique Id using current time, the Id is a floating point value. The generate_ids/2 function creates a list of unique Ids.

create_Cortex(Cx_Id,S_Ids,A_Ids,NIds) ->

  1. cortex{id = Cx_Id, sensor_ids=S_Ids, actuator_ids=A_Ids, nids = NIds}.

%The create_Cortex/4 function generates the record encoded genotypical representation of the cortex element. The Cortex element needs to know the Id of every Neuron, Sensor, and Actua-

tor in the NN.

Note that the constructor can only create sensor and actuator records that are specified in the create_Sensor/1 and create_Actuator/1 functions, and it can only create the genotype if it knows the Sensor and Actuator vl parameters. Let us now compile and test our genotype constructing algorithm:

1>c(constructor).

{ok,constructor}.

2>constructor:construct_Genotype(ffnn,rng,pts,[1,3]).

ok

It works! Make sure to open the file to which the Genotype was written (ffnn in the above example), and peruse the generated list of tuples to ensure that all the elements are properly interconnected by looking at their fanin/fanout and in- put/output ids. This list is a genotype of the NN which is composed of 3 feed for- ward neural layers, with 1 neuron in the first layer, 3 in the second, and 1 in the third. The created NN genotype uses the rng sensor and pts actuator. In the next section we will create a genotype to phenotype mapper which will convert inert genotypes of this form, into live phenotypes which can process sensory signals and act on the world using their actuators.

6.6 Developing the Genotype to Phenotype Mapping Module

We've invented a tuple based genotype representation for our Neural Network, and we have developed an algorithm which creates the NN genotypes when pro- vided with 3 high level parameters, SensorName, ActuatorName, and HiddenLayerDensities. But our genotypical representation of the NN is only used as a method of storing it in a database or some file. We now need to create a func- tion that converts the NN genotype, to an active phenotype.

In the previous chapter we have discussed how Erlang, unlike any other lan- guage, is perfect for developing fault tolerant and concurrent NN systems. The NN topology and functionality maps perfectly to Erlang's process based architecture. We now need to design an algorithm that creates a process for every tuple encoded


element (Cortex, Neurons, Actuator, Sensor) stored in the genotype, and then in- terconnects those processes to produce the proper NN topology. This mapping is an example of direct encoding, where every tuple becomes a process, and every connection is explicitly specified in the genotype. The mapping is shown in Fig- ure-6.10 .

Fig. 6.10 A direct genotype to phenotype mapping.

In our genotype to phenotype direct mapping, we first spawn every element to create a correlation from Ids to their respective process PIds, and then initialize every process's state using the information in its correlated record. But to get these processes to communicate, we still need to standardize the messages they will ex- change between each other.

Because we want our neurons to be ambivalent to whether the signal is coming from another neuron or a sensor, all signal vector messages must be of the same form. We can let the messages passed from sensors and neurons to other neurons and actuators use the following form: {Sender_PId, forward, Signal_Vector}. The Sender_PId will allow the Neurons to match the Signal_Vector with its appropri- ate weight vector.


Once the actuator has accumulated all the incoming neural signals, it should be able to notify the cortex element of this, so that the cortex can trigger the sensor processes to poll for new sensory data. The actuators will use the following mes- sages for this task: {Actuator_PId,sync}. Once the cortex has received the sync messages from all the actuators connected to its NN, it will trigger all the sensors using a messages of the form: {Cx_PId,sync}. Finally, every element other than the cortex will also accept a message of the form: {Cx_PId,terminate}. The cortex itself should be able to receive the simple ‘terminate' message. In this manner we can request that the cortex terminates all the elements in the NN it oversees, and then terminates itself.

Now that we know what messages the processes will be exchanging, and how the phenotype is represented, we can start developing the cortex, sensor, actuator, neuron, and the phenotype constructor module we'll call exoself. The ‘exoself' module will not only contain the algorithm that maps the genotype to phenotype, but also a function that maps the phenotype back to genotype. The phenotype to genotype mapping is a backup procedure, which will allow us to backup pheno- types that have learned something new, back to the database.

We now create the cortex module:

cortex.erl

-module(cortex).

-compile(export_all).

-include( “records.hrl ”).

gen(ExoSelf_PId,Node)->

spawn(Node,?MODULE,loop,[ExoSelf_PId]).

loop(ExoSelf_PId) ->

receive

{ExoSelf_PId,{Id,SPIds,APIds,NPIds},TotSteps} ->

[SPId ! {self(),sync} || SPId <- SPIds],

loop(Id,ExoSelf_PId,SPIds,{APIds,APIds},NPIds,TotSteps)

end.

%The gen/2 function spawns the cortex element, which immediately starts to wait for a the

state message from the same process that spawned it, exoself. The initial state message contains the sensor, actuator, and neuron PId lists. The message also specifies how many total Sense- Think-Act cycles the Cortex should execute before terminating the NN system. Once we im- plement the learning algorithm, the termination criteria will depend on the fitness of the NN, or some other useful property

loop(Id,ExoSelf_PId,SPIds,{_APIds,MAPIds},NPIds,0) ->

io:format( “Cortex:~p is backing up and terminating.~n ”,[Id]),

Neuron_IdsNWeights = get_backup(NPIds,[]),

ExoSelf_PId ! {self(),backup,Neuron_IdsNWeights},


[PId ! {self(),terminate} || PId <- SPIds],

[PId ! {self(),terminate} || PId <- MAPIds],

[PId ! {self(),termiante} || PId <- NPIds];

loop(Id,ExoSelf_PId,SPIds,{[APId|APIds],MAPIds},NPIds,Step) ->

receive

{APId,sync} ->

loop(Id,ExoSelf_PId,SPIds,{APIds,MAPIds},NPIds,Step);

terminate ->

io:format( “Cortex:~p is terminating.~n ”,[Id]),

[PId ! {self(),terminate} || PId <- SPIds],

[PId ! {self(),terminate} || PId <- MAPIds],

[PId ! {self(),termiante} || PId <- NPIds]

end;

loop(Id,ExoSelf_PId,SPIds,{[],MAPIds},NPIds,Step)->

[PId ! {self(),sync} || PId <- SPIds],

loop(Id,ExoSelf_PId,SPIds,{MAPIds,MAPIds},NPIds,Step-1).

%The cortex's goal is to synchronize the NN system such that when the actuators have received all their control signals, the sensors are once again triggered to gather new sensory information. Thus the cortex waits for the sync messages from the actuator PIds in its system, and once it has received all the sync messages, it triggers the sensors and then drops back to waiting for a new set of sync messages. The cortex stores 2 copies of the actuator PIds: the APIds, and the MemoryAPIds (MAPIds). Once all the actuators have sent it the sync messages, it can restore the APIds list from the MAPIds. Finally, there is also the Step variable which decrements every time a full cycle of Sense-Think-Act completes, once this reaches 0, the NN system begins its termination and backup process.

get_backup([NPId|NPIds],Acc)->

NPId ! {self(),get_backup},

receive

{NPId,NId,WeightTuples}->

get_backup(NPIds,[{NId,WeightTuples}|Acc])

end;

get_backup([],Acc)->

Acc.

%During backup, cortex contacts all the neurons in its NN and requests for the neuron's Ids and their Input_IdPs. Once the updated Input_IdPs from all the neurons have been accumulated, the list is sent to exoself for the actual backup and storage.

Now the sensor module:

sensor.erl

-module(sensor).

-compile(export_all).

-include( “records.hrl ”).


gen(ExoSelf_PId,Node)->

spawn(Node,?MODULE,loop,[ExoSelf_PId]).

loop(ExoSelf_PId) ->

receive

{ExoSelf_PId,{Id,Cx_PId,SensorName,VL,Fanout_PIds}} ->

loop(Id,Cx_PId,SensorName,VL,Fanout_PIds)

end.

%When gen/2 is executed it spawns the sensor element and immediately begins to wait for its initial state message.

loop(Id,Cx_PId,SensorName,VL,Fanout_PIds)->

receive

{Cx_PId,sync}->

SensoryVector = sensor:SensorName(VL),

[Pid ! {self(),forward,SensoryVector} || Pid <- Fanout_PIds], loop(Id,Cx_PId,SensorName,VL,Fanout_PIds);

{Cx_PId,terminate} ->

ok

end.

%The sensor process accepts only 2 types of messages, both from the cortex. The sensor can ei- ther be triggered to begin gathering sensory data based on its sensory role, or terminate if the cortex requests so.

rng(VL)->

rng(VL,[]).

rng(0,Acc)->

Acc;

rng(VL,Acc)->

rng(VL-1,[random:uniform()|Acc]).

%'rng' is a simple random number generator that produces a vector of random values, each be- tween 0 and 1. The length of the vector is defined by the VL, which itself is specified within the sensor record.

The actuator module:

actuator.erl

-module(actuator).

-compile(export_all).

-include( “records.hrl ”).

gen(ExoSelf_PId,Node)->

spawn(Node,?MODULE,loop,[ExoSelf_PId]).


loop(ExoSelf_PId) ->

receive

{ExoSelf_PId,{Id,Cx_PId,ActuatorName,Fanin_PIds}} ->

loop(Id,Cx_PId,ActuatorName,{Fanin_PIds,Fanin_PIds},[])

end.

%When gen/2 is executed it spawns the actuator element and immediately begins to wait for its initial state message.

loop(Id,Cx_PId,AName,{[From_PId|Fanin_PIds],MFanin_PIds},Acc) ->

receive

{From_PId,forward,Input} ->

loop(Id,Cx_PId,AName,{Fanin_PIds,MFanin_PIds},lists:append(Input,Acc));

{Cx_PId,terminate} ->

ok

end;

loop(Id,Cx_PId,AName,{[],MFanin_PIds},Acc)->

actuator:AName(lists:reverse(Acc)),

Cx_PId ! {self(),sync},

loop(Id,Cx_PId,AName,{MFanin_PIds,MFanin_PIds},[]).

%The actuator process gathers the control signals from the neurons, appending them to the ac- cumulator. The order in which the signals are accumulated into a vector is in the same order as the neuron ids are stored within NIds. Once all the signals have been gathered, the actuator sends cortex the sync signal, executes its function, and then again begins to wait for the neural signals from the output layer by reseting the Fanin_PIds from the second copy of the list.

pts(Result)->

io:format( “actuator:pts(Result): ~p~n ”,[Result]).

%The pts actuation function simply prints to screen the vector passed to it.

And finally the neuron module:

neuron.erl

-module(neuron).

-compile(export_all).

-include( “records.hrl ”).

gen(ExoSelf_PId,Node)->

spawn(Node,?MODULE,loop,[ExoSelf_PId]).

loop(ExoSelf_PId) ->

receive

{ExoSelf_PId,{Id,Cx_PId,AF,Input_PIdPs,Output_PIds}} ->

loop(Id,Cx_PId,AF,{Input_PIdPs,Input_PIdPs},Output_PIds,0)

end.


%When gen/2 is executed it spawns the neuron element and immediately begins to wait for its initial state message.

loop(Id,Cx_PId,AF,{[{Input_PId,Weights}|Input_PIdPs],MInput_PIdPs},Output_PIds,Acc)->

receive

{Input_PId,forward,Input}->

Result = dot(Input,Weights,0),

loop(Id,Cx_PId,AF,{Input_PIdPs,MInput_PIdPs},Output_PIds,Result+Acc);

{Cx_PId,get_backup}->

Cx_PId ! {self(),Id,MInput_PIdPs},

loop(Id,Cx_PId,AF,{[{Input_PId,Weights}|Input_PIdPs],MInput_PIdPs},Output_PIds,Acc);

{Cx_PId,terminate}->

ok

end;

loop(Id,Cx_PId,AF,{[Bias],MInput_PIdPs},Output_PIds,Acc)->

Output = neuron:AF(Acc+Bias),

[Output_PId ! {self(),forward,[Output]} || Output_PId <- Output_PIds], loop(Id,Cx_PId,AF,{MInput_PIdPs,MInput_PIdPs},Output_PIds,0);

loop(Id,Cx_PId,AF,{[],MInput_PIdPs},Output_PIds,Acc)->

Output = neuron:AF(Acc),

[Output_PId ! {self(),forward,[Output]} || Output_PId <- Output_PIds], loop(Id,Cx_PId,AF,{MInput_PIdPs,MInput_PIdPs},Output_PIds,0).

dot([I|Input],[W|Weights],Acc) ->

dot(Input,Weights,I*W+Acc);

dot([],[],Acc)->

Acc.

%The neuron process waits for vector signals from all the processes that it's connected from, taking the dot product of the input and weight vectors, and then adding it to the accumulator. Once all the signals from Input_PIds are received, the accumulator contains the dot product to which the neuron then adds the bias and executes the activation function on. After fanning out the output signal, the neuron again returns to waiting for incoming signals. When the neuron re- ceives the {Cx_PId,get_backup} message, it forwards to the cortex its full MInput_PIdPs list, and its Id. Once the training/learning algorithm is added to the system, the MInput_PIdPs would contain a full set of the most recent and updated version of the weights.

tanh(Val)->

math:tanh(Val).

%Though in this current implementation the neuron has only the tanh/1 function available to it, we will later extend the system to allow different neurons to use different activation functions.

Now we create the exoself module, which will map the genotype to phenotype, spawning all the appropriate processes. The exoself module will also provide the algorithm for the Cortex element to update the genotype with the newly trained


weights from the phenotype, and in this manner saving the trained and learned NNs for future use.

exoself.erl

-module(exoself).

-compile(export_all).

-include( “records.hrl ”).

map()->

map(ffnn).

map(FileName)->

{ok,Genotype} = file:consult(FileName),

spawn(exoself,map,[FileName,Genotype]).

map(FileName,Genotype)->

IdsNPIds = ets:new(idsNpids,[set,private]),

[Cx|CerebralUnits] = Genotype,

Sensor_Ids = Cx#cortex.sensor_ids,

Actuator_Ids = Cx#cortex.actuator_ids,

NIds = Cx#cortex.nids,

spawn_CerebralUnits(IdsNPIds,cortex,[Cx#cortex.id]),

spawn_CerebralUnits(IdsNPIds,sensor,Sensor_Ids),

spawn_CerebralUnits(IdsNPIds,actuator,Actuator_Ids),

spawn_CerebralUnits(IdsNPIds,neuron,NIds),

link_CerebralUnits(CerebralUnits,IdsNPIds),

link_Cortex(Cx,IdsNPIds),

Cx_PId = ets:lookup_element(IdsNPIds,Cx#cortex.id,2),

receive

{Cx_PId,backup,Neuron_IdsNWeights}->

U_Genotype = update_genotype(IdsNPIds,Genotype,Neuron_IdsNWeights), {ok, File} = file:open(FileName, write),

lists:foreach(fun(X) -> io:format(File, “~p.~n ”,[X]) end, U_Genotype), file:close(File),

io:format( “Finished updating to file:~p~n ”,[FileName])

end.

%The map/1 function maps the tuple encoded genotype into a process based phenotype. The map function expects for the Cx record to be the leading tuple in the tuple list it reads from the FileName. We create an ets table to map Ids to PIds and back again. Since the Cortex element contains all the Sensor, Actuator, and Neuron Ids, we are able to spawn each neuron using its own gen function, and in the process construct a map from Ids to PIds. We then use link_CerebralUnits to link all non Cortex elements to each other by sending each spawned pro- cess the information contained in its record, but with Ids converted to Pids where appropriate. Finally, we provide the Cortex process with all the PIds in the NN system by executing the link_Cortex/2 function. Once the NN is up and running, exoself starts its wait until the NN has finished its job and is ready to backup. When the cortex initiates the backup process it sends exoself the updated Input_PIdPs from its neurons. Exoself uses the update_genotype/3 function


to update the old genotype with new weights, and then stores the updated version back to its file.

spawn_CerebralUnits(IdsNPIds,CerebralUnitType,[Id|Ids])->

PId = CerebralUnitType:gen(self(),node()),

ets:insert(IdsNPIds,{Id,PId}),

ets:insert(IdsNPIds,{PId,Id}),

spawn_CerebralUnits(IdsNPIds,CerebralUnitType,Ids);

spawn_CerebralUnits(_IdsNPIds,_CerebralUnitType,[])->

true.

%We spawn the process for each element based on its type: CerebralUnitType, and the gen function that belongs to the CerebralUnitType module. We then enter the {Id,PId} tuple into our ETS table for later use.

link_CerebralUnits([R|Records],IdsNPIds) when is_record(R,sensor) ->

SId = R#sensor.id,

SPId = ets:lookup_element(IdsNPIds,SId,2),

Cx_PId = ets:lookup_element(IdsNPIds,R#sensor.cx_id,2),

SName = R#sensor.name,

Fanout_Ids = R#sensor.fanout_ids,

Fanout_PIds = [ets:lookup_element(IdsNPIds,Id,2) || Id <- Fanout_Ids],

SPId ! {self(),{SId,Cx_PId,SName,R#sensor.vl,Fanout_PIds}},

link_CerebralUnits(Records,IdsNPIds);

link_CerebralUnits([R|Records],IdsNPIds) when is_record(R,actuator) ->

AId = R#actuator.id,

APId = ets:lookup_element(IdsNPIds,AId,2),

Cx_PId = ets:lookup_element(IdsNPIds,R#actuator.cx_id,2),

AName = R#actuator.name,

Fanin_Ids = R#actuator.fanin_ids,

Fanin_PIds = [ets:lookup_element(IdsNPIds,Id,2) || Id <- Fanin_Ids],

APId ! {self(),{AId,Cx_PId,AName,Fanin_PIds}},

link_CerebralUnits(Records,IdsNPIds);

link_CerebralUnits([R|Records],IdsNPIds) when is_record(R,neuron) ->

NId = R#neuron.id,

NPId = ets:lookup_element(IdsNPIds,NId,2),

Cx_PId = ets:lookup_element(IdsNPIds,R#neuron.cx_id,2),

AFName = R#neuron.af,

Input_IdPs = R#neuron.input_idps,

Output_Ids = R#neuron.output_ids,

Input_PIdPs = convert_IdPs2PIdPs(IdsNPIds,Input_IdPs,[]),

Output_PIds = [ets:lookup_element(IdsNPIds,Id,2) || Id <- Output_Ids],

NPId ! {self(),{NId,Cx_PId,AFName,Input_PIdPs,Output_PIds}},

link_CerebralUnits(Records,IdsNPIds);

link_CerebralUnits([],_IdsNPIds)->

ok.


convert_IdPs2PIdPs(_IdsNPIds,[{bias,Bias}],Acc)->

lists:reverse([Bias|Acc]);

convert_IdPs2PIdPs(IdsNPIds,[{Id,Weights}|Fanin_IdPs],Acc)->

convert_IdPs2PIdPs(IdsNPIds,Fanin_IdPs,

[{ets:lookup_element(IdsNPIds,Id,2),Weights}|Acc]).

%The link_CerebralUnits/2 converts the Ids to PIds using the created IdsNPids ETS table. At this point all the elements are spawned, and the processes are waiting for their initial states. convert_IdPs2PIdPs/3 converts the IdPs tuples into tuples that use PIds instead of Ids, such that the Neuron will know which weights are to be associated with which incoming vector signals. The last element is the bias, which is added to the list in a non tuple form. Afterwards, the list is reversed to take its proper order.

link_Cortex(Cx,IdsNPIds) ->

Cx_Id = Cx#cortex.id,

Cx_PId = ets:lookup_element(IdsNPIds,Cx_Id,2),

SIds = Cx#cortex.sensor_ids,

AIds = Cx#cortex.actuator_ids,

NIds = Cx#cortex.nids,

SPIds = [ets:lookup_element(IdsNPIds,SId,2) || SId <- SIds],

APIds = [ets:lookup_element(IdsNPIds,AId,2) || AId <- AIds],

NPIds = [ets:lookup_element(IdsNPIds,NId,2) || NId <- NIds],

Cx_PId ! {self(),{Cx_Id,SPIds,APIds,NPIds},1000}.

%The cortex is initialized to its proper state just as other elements. Because we have not yet implemented a learning algorithm for our NN system, we need to specify when the NN should shutdown. We do this by specifying the total number of cycles the NN should execute before terminating, which is 1000 in this case.

update_genotype(IdsNPIds,Genotype,[{N_Id,PIdPs}|WeightPs])->

N = lists:keyfind(N_Id, 2, Genotype),

io:format( “PIdPs:~p~n ”,[PIdPs]),

Updated_InputIdPs = convert_PIdPs2IdPs(IdsNPIds,PIdPs,[]),

U_N = N#neuron{input_idps = Updated_InputIdPs},

U_Genotype = lists:keyreplace(N_Id, 2, Genotype, U_N),

io:format( “N:~p~n U_N:~p~n Genotype:~p~n

U_Genotype:~p~n ”,[N,U_N,Genotype,U_Genotype]),

update_genotype(IdsNPIds,U_Genotype,WeightPs);

update_genotype(_IdsNPIds,Genotype,[])->

Genotype.

convert_PIdPs2IdPs(IdsNPIds,[{PId,Weights}|Input_PIdPs],Acc)->

con-

vert_PIdPs2IdPs(IdsNPIds,Input_PIdPs,[{ets:lookup_element(IdsNPIds,PId,2),Weights}|Acc]);

convert_PIdPs2IdPs(_IdsNPIds,[Bias],Acc)->

lists:reverse([{bias,Bias}|Acc]).


%For every {N_Id,PIdPs} tuple the update_genotype/3 function extracts the neuron with the id: N_Id, and updates its weights. The convert_PIdPs2IdPs/3 performs the conversion from PIds to Ids of every {PId,Weights} tuple in the Input_PIdPs list. The updated Genotype is then returned back to the caller.

Now lets compile the cortex, neuron, sensor, actuator, and the exoself module, and test the NN system:

1> c(cortex).

ok

We now create a new NN genotype which uses the rng sensor, a pts actuator, and employs a [1,2] hidden density list. Then we map it to its phenotype by using the exoself module.

1> constructor:construct_Genotype(ffnn,rng,pts,[1,2]).

ok

2> exoself:map(ffnn).

...

It works! Our NN system has sensed, thought, and acted using its sensor, neu- rons, and the actuator, while being synchronized through the cortex process. Yet still this system does nothing but process random vectors using neural processes which themselves use random weights. We now need to develop a learning algo- rithm, and then devise a problem on which to test how well the NN can learn and solve problems using its learning method. In the next chapter we will develop an augmented version one of the most commonly used unsupervised learning algo- rithms: the Stochastic Hill-Climber.

6. 7 Summary

We have started with just a discussion of how a single artificial neuron pro- cesses an incoming signal, which is vector encoded. We then developed a simple sensor and actuator, so that the neuron has something to acquire sensory signals with, and so that it can use its output signal to act upon the world, in this case simply printing that output signal to screen. We then began designing the architec- ture of the NN system we wish to develop, and the genotype encoding we wanted to store that NN system in. After we had agreed on the architecture and the encod- ing, we created the genotype constructor which built the NN genotype, and then a mapper function which converted the genotype to its phenotype form. With this, we had now developed a system that can create NN genotypes, and convert them to phenotypes, We tested the ability of the NN to sense using its sensors, thinking

6.7 Summary

about the signals it acquired through its sensors, and then act upon the world by using its actuators; the system worked. Though our NN system does not yet have a way to learn, or be optimized for any particular task, we have developed a com- plete encoding method, a genotypical and phenotypical representation of a fully concurrent NN system, in just a few pages. With Erlang, a neuron is a process, an action potential is a message, there is a 1:1 mapping, which made developing this system so easy.

Chapter 7 Adding the “Stochastic Hill-Climber ” Learning Algorithm

Abstract In this chapter we discuss the functionality of an optimization method called the Stochastic Hill Climber , and the Stochastic Hill Climber With Random Restarts . We then implement this optimization algorithm, allowing the exoself process to train and optimize the neural network it is overlooking. Afterwards, we implement a new problem interfacing method through the use of public and pri- vate scapes , which are simulated environments, not necessarily physical. We ap- ply the new system to the XOR emulation problem, testing its performance on it. Finally, looking to the future and the need for us to be able to test and benchmark our neuroevolutionary system as we add new features to it, we create the benchmarker process, which summons the trainer and the NN it trains, applying it to some specified problem X number of times. Once the benchmarker has applied the trainer to the problem X number of times and accumulated the resulting statis- tics, it calculates the averages and the associated standard deviations for the im- portant performance parameters of the benchmark.

Though we have now created a functional NN system, synchronized by the cor- tex element and able to use sensors and actuators to interact with the world, it still lacks the functionality to learn or be trained to solve a given problem or perform some given task. What our system needs now is a learning algorithm, a method by which we can train the NN to do something useful. Beside the learning algorithm itself, we will also need some external system that can automatically train and ap- ply this learning algorithm to the NN, and a system that can monitor the NN for any unexpected crashes or errors, restoring it to a functional state when necessary. Finally, though we have implemented sensors and actuators which the NN can use to interact with the world and other programs, we have not yet developed a way to present to the NN the problems we wish it to solve, or tasks that we wish to train it to perform. It is these extensions that we will concern ourselves with in this chapter.

Here we will continue extending the NN system we've developed in the previ- ous chapter. In the following sections we will discuss and add these new algo- rithms and features, and develop the necessary modules and module extensions to incorporate the new functionality.

DOI 10.1007/978-1-4614- 4463 - 3_7 , © Springer Science+Business Media New York 2013


7.1 The Learning Method

An evolutionary algorithm (EA) is a population based optimization algorithm which works by utilizing biologically analogous operators: Selection, Mutation, and Crossover. We will begin developing a population based approach in the next chapter, at this point though, we will still use a single NN based system. In such a case, there is one particular optimization algorithm that is commonly compared to an EA due to some similarities, that algorithm is the Stochastic Hill-Climber (SHC), and is the one we will be implementing.

In an evolutionary algorithm we use a population of organisms/agents, all of whom we apply to the same problem in parallel. Afterwards, based on the fitness each agent demonstrated, we use a selection algorithm to pick the fit from the un- fit, and then create offspring from these chosen fit agents. The offspring are creat- ed by taking the fit agents and perturbing/mutating them, or by crossing two or more fit agents together to create a new one. The new population is then com- posed from some combination of the original fit agents and their offspring (their mutated forms, and/or their crossover based combinations). In this manner, through selection and mutation, with every new generation we produce organisms of greater and greater fitness.

In the Stochastic Hill-Climbing algorithm that we will apply to a single agent, we do something similar. We apply a single NN to a problem, gage its perfor- mance on the problem, save its genotype and fitness, and then mutate/perturb its genome in some manner and then re-apply the resulting mutant to the problem again. If the mutated agent performs better, then we save its genotype and fitness, and mutate it again. If the mutated agent performs worse, then we simply reset the agent's genotype to its previous state, and mutate it again to see if the new mutant will perform better. In this manner we slowly climb upwards on the fitness land- scape, taking a step back if the new mutant performs worse, and retrying in a dif- ferent direction until we generate a mutant that performs better. If at some point it is noticed that no new mutants of this agent (in our case the agent is a NN based system) seem to be increasing in fitness, we then consider our agent to have reached a local optimum, at which point we could save its genotype and fitness score, and then apply this fit NN to the problem it was optimized for. Or we could try to restart the whole thing again, generate a completely new random genotype and try to hill-climb it to greater fitness. By generating a completely new NN gen- otype we hope that its initial weight (or topology and weight) combination will spawn it in a new and perhaps better location of the fitness landscape, from which a higher local optimum is reachable. In this manner the random restart stochastic hill-climber optimization algorithm can reach local and even global optimum.

The following steps represent the Stochastic Hill-Climbing (SHC) algorithm:

1. Repeat:

2. Apply the NN to some problem.


3. Save the NN's genotype and its fitness.

4. Perturb the NN's synaptic weights, and re-apply the NN to the same prob- lem.

5. If the fitness of the perturbed NN is higher, discard original NN and keep the new. If the fitness of the original NN is higher, discard the perturbed NN and keep the old.

6. Until : Acceptable solution is found, or some stopping condition is reached.

7. Return : The genotype with the fittest combination of weights.

The algorithm is further extended by generating completely new random geno- types when the NN currently being trained ceases to make progress. The following steps extend the SHC to the Random-Restart SHC version:

8. Repeat:

9. Generate a completely new genotype, and perform steps 1-7 again.

10.If the new genotype reaches a higher fitness, overwrite the old genotype with the new one. If the new genotype reaches a lower fitness, discard the new genotype.

11. Until : Acceptable solution is found, or some stopping condition is reached. 12. Return : The final resulting genotype with its fitness score.

Steps 8-11 are necessary in the case that the original/seed combination of NN topology and weights could not be optimized (hill climbed) to the level that would solve the problem. Generating a new genotype, either of the same topology but with different random weights, or of a different topology and weights, could land this new genotype in a place of the fitness landscape from which a much higher fitness can be reached. We will create a process which can apply steps 8-12 to a NN system, and we'll call this process: trainer .

We will not use the most basic version of the SHC. Instead we will modify it to be a bit more flexible and dynamic. Rather than perturbing the NN's weights some predefined K number of times before giving up, we will make K dynamic and based on a Max_Attempts value, which itself can be defined by the researcher or based on some feature of the NN that is being optimized. If for example we have perturbed the NN's weights K==Max_Attempts number of times, and none of them improved the NN's fitness, this implies that based on where it was originally created on the fitness landscape, the particular genotype has reached a good com- bination of weights, and that the NN is roughly the best that its topology allows it to be. But if on K =< Max_Attempts the NN's fitness improves, we reset K back to 0. Since the NN has just improved its fitness, it might have just escaped from some local optimum on the fitness landscape, and thus deserves another full set of attempts to try and improve its weights further. Thus, before the NN is considered to be at the top of its reachable potential, it has to fail to improve its fitness Max_Attempts number of times in a row . Max_Attempts can be set up by the re- searcher, the larger the Max_Attempts value, the more processing we're willing to spend on tunning the NN's synaptic weights. This method will ensure that if the NN can jump to a fitter place from the location on the fitness landscape that it's


currently on, it will be given that chance. Using this augmented SHC algorithm we will tune the NN's synaptic weights, allowing the particular NN topology to reach its potential, before considering that this NN and its topology is at its limits.

The number of times the trainer process should create a new genotype (same topology but with a new set of random weights), will also be based on a similar approach. The trainer process will use Trainer_MaxAttempts variable for this. And it is only after Trainer_MaxAttempts number of genotypes in a row fail to produce higher fitness, will the training phase be considered complete. Once training is complete, the trainer process will return the best found genotype, and store it in the file with the name “best ”. This augmented SHC is diagrammed in Fig-7.1 .

Fig. 7.1 Augmented Stochastic Hill Climber algorithm.

Further deviation from the standard algorithm will be with regards to the inten- sity of the weight perturbations we'll apply, and which weights and neurons we will apply those perturbations to:

1. Each neuron in the trained NN will be chosen for perturbation with a probabil- ity of 1/sqrt(NN_Size), where NN_Size is the total number of neurons in the NN.

2. Within the chosen neuron, the weights which will be perturbed will be chosen with the probability of 1/sqrt(Tot_Weights), where Tot_Weights is the total number of synaptic weights in the weights list.

3. The intensity of the synaptic weight perturbation will be randomly chosen with uniform distribution between -Pi and Pi.

4. Max_Attempts variable will be proportional to sqrt(NN_Size).

Features 1, 2 & 3 allow our system to have a chance of producing very high in- tensity mutations, where perturbations are large and applied to many neurons and many synaptic weights within those neurons, all at once. At the same time there is


a chance that the general perturbation intensity will be small, and applied only to a few neurons and a few of their synaptic weights. Thus this approach allows our system to use highly variable mutation/perturbation intensities. Small intensity mutations can fine tune the weights of the NN, while large intensity mutations give our NN a chance to jump out of local synaptic weight optima. Feature 4 will ensure that larger NNs will have a greater amount of processing resources allocat- ed to the tuning of their synaptic weights, because the larger the NN the greater the chance that a larger combination of weights needs to be set to just the right values all at once for the NN to become fitter, which requires a larger number of attempts to do so.

7.1.1 Comparing EA to Random Restart SHC

Note how similar the random-restart stochastic hill-climber (RR-SHC) and the standard evolutionary algorithm (EA) are. The RR-SHC is almost like a sequential version of the EA. The following list is a comparison of that:

1.

EA : In generational EA we start by creating a population of NNs, each with the same topology and thus belonging to the same species. Though each individual will have the same NN topology, each one will have a dif- ferent combination of synaptic weights.

RR-SHC : In RR-SHC we create a particular genotype, a NN topology. We then try out different combinations of weights. Each new combination of weights represents an individual belonging to the same species. We are thus trying out the different individuals belonging to the same specie se- quentially, one at a time, instead of all in parallel.

  • But unlike in EA, these different individuals belonging to the same species are not generated randomly, and are not tried in a random order. Instead we generate one individual by perturbing its “ancestor ”. Each new indi- vidual in the specie is based on the previous version of itself, which is used as a stepping stone in an attempt to create a fitter version.

2.

EA : After all the organisms have been given their fitness, we create the next generation from the subset of the fit organisms. The next generation might contain new NN topologies and new species, generated from the old species through mutation and crossover. Again, each NN species will con- tain multiple members whose only difference is in their synaptic weights. *But here the new species are generated as an attempt at improvement (hill climbing), with the parent individuals acting as the stepping stone for the next topological and parametric innovation.


RR-SHC : After trying out the different permutations of synaptic weights for the original topology, we generate a new one (either the same or a new topology). We then again try out the different synaptic weight combina- tions for the genotype that starts from a different spot on the fitness land- scape.

  • But here we are trying out the same topology. Although of course the al- gorithm can be augmented, and during this step we could generate a new genotype, not simply having a different set of starting weights but one hav- ing a topology that is a perturbation of the previous version, or even com- pletely new and random.

The EA will eventually have a population composed of numerous species, each with multiple members, and thus EA explores the solution space far and wide. If one species or a particular combination of weights in a species leads to a dead end on the fitness landscape, there will be plenty of others working in parallel which will, if there is a path, find a way to higher fitness by another route.

On the other hand the RR-SHC, during its generation of a new genotype or per- turbation of weights, can only select that 1 option, the first combination that leads to a greater fitness is the path selected. If that particular selection is on the path to a dead end, it will not be known until it's too late and the system begins to fail to produce individuals of greater fitness. A dead-end that is reached when perturbing weights is not so bad because our system also generates and tries out new geno- types. But if we start generating new topologies based on the old ones in this se- quential hill climbing manner, then the topological dead end reached at this point could be much more dire, since we will not know when the first step towards the dead end was taken, and we will not have multiple other species exploring paths around the topological dead end in parallel... Fig-7.2 shows the similarities and differences in the evolutionary paths taken by organisms optimized using these two methods.

In the following figure, the red colored NNs represent the most fit NNs within the particular specie they belong to. For the evolutionary computation, the flow of time is from top to bottom, whereas for the RR-SHC it is from top to bottom for a species, but the NNs and species are tried one after the other, so it is also from left to right. Evolutionary process for both, the evolutionary computation algorithm and the RR-SHC algorithm, is outlined by the step numbers, discussed next.


Fig. 7.2 Similarities and Differences in the evolutionary paths taken by organisms evolved through EA and RR-SHC.

Let us first go through the evolutionary path taken by the Evolutionary Compu- tation algorithm, by following the specified steps:

1.

2.

3.

4.

Species-1 and Species-2 are created, each with 4 NNs, thus the population is composed of 8 NNs in total, separated into 2 species.

In Species-1, NN2 and NN3 are the most fit. NN2 creates 3 offspring, which all belong to Species-1. NN3 creates offspring which belong to Species-4. In Species-2, NN1 and NN2 are the most fit. The NN1 of Species-2 creates off- spring which make up Species-3, while NN2 creates offspring which go into Species-4. In this scenario, the parents do not survive when creating an off- spring, and thus Species-2 does not make it to the next generation.

During the second generation, the population is composed of Species-1, Species-4, and Species-3, and 9 NNs in total. In this population, the most fit NNs are NN2 and NN3 of Species-1, and NN3 of Species-4. NN2 of Species-1 creates 3 offspring, all of which have the topology of Species-3. NN3 creates 3 offspring, all of which have the topology of Species-4. Finally, NN3 of Species-4 creates 3 offspring, all of which have a new topology, and are thus designated as Spe- cies-5.

The third generation, composed of Species-3, Species-4, and Species-5, make up the population of size 9. The most fit of this population is the NN1 of Spe- cies-3, NN2 of Species-4, and NN1 of Species-5. The most fit of the three NNs, is NN1 of Species-5, which is designated as Champion.

Similarly, let us go through the steps taken by the RR-SHC algorithm:


1. The RR-SHC creates Species-2 (if we are to compare the topologies to those created by the evolutionary computation algorithm that is, otherwise we can designate the topology with any value).

2. First NN1 is created and tested. Then NN2 is created by perturbing NN1. NN2 has a higher fitness than NN1. The fitness has thus just increased, and NN2's fitness is now the highest reached thus far for this run.

3. We perturb NN2, creating a NN3. NN3 does not have a higher fitness value, so we re-perturb NN2 and create NN4, which also does not have a higher fitness than NN2. We perturb NN2 again and create NN5, which does have a higher fitness than NN2. We designate NN5 as the most fit at this time.

4. NN5 is perturbed to create NN6, which is not fitter. NN5 is perturbed again to create NN7, which is fitter than NN5, thus NN7 is now designated as the most fit.

5. NN7 is perturbed to create NN8, which is not fitter, and it is then perturbed to create NN9, which is also not fitter. We perturb NN7 for the third time to create NN10, which is also not fitter. If we assume that we set our RR-SHC algo- rithm's Max_Attempts to 3, then our system has just failed to produce a fitter agent 3 times in a row. Thus we designate NN7 of this optimization run, to be the most fit.

6. NN7 is saved as the best NN achieved during this stochastic hill climbing run. 7. The RR-SHC creates a new random NN11, whose topology designates it to be

of Species-3.

8. The NN11 is perturbed to create NN12, which is not fitter than NN11. It is per- turbed again to create NN13, which is fitter than NN11, and is thus designated as the fittest thus far achieved in this particular optimization run.

9. This continues until termination condition is reached for this hill climbing run as well. At which point we see that NN20 is the most fit.

10. A new random NN21 is generated. It is optimized through the same process. Until finally NN30 is designated to be the most fit of this hill climbing run.

11. Amongst the three most fit NNs, [NN7, NN20, NN30], produced from 3 ran- dom restarts of the hill climbing optimization algorithm, NN7 is the most fit. Thus, NN7 is designated as the champion produced by the RR-SHC algorithm.

We will return to these issues again in later chapters. And eventually build a hybrid composed of these two approaches, in an attempt to take advantage of the greedy and effective manner in which SHC can find good combinations of synap- tic weights, and the excellent global optima searching abilities of the population based evolutionary approach.

7.2 The Trainer

The trainer will be a very simple program that first generates a NN genotype under the name “experimental ”, and then applies it to the problem. After the exoself (discussed in the next section) finishes performing synaptic weight tuning

7.3 The Exoself

of its NN, it sends the trainer process a message with the NN's highest achieved fitness score and total number of evaluations it took to reach it (the number of times NN's total fitness had been evaluated). The trainer will then compare this NN's fitness to the fitness of the genotype under the name “best ” (which will be 0 if there is no genotype under such name yet). If the fitness of “experimental ” is higher than that of “best ”, the trainer will rename experimental to best, thus over- writing and removing it. If the fitness of “best ” is higher than that of “experi- mental ”, the trainer will not overwrite the old genotype. In either case, the trainer then generates a new genotype under the name “experimental ”, and repeats the process.

As noted in section 7.1, the trainer will use Trainer_MaxAttempts variable to determine how many times to generate a new genotype. Only once the Trainer fails to generate a fitter genotype Trainer_MaxAttempts number of times in a row, will it be finished, at which point the trainer process will have stored the fittest genotype in the file “best ”.


We now need to create an external process to the NN system which tunes the NN's weights through the use of augmented SHC algorithm. A modified version of the Exoself process we created in the previous chapter is an excellent contender for this role. We cannot use the cortex element for this new functionality because it is part of the NN system itself. The cortex is the synchronizer, and also the ele- ment that can perform duties that require global view of the neural topology. For example in something like competitive learning where the neural response intensi- ties to some signal need to be compared to one another for the entire network, an element like cortex can be used. But if for example it is necessary to shut down the entire NN system, or to add new sensors and actuators or new neurons while the NN itself is live and running, or update the system's source code, or recover a previous state of the NN system, that duty could be better performed by an exter- nal process like the exoself .

Imagine an advanced ALife scenario where a simulated organism is controlled by a NN. In this simulation the organism is already able to modify its own neural topology and add new sensors and actuators to itself. To survive in the environ- ment, it learned to experiment with its own sensors, actuators, and neural architec- ture so as to give itself an advantage. During one of its experimentations, some- thing goes terribly wrong: synchronization becomes broken and the whole NN system stops functioning. For example the NN based agent (an infomorph ) is ex- perimenting with its own neural topology, and by mistake deletes a large chunk of itself. The system would become too broken to fix itself after such a thing, and thus the problem would have to be fixed from the outside of the NN system, by


some process that is monitoring it and can revert it back to its previous functional state. For such, and more common events, we need a process which would act as a constant monitor of the self , while being external to the self . This process is the Exoself .

The exoself is a process that will function in a cooperative manner with the self (NN). It will perform jobs like backing up the NN's genotype to file, reverting to earlier versions of the NN, adding new neurons in live systems when asked by the NN itself... In slightly less advanced scenarios, the exoself will be used to opti- mize the weights of the NN it is monitoring. Since the exoself is outside the NN, it could keep track of which combination of weights in the NN produced a fitter in- dividual, and then tell the NN when to perturb its neural weights, and when to re- vert to the previous combination of the weights if that yielded a fitter form. The architecture of such a system is presented in Fig-7.3 .

Fig. 7.3 The architecture of a NN system with an exoself process.

7.4 The Scape

Having now covered the trainer and the exoself, which act as the appliers of the augmented RR-SHC optimization algorithm to the NN system, we now need to come up with a way to apply the NN to the problems, or present tasks to the NN that we wish it to learn how to perform. The simulations, tasks, and problems that we wish to apply our NN to, will be abstracted into scape packages, which we dis- cuss in the next section.


Scapes are composed of two parts, a simulation of an environment or a problem we are applying the NN to, and a function that can keep track of the NN's perfor- mance. Scapes run outside the NN systems, as independent processes with which the NNs interact using their sensors and actuators. There are 2 types of scapes. One type of scapes, private, is spawned for each NN during the NN's creation, and destroyed when that NN is taken offline. Another type of scapes, public, is persis- tent, they exist regardless of the NNs, and allow multiple NNs to interact with them at the same time, and thus they can allow those NNs to interact with each other too. The following are examples of these two types of scapes:

1. For example, let's assume we wish to use a NN to control a simulated robot in a simulated environment where other simulated robots exist. The fitness in this environment is based on how many simulated plants the robot eats before run- ning out of energy. The robot's energy decreases at some constant rate. First we generate the NN with appropriate Sensors and Actuators with which it can in- terface with the scape. The sensors could be cameras, and the actuators could control the speed and direction of the robot's navigation through the environ- ment, as shown in Fig-7.4 . This scape exists outside the NN system, and for the scape to be able to judge how well the NN is able to control the simulated ro- bot, it needs a way to give the NN fitness points. Furthermore, the scape can ei- ther give the NN a fitness point every time the robot eats a plant (event based), or it can keep track of the number of plants eaten throughout the robot's life- time, until the robot runs out of energy, at which point the scape would use some function to give the NN its total fitness (life based) by processing the to- tal number of plants eaten using that function. The simulated environment could have many robots in it, interacting with the simulated environment and each other. When one of the simulated robots within the scape runs out of ener- gy, or dies, it is removed from the scape, while the remaining organisms in the scape persist. The scape continues to exist independently of the NNs interacting with it, it is a public scape .


Fig. 7.4 Public simulation, with multiple organisms being controlled by NNs.

2. We want to create a NN that is able to balance a pole on a cart which can be pushed back and forth on a 2 meter track, as shown in Fig-7.5 . We could create a scape with the physics simulation of the pole and the cart, which can be inter- acted with through the NN's sensors and actuators. The job of the NN would be to push the cart back and forth, thus balancing the pole on it. The NN's sensors would gather information like the velocity and the position of the cart, and the angular velocity and the position of the pole. The NN's actuators would push the cart back and forth on the track. The scape, having access to the whole sim- ulation, could then distribute fitness points to the NN based on how well and how long it balances the pole. When the NN is done with its pole balancing training and is deactivated, the scape is deactivated with it. If there are 3 differ- ent NNs, each will have its own pole balancing simulation scape created, and those simulations will only exist for as long as the NNs exist. Some problems or simulations are created specifically, and only, for that NN which needs it. This is an example of a private scape .

Fig. 7.5 Private simulation, with multiple NN based agents, each with its own private scape.

7.5 Scapes, Sensors, Actuators, Morphologies, and Fitness

Though both of these problems (Artificial life and pole balancing) are repre- sented as scapes, they have an important difference. The first scape is a 3d envi- ronment where multiple robots can exist, the scape's existence is fully independ- ent of the NNs interacting with it, it is a public scape. The scape exists on its own, whether there are organisms interacting with it or not. It has to be spawned inde- pendently and before the NNs can start interacting with it. The second scape ex- ample on the other hand is created specifically for each NN when that neural net- work goes online, it is an example of a private scape. A private scape is in a sense summoned by the NN itself for the purpose of practicing something, or training to perform some task in isolation, a scape to which other organisms should not have access. It's like a privately spawned universe which is extinguished when the NN terminates.

We will present the problems we wish our NNs to learn to solve through these scape environments. Furthermore, we will specify within the sensors and actuators which scapes the NN should spawn (if the scape is private), or interface with (if the scape is public). Sensors and actuators are created to interact with things, thus they provide a perfect place where we can specify what scapes, if any, they should spawn or interface with. The next important design decision is: how do we want the scapes to notify the NNs of their fitness scores. A problem we solve in the next section.


Now that we've agreed on the definition of a scape, (a simulation of an envi- ronment, not necessarily physical or 3 dimensional, that can also gage fitness and which can be interfaced with, through sensors and actuators), we will need to de- vise a way by which the Scape can contact the NN and give it its fitness score.

The NN uses the sensors and actuators to interface with the scape, and since a scape exists as a separate process, the sensors and actuators will interact with it by sending it messages. When the cortex sends a particular sensor the {Cx_PId, sync} message, that sensor is awoken to action and if it is the type of sensor that interfaces with a scape, it will send the scape a message and tell it what kind of sensory data it needs. We will make the message from sensor to scape have the following format: {Sensor_PId, sensor_name}, and the scape will then send the Sensor_PId the sensory signals based on the sensor_name. If the sensor is a cam- era, it will tell the scape that it needs camera based data associated with that sen- sor. If the sensor is a sonar scanner, then it would request sonar data. The scape then sends some sensor-specific data as a reply to the sensor's message. Finally, the sensor could then preprocess this data, package it, and forward the vector to the neurons and neural structures it's connected to.

After the NN has processed the sensory data, the actuator will have received the output signals from all the neurons connecting to it. The actuator could then postprocess the accumulated vector and, if it is the type of actuator that interfaces


with a scape, it will then send this scape the {Actuator_PId, actuator_name, Ac- tion} message. At this point the actuator could be finished, and then inform the cortex of it by sending it the {self(), sync} message. But we could also, instead of making our actuator immediately send the cortex a sync message, take this oppor- tunity to make the actuator wait for the scape to send it some message. For exam- ple at this point, based on the actuator's action, the scape could send back to it a fitness score message. Or it could send the NN a message telling it that the simu- lated robot had just exploded. If the scape uses this opportunity to send some fit- ness based information back to the actuator, the actuator could then, instead of simply sending the {self(), sync} message to the cortex, send a message contain- ing the fitness points it was rewarded with by the scape, and whether the scape has notified it that the simulation or some training session has just ended. For exam- ple, in the case where the simulated organism dies within the simulated environ- ment, or in the case where the NN has solved or won the game... there needs to be a way for the scape to notify the NN that something significant had just happened. This approach could be a way to do just that.

What makes the actuator a perfect process to which the scape should send this information, is: 1. Because each separate actuator belonging to the same NN could be interfacing with a different scape and so different scapes could each send to the cortex a message through its own interfacing actuator, and 2. Because the cortex synchronizes the sensors based on the signals it receives from actuators. This is important because if at any point one of the actuators sends it a halt message, the cortex has the chance to stop or pause the whole thing by simply not triggering the sensors to action. Thus if any of the actuators propagates the “end of the simula- tion/training message ” from the scape to the cortex, the cortex will then know that the simulation is over, that it should not trigger the sensors to action, and that it should await new instructions from the exoself. If we were to allow the scape to contact the cortex directly, for example by sending it a message containing the gaged performance of the NN, and send it information of whether the training ses- sion is over or not, then there would be a chance that all the actuators had already contacted it, and that the cortex has already triggered its sensors to action. Using the actuators for this purpose ensures that we can stop the NN at the end of its training session and sense-think-act cycle.

Once the cortex receives some kind of halting message from the actuators, it could inform the exoself that it's done, and pass to it the accumulated fitness points as the NN's final fitness score. The exoself could then decide what to do next, whether to perturb the NN's weights, or revert the weights back to their pre- vious state... the cortex will sit and wait until the exoself decides on its next action.

We will implement the above described interface between the sen- sor/actuator/cortex and scape programs. Figure-7.6 shows the signal exchange steps. After exoself spawns the NN, the cortex immediately calls the sensors to ac- tion by sending them the: {CxPId,sync} messages (1). When a sensor receives the sync message from the cortex, it contacts the scape (if any) that it interfaces with,


by sending it the: {SPId,ControlMsg} message (2). When the scape receives a message from a sensor, it executes the function associated with the ControlMsg, and then returns the sensory signal back to the calling sensor process. The sensory signal is sent to the sensor through the message: {ScPId,percept,SensoryVector} (3). After the sensor preprocesses (if at all) the sensory signal, it fans out the sen- sory vector to the neurons it is connected to (4). Then the actual neural net pro- cesses the sensory signal, (5), this is the thinking part of the NN system's sense- think-act loop. Once the neural net has processed the signal, the actuator gathers the processed signals from the output layer neurons (6). At this point the actuator does some postprocessing (if at all) of the output vector, and then executes the ac- tuator function. The actuator function, like the sensory function, could be an ac- tion in itself, or an action message sent to some scape. In the case where the actua- tor is interfacing with a scape, it sends the scape a message of the form: {APid,ControlMsg,Output} (7). The scape executes the particular function associ- ated with the ControlMsg atom, with the Output as the parameter. The executed function IS the action that the NN system takes within the virtual environment of the scape. After the scape executes the function that was requested by the actuator, it sends that same actuator a message: {SCPId,Fitness,HaltFlag}. This message contains the NN's complete or partial gage of fitness, and a notification if the sim- ulation has ended, or if the avatar that the NN system controls within the scape has perished... or anything else one might wish to use the HaltFlag for (8). Finally, the actuator passes the message to the cortex in the form of a message: {APId,sync,Fitness,HaltFlag}, (9).

Fig. 7.6 The signal flow between the scape, sensors, actuators, cortex, and the exoself.


The cortex accumulates all the messages from the actuators, adding together the fitness scores, and seeing if any of the scapes have triggered the HaltFlag, which is set to 1 when triggered. If none of the HaltFlags were set to 1, then the cortex process syncs up the sensors, calling them to action, and the steps 1-9 re- peat again. If any of the scapes set the HaltFlag to 1, then the cortex process takes a different course of action. At this point we could make the cortex decide whether to restart the scape, whether to start off some particular set of functions, or do any other useful task. In the version of the NN we are building in this chapter though, the XOR simulation scape will activate the HaltFlag when the training has ended, when the simulation has ran to completion. At this point the cortex simply pauses and informs the exoself process that it has finished and that the NN has been eval- uated, by sending exoself the message: {CxPId, evaluation_complete, FitnessAcc, CycleAcc, TimeAcc} (a). Once again, the actions that the exoself takes are based on operational mode we chose for the system, and in the version of the system we're building in this chapter, it will apply the SHC optimization algorithm to the NN, and then reactivate the NN system by sending cortex the following message: {ExoselfPId, reactivate} (b). After receiving the reactivation message from the exoself, the cortex's process again loops through the steps 1-9. Note that these 9 steps represent the standard Sense-Think-Act cycle, where Sense is steps 1-4, Think is step 5, and Act is steps 6-9.

In the next section we extend the previous chapter's architecture by developing and adding the new features we've discussed here.

7.6 Developing the Extended Architecture

Having discussed what new features are needed to extend the architecture so that it can be applied to various problems and simulations presented through scapes, and having decided what algorithms our system should use to optimize its neural weights, we are ready to extend the last chapter's system. After we finish modifying the source code, our NN systems should be optimizable by the exoself & trainer processes, and be able to interact with both, public and private scapes in various manners as shown in Fig-7.7 .

In the following subsections we will add the new trainer and scape modules, and modify the exoself, cortex, morphology, sensor, actuator, and neuron mod- ules. Finally, we will also switch from lists to ets for the purpose of storing geno- types. And because we isolated the genotype reading, writing, loading, and saving functions in genotype.erl, the switch will be composed of simply modifying 4 functions to use ets instead of lists. Because we decoupled the storage and inter- face methods in the last chapter, it is easy for us to move to whatever storage method we find most effective for our system. Thus, modifying the genotype module to use ets instead of lists will be our first step.


Fig. 7.7 Possible combinations of NN systems and their public/private scape interaction. 7.6.1 Modifying the genotype Module

Ets tables provide us with an efficient way to store and retrieve terms. Particu- larly if we need to access random terms within the table, it is better done using ets. The quick modification applied to the genotype.erl to switch from lists to ets is shown in the following listing.

Listing-7.1 Changing the genotype storage method from the simple list and file to the ets table. save_genotype(FileName,Genotype)->

TId = ets:new(FileName, [public,set,{keypos,2}]),

[ets:insert(TId,Element) || Element <- Genotype],

ets:tab2file(TId,FileName).

%The save_genotype/2 function expects that the Genotype is a list composed of the neuron,

sensor, actuator, cortex, and exoself elements. The function creates a new ets table, writes all

the element representing tuples from the Genotype list to the ets table, and then writes the ets table to file.

save_to_file(Genotype,FileName)->

ets:tab2file(Genotype,FileName).

%The save_to_file/2 function saves the ets table by the name Genotype to the file by the name FileName.


load_from_file(FileName)->

{ok,TId} = ets:file2tab(FileName),

TId.

%The load_from_file/1 loads an ets representing file by the name FileName, returning the ets table id to the caller.

read(TId,Key)->

[R] = ets:lookup(TId,Key),

R.

%The read/2 function reads a record associated with Key from the ets table with the id TId, re- turning the record R to the caller. It expects that only a single record exists with the specified

Key.

write(TId,R)->

ets:insert(TId,R).

%The function write/2 writes the record R to the ets table with the id TId.

print(FileName)->

Genotype = load_from_file(FileName),

Cx = read(Genotype,cortex),

SIds = Cx#cortex.sensor_ids,

NIds = Cx#cortex.nids,

AIds = Cx#cortex.actuator_ids,

io:format( “~p~n ”,[Cx]),

[io:format( “~p~n ”,[read(Genotype,Id)]) || Id <- SIds],

[io:format( “~p~n ”,[read(Genotype,Id)]) || Id <- NIds],

[io:format( “~p~n ”,[read(Genotype,Id)]) || Id <- Aids].

%The function print/1 reads a stored Genotype from the file FileName, and then prints to con- sole all the elements making up the NN's genotype.

The function print/1 is a new addition. It is a simple and easy way to dump the tuple encoded genotype to terminal. It is useful when you wish to see what the to- pology looks like, and when you need to analyze the genotype when debugging the system.

7.6.2 Modifying the morphology Module

As we discussed earlier, we want to specify the scape and its type in the sensors and actuators belonging to some particular morphology. We thus modify the rec- ords.hrl file and extend the sensor and actuator records to also contain the scape element. The new sensor and actuator records will be represented as:


-record(sensor,{id,name,cx_id,scape,vl,fanout_ids}).

-record(actuator,{id,name,cx_id,scape,vl,fanin_ids}).

In the previous chapter, we only had one type of morphology called: test , we now modify the morphology module by removing “test ”, and adding xor_mimic . Both, the sensor and the actuator of the xor_mimic morphology, interact with a private scape called xor_sim . Because the same private scape is specified for both the sensor and the actuator, only one private scape by the name xor_sim will be spawned, and they (the sensor and the actuator) will both connect to it, rather than each separately spawning its own separate xor_sim scape. The following listing shows the new xor_mimic morphology with its lists of sensors and actuators.

Listing-7.2: The new morphological type and specification added to morphology.erl xor_mimic(sensors)->

[

  1. sensor{id={sensor,generate_id()}, name=xor_GetInput, scape={private,xor_sim},

vl=2}

];

xor_mimic(actuators)->

[

  1. actuator{id={actuator,generate_id()}, name=xor_SendOutput,

scape={private,xor_sim}, vl=1}

].

Having now modified the genotype to use ets tables, and having extended the records of our sensors and actuators, we are ready to start adding new features to our system.

7.6.3 Developing the trainer Module

When training a NN we should be able to specify all the stopping conditions, the NN based agent's morphology, and the neural network's topology. Thus the trainer process should perform the following steps:

1. Repeat:

2. Create a NN of specified topology with random weights.

3. Wait for the trained NN's final fitness

4. Compare the trained NN's fitness to an already stored NN's fitness, if any. 5. If the new NN is better, overwrite the old one with it.

6. Until: One of the stopping conditions is reached

7. Return: Best genotype and its fitness score.


We will implement all 3 types of stopping conditions:

1. A fitness score that we wish to reach,

2. The Max_Attempts that we're willing to perform before giving up.

3. The maximum number of evaluations we're willing to perform before giving up.

If any one of these conditions is triggered, the training ends. For default, we set maximum number of evaluations (EVAL_LIMIT) and the minimum required fit- ness (FITNESS_TARGET) to inf, which means that these conditions will never be reached since in Erlang an atom is considered greater than a number. Thus when starting the trainer by executing go/2, the process will default to simply using MAX_ATTEMPTS, which is set to 5. The complete source code for trainer.erl is shown in the following listing.

Listing-7.3 The implementation of the trainer module.

-module(trainer).

-compile(export_all).

-include( “records.hrl ”).

-define(MAX_ATTEMPTS,5).

-define(EVAL_LIMIT,inf).

-define(FITNESS_TARGET,inf).

go(Morphology,HiddenLayerDensities)->

go(Morphology,HiddenLayerDensities,?MAX_ATTEMPTS,?EVAL_LIMIT,

?FITNESS_TARGET).

go(Morphology,HiddenLayerDensities,MaxAttempts,EvalLimit,FitnessTarget)->

PId = spawn(trainer,loop,[Morphology,HiddenLayerDensities,FitnessTarget,

{1,MaxAttempts},{0,EvalLimit},{0,best},experimental]),

register(trainer,PId).

%The function go/2 is executed to start the training process based on the Morphology and HiddenLayerDensities specified. The go/2 function uses a default values for the Max_Attempts, Eval_Limit, and Fitness_Target parameters, which makes the training based purely on the Max_Attempts value. Function go/5 allows for all the stopping conditions to be specified.

loop(Morphology,_HLD,FT,{AttemptAcc,MA},{EvalAcc,EL},{BestFitness,BestG},_ExpG, CAcc,TAcc) when (AttemptAcc>=MA) or (EvalAcc>=EL) or (BestFitness>=FT)->

genotype:print(BestG),

io:format( “ Morphology:~p Best Fitness:~p EvalAcc:~p~n ”, [Morphology, BestFitness, EvalAcc]);

loop(Morphology,HLD,FT,{AttemptAcc,MA},{EvalAcc,EvalLimit},{BestFitness,BestG},

ExpG,CAcc,TAcc)->

genotype:construct(ExpG,Morphology,HLD),

Agent_PId=exoself:map(ExpG),


receive

{Agent_PId,Fitness,Evals,Cycles,Time}->

U_EvalAcc = EvalAcc+Evals,

U_CAcc = CAcc+Cycles,

U_TAcc = TAcc+Time,

case Fitness > BestFitness of

true ->

file:rename(ExpG,BestG),

?MODULE:loop(Morphology,HLD,FT,{1,MA}, {U_EvalAcc,

EvalLimit}, {Fitness,BestG},ExpG,U_CAcc,U_TAcc);

false ->

?MODULE:loop(Morphology,HLD,FT,{AttemptAcc+1,MA},

{U_EvalAcc,EvalLimit}, {BestFitness,BestG}, ExpG,U_CAcc,U_TAcc)

end;

terminate ->

io:format( “Trainer Terminated:~n ”),

genotype:print(BestG),

io:format( “ Morphology:~p Best Fitness:~p EvalAcc:~p~n ”, [Morphology,

BestFitness,EvalAcc])

end.

%loop/7 generates new NNs and trains them until a stopping condition is reached. Once any

one of the stopping conditions is reached, the trainer prints to screen the genotype, the morpho- logical name of the organism being trained, the best fitness score achieved, and the number of evaluations taken to find this fitness score.

7.6.4 Modifying the exoself Module

The Exoself's purpose is to train and monitor the NN system. Though at this point we will only implement the NN activation, training, and termination. But the way we will design the exoself process, and its general position in the NN based agent's architecture, will allow it to be modified (and we eventually will modify it) to support the NN's fault tolerance and self repair functionality. In the follow- ing listing we add to the exoself module the necessary source code it needs to train the NN system using the augmented SHC algorithm we covered in section 7.1.

Listing-7.4: Modifications added to the exoself.erl

prep(FileName,Genotype)->

{V1,V2,V3} = now(),

random:seed(V1,V2,V3),

IdsNPIds = ets:new(idsNpids,[set,private]),

Cx = genotype:read(Genotype,cortex),

Sensor_Ids = Cx#cortex.sensor_ids,

Chapter 7 Adding the “Stochastic Hill-Climber ” Learning Algorithm Actuator_Ids = Cx#cortex.actuator_ids,

NIds = Cx#cortex.nids,

ScapePIds=spawn_Scapes(IdsNPIds,Genotype,Sensor_Ids,Actuator_Ids),

spawn_CerebralUnits(IdsNPIds,cortex,[Cx#cortex.id]),

spawn_CerebralUnits(IdsNPIds,sensor,Sensor_Ids),

spawn_CerebralUnits(IdsNPIds,actuator,Actuator_Ids),

spawn_CerebralUnits(IdsNPIds,neuron,NIds),

link_Sensors(Genotype,Sensor_Ids,IdsNPIds),

link_Actuators(Genotype,Actuator_Ids,IdsNPIds),

link_Neurons(Genotype,NIds,IdsNPIds),

{SPIds,NPIds,APIds}=link_Cortex(Cx,IdsNPIds),

Cx_PId = ets:lookup_element(IdsNPIds,Cx#cortex.id,2),

loop(FileName,Genotype,IdsNPIds,Cx_PId,SPIds,NPIds,APIds,ScapePIds,0,0,0,0,1).

%Once the FileName and the Genotype are dropped into the prep/2 function, the function uses the current time to create a new random seed. Then the cortex is extracted from the genotype and the Sensor, Actuator, and Neural Ids are extracted from it. The sensors and actuators are dropped into the spawn_Scapes/4, which extracts the scapes that need to be spawned, and then spawns them. Afterwards, the sensor, actuator, neuron, and the cortex elements are spawned. Then the exoself process sends these spawned elements the PIds of the elements they are con- nected to, thus linking all the elements together into a proper interconnected structure. The cor- tex element is the last one to be linked, because once it receives the message from the exoself with all the data, it immediately starts synchronizing the NN by prompting the sensors to action. Afterwards, prep/2 drops into the exoself's main process loop.

loop(FileName,Genotype,IdsNPIds,Cx_PId,SPIds,NPIds,APIds,ScapePIds,HighestFitness,

EvalAcc,CycleAcc,TimeAcc,Attempt)->

receive

{Cx_PId,evaluation_completed,Fitness,Cycles,Time}->

{U_HighestFitness,U_Attempt}=case Fitness > HighestFitness of

true ->

[NPId ! {self(),weight_backup} || NPId <- NPIds],

{Fitness,0};

false ->

Perturbed_NPIds=get(perturbed),

[NPId ! {self(),weight_restore} || NPId <- Perturbed_NPIds], {HighestFitness,Attempt+1}

end,

case U_Attempt >= ?MAX_ATTEMPTS of

true -> %End training

U_CycleAcc = CycleAcc+Cycles,

U_TimeAcc = TimeAcc+Time,

backup_genotype(FileName,IdsNPIds,Genotype,NPIds),

terminate_phenotype(Cx_PId,SPIds,NPIds,APIds,ScapePIds), io:format( “Cortex:~p finished training. Genotype has been


backed up.~n Fitness:~p~n TotEvaluations:~p~n TotCycles:~p~n TimeAcc:~p~n ”, [Cx_PId, U_HighestFitness, EvalAcc, U_CycleAcc, U_TimeAcc]),

case whereis(trainer) of

undefined ->

ok;

PId ->

PId!{self(),U_HighestFitness, EvalAcc,

U_CycleAcc, U_TimeAcc}

end;

false -> %Continue training

Tot_Neurons = length(NPIds),

MP = 1/math:sqrt(Tot_Neurons),

Perturb_NPIds=[NPId || NPId <- NPIds,random:uniform()<MP], put(perturbed,Perturb_NPIds),

[NPId ! {self(),weight_perturb} || NPId <- Perturb_NPIds],

Cx_PId ! {self(),reactivate},

loop(FileName,Genotype, IdsNPIds,Cx_PId,SPIds, NPIds,APIds,

ScapePIds,U_HighestFitness, EvalAcc+1, CycleAcc+Cycles, TimeAcc+Time, U_Attempt)

end

end.

%The main process loop waits for the NN to complete the task, receive its fitness score, and send Exoself the: {Cx_PId,evaluation_completed,Fitness,Cycles,Time} message. The message contains all the information about that particular evaluation, the acquired fitness score, the number of total Sense-Think-Act cycles executed, and the time it took to complete the evalua- tion. The exoself then compares the Fitness to the one it has on record (if any), and based on that decides whether to revert the previously perturbed neurons back to their original state or not. If the new Fitness is lower, then the perturbed neurons are contacted and their weights are reverted. If the new Fitness is greater than the one stored on record, then the NN is backed up to file, and the variable EvalAcc is reset to 0. Finally, depending on whether the NN has failed to improve its fitness Max_Attempts number of times, the exoself decides whether another NN perturbation attempt is warranted. If it is warranted, then the exoself chooses which neurons to mutate by randomly choosing each neuron with the probability of 1/sqrt(Tot_Neurons), where Tot_Neurons is the total number of neurons in the neural network. The exoself saves the PIds of those chosen neurons to process dictionary, and then sends those neurons a signal that they should perturb their weights. Finally it tells cortex to reactivate and start syncing the sensors and actuators again. But if the NN has failed to improve its fitness for Max_Attempts number of times, if EvalAcc > Max_Attempts, then the exoself terminates all the elements in the NN, and if there is a registered process by the name ‘trainer', the exoself sends it the HighestFitness score that its NN achieved and the number of total evaluations it took to achieve it.

spawn_Scapes(IdsNPIds,Genotype,Sensor_Ids,Actuator_Ids)->

Sensor_Scapes = [(genotype:read(Genotype,Id))#sensor.scape || Id<-Sensor_Ids], Actuator_Scapes = [(genotype:read(Genotype,Id))#actuator.scape || Id<-


Actuator_Ids],

Unique_Scapes = Sensor_Scapes++(Actuator_Scapes--Sensor_Scapes),

SN_Tuples=[{scape:gen(self(),node()),ScapeName} || {private,ScapeName}<-

Unique_Scapes],

[ets:insert(IdsNPIds,{ScapeName,PId}) || {PId,ScapeName} <- SN_Tuples],

[ets:insert(IdsNPIds,{PId,ScapeName}) || {PId,ScapeName} <-SN_Tuples],

[PId ! {self(),ScapeName} || {PId,ScapeName} <- SN_Tuples],

[PId || {PId,_ScapeName} <-SN_Tuples].

%spawn_Scapes/4 first extracts all the scape names from sensors and actuators, then builds a

list of unique scapes, and then finally extracts and spawns the private scapes. The public scapes are not spawned since they are independent of the NN, and should already be running. The rea- son for extracting the list of unique scapes is because if both, a sensor and an actuator are point- ing to the same scape, then that means that they will interface with the same scape, and it does not mean that each one should spawn its own scape of the same name. Afterwards we use the IdsNPids ETS table to create a map from scape PId to scape name, and from scape name to scape PId for later use. The function then sends each spawned scape a message composed of the exoself's PId, and the scape's name: {self(),ScapeName}. Finally, a spawned scape PId list is composed and returned to the caller.

We also modify the terminate_Phenotype function to also accept the ScapePIds parameter, and terminate all the scapes before terminating the Cortex process. Thus the following line is added to the function:

[PId ! {self(),terminate} || PId <- ScapePIds]

Having created the function which extracts the names of the scapes from the sensors and actuators of the NN system, we now develop our first scape and the very first problem on which we'll test our learning algorithm on.

7.6.5 Developing the scape Module

Looking ahead, we will certainly apply our Neuroevolutionary system to many different problems. Our system should be able to deal with ALife problems as eas- ily as with pole balancing, financial trading, circuit design & optimization, image analysis, or any other of the infinite problems that exist. It is for this reason that we've made the sensors/actuators/scape a separate part from the NN itself, all specified through the morphology module. This way, for every problem we wish to apply our neuroevolutionary system to, we can keep the NN specific modules the same, and simply create a new morphology/scape packages.


Because there will be many scapes, a new scape for almost every problem that we'll want to solve or apply our neuroevolutionary system to, we will use the same scape.erl module and specify the different scapes within it by function name. Because the first and standard problem to apply a NN to is the XOR (Exclusive- OR) problem, our first scape will be a scape called xor_sim. The xor problem is the “hello world ” of NN problems. The goal is to teach a NN to act like a XOR operation. The truth table of the 2 input XOR operation is presented in the follow- ing listing.

Listing-7.5: Truth table of the XOR operation.

X1

false

false

true

true

X2 X1 XOR X2

true true

false false

false true

true false

Because the range of tanh, the activation function of our neuron, is between -1 and 1, we will represent false as -1, and true as 1 (a bipolar encoding), rather than 0 and 1 (a unipolar encoding). This way we can use the full output range of our neurons. The following listing shows the complete source code of the scape mod- ule, which at this point contains only the single scape named xor_sim.

Listing-7.6: The complete scape module.

-module(scape).

-compile(export_all).

-include( “records.hrl ”).

gen(ExoSelf_PId,Node)->

spawn(Node,?MODULE,prep,[ExoSelf_PId]).

prep(ExoSelf_PId) ->

receive

{ExoSelf_PId,Name} ->

scape:Name(ExoSelf_PId)

end.

%gen/2 is executed by the exoself. The function spawns prep/1 process, and awaits the name of the scape from the exoself. Each scape is a separate and independent process, a self contained system that was developed to interface with the sensors and actuators from which its name was extracted. The name of the scape is the name of its main process loop.


xor_sim(ExoSelf_PId)->

XOR = [{[-1,-1],[-1]},{[1,-1],[1]},{[-1,1],[1]},{[1,1],[-1]}],

xor_sim(ExoSelf_PId,{XOR,XOR},0).

xor_sim(ExoSelf_PId,{[{Input,CorrectOutput}|XOR],MXOR},ErrAcc) ->

receive

{From,sense} ->

From ! {self(),percept,Input},

xor_sim(ExoSelf_PId,{[{Input,CorrectOutput}|XOR],MXOR},ErrAcc);

{From,action,Output}->

Error = list_compare(Output,CorrectOutput,0),

case XOR of

[] ->

MSE = math:sqrt(ErrAcc+Error),

Fitness = 1/(MSE+0.00001),

From ! {self(),Fitness,1},

xor_sim(ExoSelf_PId,{MXOR,MXOR},0);

_ ->

From ! {self(),0,0},

xor_sim(ExoSelf_PId,{XOR,MXOR},ErrAcc+Error)

end;

{ExoSelf_PId,terminate}->

ok

end.

list_compare([X|List1],[Y|List2],ErrorAcc)->

list_compare(List1,List2,ErrorAcc+math:pow(X-Y,2));

list_compare([],[],ErrorAcc)->

math:sqrt(ErrorAcc).

%xor_sim/3 is a scape that simulates the XOR operation, interacts with the NN, and gages the NN's performance. xor_sim expects two types of messages from the NN, one message from the sensor and one from the actuator. The message: {From,sense} prompts the scape to send the NN the percept, which is a vector of length 2 and contains the XOR input. The second expected message from the NN is the message from the actuator, which is expected to be an output of the NN and packaged into the form: {From,action,Output}. At this point xor_sim/3 compares the Output with the expected output that is associated with the sensory message that should have been gathered by the sensors, and then sends back to the actuator process a message composed of the scape's PId, Fitness, and a HaltFlag which specifies whether the simulation has ended for the NN. The scape keeps track of the Mean Squared Error between the NN's output and the correct output. Once the NN has processed all 4 signals for the XOR, the scape computes the total MSE, converts it to fitness, and finally forwards this fitness and the HaltFlag=1 to the NN. This particular scape uses the lifetime based fitness, rather than step-based fitness. During all the other steps the scape sends the actuator the signal: {Scape_PId,0,0}, while it accumulates the errors, and only at the very end does it calculate the total fitness, which is the inverse of the error with a small extra added value to avoid the divide by 0 errors. Afterwards, xor_sim resets back to its initial state and awaits anew for signals from the NN.


7.6.6 Modifying the cortex Module

As in the previous chapter, the cortex element still acts as the synchronizer of the sensors and actuators. But now, it will also keep track of the accumulated fit- ness score, propagated to it by the actuators. The cortex will also keep track of whether the HaltFlag==1 was sent to it by any of the actuators, which would sig- nify that the cortex element should halt, notify its exoself of the achieved fitness score, and then await for further instructions from it. Finally, the cortex will also keep track of the number of sense-think-act cycles performed during its lifetime. The augmented cortex module is shown in the following listing.

Listing-7.7 The updated cortex module.

-module(cortex).

-compile(export_all).

-include( “records.hrl ”).

-record(state,{id,exoself_pid,spids,npids,apids,cycle_acc=0,fitness_acc=0,endflag=0,status}).

gen(ExoSelf_PId,Node)->

spawn(Node,?MODULE,prep,[ExoSelf_PId]).

prep(ExoSelf_PId) ->

{V1,V2,V3} = now(),

random:seed(V1,V2,V3),

receive

{ExoSelf_PId,Id,SPIds,NPIds,APIds} ->

put(start_time,now()),

[SPId ! {self(),sync} || SPId <- SPIds],

loop(Id,ExoSelf_PId,SPIds,{APIds,APIds},NPIds,1,0,0,active)

end.

%The gen/2 function spawns the cortex element, which immediately starts to wait for its initial state message from the same process that spawned it, exoself. The initial state message contains the sensor, actuator, and neuron PId lists. Before dropping into the main loop, CycleAcc, FitnessAcc, and HFAcc (HaltFlag Acc), are all set to 0, and the status of the cortex is set to ac- tive, prompting it to begin the synchronization process and call the sensors to action.

loop(Id, ExoSelf_PId, SPIds, {[APId|APIds], MAPIds}, NPIds, CycleAcc, FitnessAcc, HFAcc, active) ->

receive

{APId,sync,Fitness,HaltFlag} ->

loop(Id,ExoSelf_PId,SPIds,{APIds,MAPIds},NPIds,CycleAcc,FitnessAcc+

Fitness, HFAcc+HaltFlag,active);

terminate ->

io:format( “Cortex:~p is terminating.~n ”,[Id]),

[PId ! {self(),terminate} || PId <- SPIds],


[PId ! {self(),terminate} || PId <- MAPIds],

[PId ! {self(),termiante} || PId <- NPIds]

end;

loop(Id,ExoSelf_PId,SPIds,{[],MAPIds},NPIds,CycleAcc,FitnessAcc,HFAcc,active)->

case EFAcc > 0 of

true -> %Organism finished evaluation

TimeDif=timer:now_diff(now(),get(start_time)),

ExoSelf_PId ! {self(),evaluation_completed,FitnessAcc,CycleAcc,TimeDif},

loop(Id,ExoSelf_PId,SPIds,{MAPIds,MAPIds},NPIds,CycleAcc,FitnessAcc,

HFAcc, inactive);

false ->

[PId ! {self(),sync} || PId <- SPIds],

loop(Id,ExoSelf_PId,SPIds,{MAPIds,MAPIds},NPIds,CycleAcc+1,FitnessAcc,

HFAcc,active)

end;

loop(Id, ExoSelf_PId, SPIds, {MAPIds,MAPIds}, NPIds, _CycleAcc, _FitnessAcc, _HFAcc, inactive)->

receive

{ExoSelf_PId,reactivate}->

put(start_time,now()),

[SPId ! {self(),sync} || SPId <- SPIds], loop(Id,ExoSelf_PId,SPIds,{MAPIds,MAPIds},NPIds,1,0,0,active);

{ExoSelf_PId,terminate}->

ok

end.

%The cortex's goal is to synchronize the NN system's sensors and actuators. When the actua- tors have received all their control signals, they forward the sync messages, the Fitness, and the HaltFlag messages to the cortex. The cortex accumulates these Fitness and HaltFlag signals, and if any of the HaltFlag signals have been set to 1, HFAcc will be greater than 0, signifying that the cortex should halt. When EFAcc > 0, the cortex calculates the total amount of time it has ran (TimeDiff), and forwards to exoself the values: FitnessAcc, CycleAcc, and TimeDiff. Afterwards, the cortex enters the inactive mode and awaits further instructions from the exoself. If none of the HaltFlags were set to 0, then the value HFAcc == 0, and the cortex triggers off another Sense-Think-Act cycle. The reason the cortex process stores 2 copies of the actuator PIds: the APIds, and the MemoryAPIds (MAPIds), is so that once all the actuators have sent it the sync messages, it can restore the APIds list from the MAPIds.

7.6.7 Modifying the neuron Module

To allow the exoself process to optimize the weights of the NN through the SHC algorithm, we will need to give our neurons the ability to have their weights perturbed when requested to do so by the exoself, and have their weights reverted


when/if requested by the same. We will also specify the range of these weight per- turbations in this module, and the weight saturation values, the maximum and minimum values that the weights can take. It is usually not a good idea to let the weights reach very large positive or negative values, as that would allow any sin- gle weight to completely overwhelm other synaptic weights of the same neuron. For example if a neuron has 100 weights in total, and one of the weights has a val- ue of 1000000, no other weight can compete with it unless it too is raised to such a high value. This results in a single weight controlling the information processing ability of the entire neuron. It is important that no weight can overwhelm all others (which prevents the neuron from performing coherent processing of signals), for this reason we will set the saturation limit to 2*Pi, and the perturbation intensity to half that. The perturbation intensity is half the value of weight saturation point so that there will always be a chance that the weight could flip from a positive to a negative value. The range of the weight perturbation intensity is specified by the DELTA_MULTIPLIER macro as: -define(DELTA_MULTIPLIER,math:pi()*2), since we will multiply it by (random:uniform()-0.5), the actual range will be be- tween -Pi and Pi.

The complete neuron module is shown in the following listing.

Listing-7.8 The neuron module.

-module(neuron).

-compile(export_all).

-include( “records.hrl ”).

-define(DELTA_MULTIPLIER,math:pi()*2).

-define(SAT_LIMIT,math:pi()*2).

gen(ExoSelf_PId,Node)->

spawn(Node,?MODULE,prep,[ExoSelf_PId]).

prep(ExoSelf_PId) ->

{V1,V2,V3} = now(),

random:seed(V1,V2,V3),

receive

{ExoSelf_PId,{Id,Cx_PId,AF,Input_PIdPs,Output_PIds}} ->

loop(Id,ExoSelf_PId,Cx_PId,AF,{Input_PIdPs,Input_PIdPs},Output_PIds,0)

end.

%When gen/2 is executed it spawns the neuron element, which seeds the pseudo random num- ber generator, and immediately begins to wait for its initial state message. It is essential that we seed the random number generator to make sure that every NN will have a different set of mu- tation probabilities and different combination of perturbation intensities. Once the initial state signal from the exoself is received, the neuron drops into its main loop.


loop(Id,ExoSelf_PId,Cx_PId,AF,{[{Input_PId,Weights}|Input_PIdPs],MInput_PIdPs},Output_

PIds,Acc)->

receive

{Input_PId,forward,Input}->

Result = dot(Input,Weights,0),

loop(Id,ExoSelf_PId,Cx_PId,AF,{Input_PIdPs,MInput_PIdPs},Output_PIds,Result+Acc);

{ExoSelf_PId,weight_backup}->

put(weights,MInput_PIdPs),

loop(Id,ExoSelf_PId,Cx_PId,AF,{[{Input_PId,Weights}|Input_PIdPs],MInput_PIdPs},Outp

ut_PIds,Acc);

{ExoSelf_PId,weight_restore}->

RInput_PIdPs = get(weights),

loop(Id,ExoSelf_PId,Cx_PId,AF,{RInput_PIdPs,RInput_PIdPs},Output_PIds,Acc);

{ExoSelf_PId,weight_perturb}->

PInput_PIdPs=perturb_IPIdPs(MInput_PIdPs),

loop(Id,ExoSelf_PId,Cx_PId,AF,{PInput_PIdPs,PInput_PIdPs},Output_PIds,Acc);

{ExoSelf_PId,get_backup}->

ExoSelf_PId ! {self(),Id,MInput_PIdPs},

loop(Id,ExoSelf_PId,Cx_PId,AF,{[{Input_PId,Weights}|Input_PIdPs],MInput_PIdPs},Outp

ut_PIds,Acc);

{ExoSelf_PId,terminate}->

ok

end;

loop(Id,ExoSelf_PId,Cx_PId,AF,{[Bias],MInput_PIdPs},Output_PIds,Acc)->

Output = neuron:AF(Acc+Bias),

[Output_PId ! {self(),forward,[Output]} || Output_PId <- Output_PIds], loop(Id,ExoSelf_PId,Cx_PId,AF,{MInput_PIdPs,MInput_PIdPs},Output_PIds,0);

loop(Id,ExoSelf_PId,Cx_PId,AF,{[],MInput_PIdPs},Output_PIds,Acc)->

Output = neuron:AF(Acc),

[Output_PId ! {self(),forward,[Output]} || Output_PId <- Output_PIds], loop(Id,ExoSelf_PId,Cx_PId,AF,{MInput_PIdPs,MInput_PIdPs},Output_PIds,0).

dot([I|Input],[W|Weights],Acc) ->

dot(Input,Weights,I*W+Acc);

dot([],[],Acc)->

Acc.


%The neuron process waits for vector signals from all the processes that it's connected from. As the presynaptic signals fanin, the neuron takes the dot product of the input and their associ- ated weight vectors, and then adds it to the accumulator. Once all the signals from Input_PIds are received, the accumulator contains the dot product to which the neuron then adds the bias (if it exists) and executes the activation function. After fanning out the output signal, the neuron again returns to waiting for incoming signals. When the neuron receives the {ExoSelf_PId, get_backup} message, it forwards to the exoself its full MInput_PIdPs list, and its Id. The MInput_PIdPs contains the current version of the neural weights. When the neuron receives the {ExoSelf_PId,weight_perturb} message, it executes the perturb_IPIdPs/1, after which the neu- ron drops back into the loop but with MInput_PIdPs replaced by the new PInput_PIdPs. It is important to note that the neuron expects to be synchronized, and expects that it has at this point not received any signals from the other elements it is connected from, because if it has and it then changes out the Input_PIdPs with PInput_PIdPs, it might start waiting for signals from the elements from which it has already received the signals. When the neuron receives the {ExoSelf_PId,weight_backup}, it stores its weights in its process dictionary. When the neuron receives the {ExoSelf,weight_restore}, it restores its weights to the state they were before being perturbed by restoring the saved synaptic weights from its process dictionary.

tanh(Val)->

math:tanh(Val).

%The activation function is a sigmoid function, tanh.

perturb_IPIdPs(Input_PIdPs)->

Tot_Weights=lists:sum([length(Weights) || {_Input_PId,Weights}<-Input_PIdPs]),

MP = 1/math:sqrt(Tot_Weights),

perturb_IPIdPs(MP,Input_PIdPs,[]).

perturb_IPIdPs(MP,[{Input_PId,Weights}|Input_PIdPs],Acc)->

U_Weights = perturb_weights(MP,Weights,[]),

perturb_IPIdPs(MP,Input_PIdPs,[{Input_PId,U_Weights}|Acc]);

perturb_IPIdPs(MP,[Bias],Acc)->

U_Bias = case random:uniform() < MP of

true-> sat((random:uniform()-0.5)*?DELTA_MULTIPLIER+Bias,-

?SAT_LIMIT,?SAT_LIMIT);

false -> Bias

end,

lists:reverse([U_Bias|Acc]);

perturb_IPIdPs(_MP,[],Acc)->

lists:reverse(Acc).

%perturb_IPIdPs/1 first calculates the probability that a weight will be perturbed, the probabil- ity being the inverse square root of the total number of weights in the neuron. The function then drops into perturb_IPIdPs/3, which executes perturb_weights/3 for every set of weights associ- ated with a particular Input_PId in the Input_PIdPs list. If bias is present in the weights list, it is


reached last and perturbed just as any other weight, based on the probability. Afterwards, the perturbed and inverted version of the Input_PIdPs is reversed back to the proper order and re- turned to the calling function.

perturb_weights(MP,[W|Weights],Acc)->

U_W = case random:uniform() < MP of

true->

sat((random:uniform()-0.5)*?DELTA_MULTIPLIER+W,-

?SAT_LIMIT,?SAT_LIMIT);

false ->

W

end,

perturb_weights(MP,Weights,[U_W|Acc]);

perturb_weights(_MP,[],Acc)->

lists:reverse(Acc).

sat(Val,Min,Max)->

if

Val < Min -> Min;

Val > Max -> Max;

true -> Val

end.

%perturb_weights/3 accepts a probability value, a list of weights, and an empty list to act as an accumulator. The function then goes through the weight list perturbing each weight with a probability of MP. The weights are constrained to be within the range of -?SAT_LIMIT and SAT_LIMIT through the use of the sat/3 function.

7.6.8 Modifying the sensor Module

We have already created the xor_sim scape which expects a particular set of messages from the sensors and actuators interfacing with it. The scape expects the message: {Sensor_PId,sense} from the sensor, to which it responds with the sensory data sent in the: {Scape_PId,percept,SensoryVector} format. We now add to the sensor module a new function which can send and receive such messages. As we decided in the morphology module, the name of the new sensor will be xor_GetInput. The new function is shown in the following listing.

Listing-7.9 The implementation of the xor_GetInput sensor.

xor_GetInput(VL,Scape)->

Scape ! {self(),sense},

receive

{Scape,percept,SensoryVector}->

case length(SensoryVector)==VL of

7.7 Compiling Modules & Simulating the XOR Operation

true ->

SensoryVector;

false ->

io:format( “Error in sensor:xor_sim/2, VL:~p

SensoryVector:~p~n ”, [VL,SensoryVector]),

lists:duplicate(VL,0)

end

end.

%xor_GetInput/2 contacts the XOR simulator and requests the sensory vector, which in this

case should be a vector of length 2. The sensor checks that the incoming sensory signal, the

percept, is indeed of length 2. If the vector length differs, then this is printed to the console and

a dummy vector of appropriate length is constructed and used. This prevents unnecessary

crashes in the case of errors, and gives the researcher a chance to fix the error and hotswap the code.

7.6.9 Modifying the actuator Module

The scape expects the message: {Actuator_PId,action,Output} from the actua- tor, to which it responds with the: {Scape_PId,FItness,EndFlag} message. We de- cided in the morphology module to call the actuator that will interface with the xor_sim scape: xor_SendOutput. The following listing shows this newly added function to the actuator module.

Listing-7.10 The implementation of the xor_SendOutput actuator. xor_SendOutput(Output,Scape)->

Scape ! {self(),action,Output},

receive

{Scape,Fitness,HaltFlag}->

{Fitness,HaltFlag}

end.

%xor_SendOutput/2 function simply forwards the Output vector to the XOR simulator, and

then waits for the resulting Fitness and HaltFlag message from the scape.


We have now added all the new features, functions, and elements we needed to implement the learning algorithm with our NN system. When modifying our sys- tem to give it the ability to learn through the use of the augmented stochastic hill- climbing, we also implemented the xor_sim scape and the associated morphology with its sensors and actuators. We now compile all the modules we've created and


modified to see if they work, and then test if our system can indeed learn. To compile everything in one step, make sure you're in the folder where all the mod- ules are stored, and then execute the following command:

1>make:all([load]).

up_to_date

Now that everything is compiled, we can test to see if our NN can learn to sim- ulate the XOR operation. The XOR operation cannot be performed by a single neuron. To solve this problem by a strictly feed forward neural network, the min- imum required topology to perform this task is: [2,1], 2 neurons in the first layer and 1 in the output layer. We can specify the morphology, the NN topology, and the stopping condition, right from the trainer. The morphology, xor_mimic, speci- fies the problem we wish to apply the NN to, and the sensors and actuators that will interface with the xor_sim scape which will simulate the XOR operation and gage the NN's ability to simulate it. For the stopping condition we will choose to use fitness. We'll decide on the minimum fitness we wish our NN to achieve by calculating the total error in the NN's approximation of XOR that we're willing to accept. For example, a maximum error of 0.001 translates into a minimum fitness of 1/(0.01+ 0.00001), or: 99.9. Since we do not wish to use other stopping condi- tions, we'll set them to inf. Thus each training session will run until the NN can approximate XOR with an error no greater than 0.001 (a fitness no less than 99.9).

1>trainer:go(xor_mimic,[2],inf,inf,99.9).

Finished updating genotype to file:experimental

Cortex:<0.104.0> finished training. Genotype has been backed up.

Fitness:188.94639182995695

TotEvaluations:224

TotCycles:896

{cortex,cortex,

[{sensor,7.617035388076853e-10}],

[{actuator,7.617035388076819e-10}],

[{neuron,{1,7.61703538807679e-10}},

{neuron,{1,7.617035388076778e-10}},

{neuron,{2,7.617035388076755e-10}}]}

{sensor,{sensor,7.617035388076853e-10},

xor_GetInput,cortex,undefined,

{private,xor_sim},

2

[{neuron,{1,7.61703538807679e-10}},{neuron,{1,7.617035388076778e-10}}],

undefined,[],[]}

{neuron,{neuron,{1,7.61703538807679e-10}},

cortex,tanh,

[{{sensor,7.617035388076853e-10},


[-6.283185307179586,6.283185307179586]},

{bias,-6.283185307179586}],

[{neuron,{2,7.617035388076755e-10}}]}

{neuron,{neuron,{1,7.617035388076778e-10}},

cortex,tanh,

[{{sensor,7.617035388076853e-10},

[-5.663623085487123,6.283185307179586]},

{bias,6.283185307179586}],

[{neuron,{2,7.617035388076755e-10}}]}

{neuron,{neuron,{2,7.617035388076755e-10}},

cortex,tanh,

[{{neuron,{1,7.617035388076778e-10}},[-6.283185307179586]},

{{neuron,{1,7.61703538807679e-10}},[6.283185307179586]},

{bias,6.283185307179586}],

[{actuator,7.617035388076819e-10}]}

{actuator,{actuator,7.617035388076819e-10},

xor_SendOutput,cortex,undefined,

{private,xor_sim},

1

[{neuron,{2,7.617035388076755e-10}}],

undefined,[],[]}

Morphology:xor_mimic Best Fitness:188.94639182995695 EvalAcc:224

It works!. The trainer used the specified morphology to generate genotypes with the particular set of sensors and actuators to interface with the scape, and eventually produced a trained NN. In this particular case, the best fitness was 188.9, and it took only 224 evaluations to reach it. The trainer also used the geno- type:print/1 function to print the genotype topology to screen when it was done, which allows us to now analyze the genotype and double check it for accuracy.

Since the learning algorithm is stochastic, the number of evaluations it takes will differ from one attempt to another, some will go as high as a few thousand, other will stay in the hundreds. But if you've attempted this exercise, you might have also noted that you got roughly the same fitness, and this requires an expla- nation.

We're using tanh as the activation function. This activation function has 1 and - 1 as its limit, and because we're using the weight saturation limit set to 2*PI , the neuron's output can only get so close to -1 and 1 through the tanh function, and hence the final error in the XOR operation approximation. Thus the fitness we can achieve is limited by how close tanh(PI*2) can get to 1 , and tanh(-PI*2) can get to -1 . Since you and I are using the same SAT_LIMIT parameters, we can achieve the same maximum fitness scores, which is what we saw in the above result: (188.9), when using the SAT_LIMIT = 2*PI. If we for example modify the SAT_LIMIT in our neuron module as follows:


From: -define(SAT_LIMIT,math:pi()*2)

To: -define(SAT_LIMIT,math:pi()*20)

Then recompile and apply the NN to the problem again, then the best fitness will come out to be 99999.99 . Nevertheless, the SAT_LIMIT equaling to math:pi()*2 is high enough for most situations, and as noted, when we allow the weights to take a value of any magnitude, we run the risk of any one synaptic weight to overwhelm the whole neuron and make it essentially useless. Also, re- turning the neural weight that has ran afoul and exploded in magnitude back to an appropriate value would be difficult with small perturbations... Throughout the years I've found that having the SAT_LIMIT set to math:pi() or math:pi()*2 is the most effective choice.

We've now created a completely functional, static feed forward neural network system that can be applied to a lot of different problem types through the use of the system's scape , sensors , and actuators packages. It's an excellent start, and because our learning algorithm is unsupervised, we can even use our NNs as con- trollers in ALife. But there is also a problem, our NN system is only feedforward and so it does not possess memory, achievable through recursive connections. Al- so, our system only uses the tanh activation function, which might make problems like fourier analysis, fourier synthesis, and many other problems difficult to tackle. Our system can achieve an even greater flexibility if it can use other activation functions, and just like randomly choosing synaptic weights during neuron crea- tion, it should be able to randomly choose activation functions from some prede- termined list of said functions. Another problem we've noticed even when solving the XOR problem is that we had to know the minimal topology beforehand. If in the previous problem we would have chosen a topology of: [1] or [2], our NN would not have been able to solve that problem. Thus our NN has a flaw, the flaw is that we need to know the proper topology, or the minimal topology which can solve the problem we're applying our NN to. We need to devise a plan to over- come this problem, by letting our NN evolve topology as well. Another problem is that even though our NN “learns ”, it actually does not. It is, in reality, just being optimized by an external process, the exoself. What is missing is neural plasticity, the ability of the neurons to self modify based on sensory signals, and past experi- ence... that would be true learning, learning as self modification based on interac- tion with the environment within the lifetime of the organism. Finally, even though our system does have all the necessary features for us to start implement- ing supervision trees and process monitoring, we will first implement and develop the neuroevolutionary functionality, before transforming our system into a truly fault tolerant distributed CI system.

Before we continue on to the next chapter where we will start adding some of these mentioned features, and finally move to a population based approach and topological evolution, we first need to create a small benchmarking function. For example, we've used the trainer to solve the XOR problem and we noted that it

7.8 Adding the benchmarker Module

took K number of evaluations to solve it. When you've solved it with a trainer on your computer, you probably have gotten another value... we need to create a function that automates the process of applying the trainer to some problem many times, and then calculates the average performance of the system. We need a benchmarking method because it will allow us to calculate dependable average performance of our system and allow us to compare it to other machine learning approaches. It will also allow us to test new features and get a good idea of what affect they have on our system's performance across the spectrum of problems we might have in our benchmark suit. Thus, in the next section we will develop a small benchmarking function.


In this section we want to create a small system that performs benchmarking of the NN system we've developed, on some problem or problem set. This bench- marking system should be able to summon the trainer process X number of times, applying it to some problem of our choosing. After the benchmarker process has spawned the trainer, it should wait for it to finish optimizing the NN system. At some point the trainer will reach its stopping condition, and then send the bench- marking process the various performance statistics of the optimization run. For example the trainer could send to the benchmarker the number of evaluations it took to train the NN to solve some problem, the amount of time it took it, and the NN size required to solve it. The benchmarker should accumulate a list of these values, and once it has applied the trainer to the chosen problem X number of times, it should calculate the averages and standard deviations of these various performance statistics. Finally, our benchmarking system should print these per- formance results to console. At this point the researcher could use this perfor- mance data to for example compare his system to other state of the art machine learning algorithms, or the researcher could vary some parameter or add some new features to his system and benchmark it again, and in this manner see if the new features make the system more or less effective. Thus our benchmarker should perform the following steps:

1. Repeat:

2. Apply the trainer to some experiment or problem.

3. Receive from the trainer the resulting data (total evaluations, total cycles, NN size...).

4. Add this data to the statistics accumulator.

5. Until : The trainer has been applied to the given problem, X number of times. 6. Return : Calculate averages and standard deviations of the various features in

the accumulator.


The following listing shows the implementation of the benchmarker module. Listing-7.11: The implementation of the benchmarker module.

-module(benchmarker).

-compile(export_all).

-include( “records.hrl ”).

-define(MAX_ATTEMPTS,5).

-define(EVAL_LIMIT,inf).

-define(FITNESS_TARGET,inf).

-define(TOT_RUNS,100).

-define(MORPHOLOGY,xor_mimic).

go(Morphology,HiddenLayerDensities)->

go(Morphology,HiddenLayerDensities,?TOT_RUNS).

go(Morphology,HiddenLayerDensities,TotRuns)->

go(Morphology,HiddenLayerDensities,?MAX_ATTEMPTS,?EVAL_LIMIT,

?FITNESS_TARGET,TotRuns).

go(Morphology,HiddenLayerDensities,MaxAttempts,EvalLimit,FitnessTarget,TotRuns)->

PId = spawn(benchmarker,loop,[Morphology,HiddenLayerDensities,MaxAttempts,

EvalLimit,FitnessTarget,TotRuns,[],[],[],[]]),

register(benchmarker,PId).

% The benchmarker is started through the go/2, go/3, or go/6 function. The parameters the benchmark uses can be specified through the macros, and then used by executing go/2 or go/3 for which the researcher simply specifies the Morphology (the problem on which the NN will be benchmarked) and the HiddenLayerDensities (NN topology). The go/2 and go/3 functions execute go/6 function with default parameters. The benchmarker can also be started through go/6, using which the researcher can manually specify all the parameters: morphology, NN to- pology, Max Attempts, Max Evaluations, target fitness, and the total number of times to run the trainer. Before dropping into the main loop, go/6 registers the benchmarker process so that the trainer can send it the performance stats when it finishes.

loop(Morphology,_HiddenLayerDensities,_MA,_EL,_FT,0,FitnessAcc,EvalsAcc,CyclesAcc,

TimeAcc)->

io:format( “Benchmark results for:~p~n ”,[Morphology]),

io:format( “Fitness::~n Max:~p~n Min:~p~n Avg:~p~n Std:~p~n ”, [lists:max(FitnessAcc),lists:min(FitnessAcc),avg(FitnessAcc),std(FitnessAcc)]),

io:format( “Evals::~n Max:~p~n Min:~p~n Avg:~p~n Std:~p~n ”, [lists:max(EvalsAcc),lists:min(EvalsAcc),avg(EvalsAcc),std(EvalsAcc)]),

io:format( “Cycles::~n Max:~p~n Min:~p~n Avg:~p~n Std:~p~n ”, [lists:max(CyclesAcc),lists:min(CyclesAcc),avg(CyclesAcc),std(CyclesAcc)]),

io:format( “Time::~n Max:~p~n Min:~p~n Avg:~p~n Std:~p~n ”,

[lists:max(TimeAcc),lists:min(TimeAcc),avg(TimeAcc),std(TimeAcc)]);


loop(Morphology,HiddenLayerDensities,MA,EL,FT,BenchmarkIndex,FitnessAcc,EvalsAcc,

CyclesAcc,TimeAcc)->

Trainer_PId = trainer:go(Morphology,HiddenLayerDensities,MA,EL,FT),

receive

{Trainer_PId,Fitness,Evals,Cycles,Time}->

loop(Morphology,HiddenLayerDensities,MA,EL,FT,BenchmarkIndex-1,

[Fitness|FitnessAcc],[Evals|EvalsAcc],[Cycles|CyclesAcc],[Time|TimeAcc]);

terminate ->

loop(Morphology,HiddenLayerDensities,MA,EL,FT,0,FitnessAcc,EvalsAcc,

CyclesAcc,TimeAcc)

end.

% Once the benchmarker is started, it drops into its main loop. The main loop spawns the train- er and waits for it to finish optimizing the NN system, after which it sends to the benchmarker the performance based statistics. The benchmarker accumulates these performance statistics in lists, rerunning the trainer TotRuns number of times. Once the benchmarker has ran the trainer TotRuns number of times, indicated to be so when BenchmarkIndex reaches 0, it calculates the Max, Min, Average, and Standard Deviation values for every statistic list it accumulated.

avg(List)->

lists:sum(List)/length(List).

avg_std(List)->

Avg = avg(List),

std(List,Avg,[]).

std([Val|List],Avg,Acc)->

std(List,Avg,[math:pow(Avg-Val,2)|Acc]);

std([],_Avg,Acc)->

Variance = lists:sum(Acc)/length(Acc),

math:sqrt(Variance).

%avg/1 and std/1 functions calculate the average and the standard deviation values of the lists passed to them.

To make the whole system functional, we also have to slightly modify the trainer module so that when the stopping condition is reached, the trainer prints the genotype to console, unregisters itself, checks if a process by the name benchmarker exists, and if it does, sends it the performance stats of the optimiza- tion session. To make this modification, we add the following lines of code to our trainer module:

unregister(trainer),

case whereis(benchmarker) of

undefined ->

ok;

PId ->

PId ! {self(),BestFitness,EvalAcc,CAcc,TAcc}

end;


We now compile and recompile the benchmarker and the trainer modules re- spectively, and then test our new benchmarker system. To test it, we apply it to the XOR problem, executing it with the following parameters:

Morphology: xor_mimic

HiddenLayerDensities: [2]

MaxAttempts: inf

EvalLimit: inf

FitnessTarget: 100

TotRuns: 100

Based on these parameters, each trainer will generate genotypes until one of them solves the problem with a fitness of at least 100. Thus, the benchmarker will calculate the resulting performance statistics from 100 experiments. To start the benchmarker, execute the following command:

1>benchmarker:go(xor_mimic,[2],inf,inf,100,100).

Benchmark results for:xor_mimic

Fitness::

Max:99999.99999999999

Min:796.7693071321515

Avg:96674.025859051

Std:16508.11828048093

Evals::

Max:2222

Min:258

Avg:807.1

Std:415.8308670601546

...

It works! The benchmarker ran 100 training sessions and calculated averages, standard deviations, maxs, and mins for the accumulated Fitness, Evaluations, Cy- cles, and Time lists. Our system now has all the basic features of a solid machine learning platform.

7.9 Summary

In this chapter we added the augmented stochastic hill-climber optimization al- gorithm to our system, and extended the exoself process so that it can use it to tune its NN's synaptic weights. We also developed a trainer, a system which fur- ther extends the SHC optimization algorithm by restarting genotypes when the exoself had tuned the NN's synaptic weights and reached its stopping condition. This effectively allows the trainer process to use the Random Restart Stochastic


Hill Climbing optimization algorithm to train NNs. Finally, we created the benchmarker program, a system that can apply the trainer process to some prob- lem, X number of times, and then average the performance statistics and print the results to console.

Our NN system now has all the features necessary to solve and be applied to various problems and simulations. The learning algorithm our system implements is the simple yet very powerful augmented version of the random-restart stochastic hill-climber. We also now have a standardized method of presenting simulations, training scenarios, and problems to our NN system, all through the decoupled scape packages and morphologies.

In the next chapter we will take this system even further, combining it with population based evolutionary computation and topological mutation, thus creat- ing a simple topology and weight evolving artificial neural network system.

Chapter 8 Developing a Simple Neuroevolutionary Platform

Abstract In this chapter, we take our first step towards neuroevolution. Having developed a NN system capable of having its synaptic weights optimized, we will combine it with an evolutionary algorithm. We will create a population_monitor, a process that spawns a population of NN systems, monitors their performance, ap- plies a selection algorithm to the NNs in the population, and generates the mutant offspring from the fit NNs, while removing the unfit. In this chapter we also add topological mutation operators to our neuroevolutionary system, which will allow the population_monitor to evolve the NNs by adding new neural elements to their topologies. By the end of this chapter, our system becomes a fully-fledged Topol- ogy and Weight Evolving Artificial Neural Network.

In this book, we develop an entire neuroevolutionary platform, from a simple neuron, to an advanced topology and weight evolving artificial neural network platform. The final platform will be able to evolve fully distributed NNs, substrate encoded NNs, circuits, and NNs capable of learning within their lifetime through neural plasticity. Thus far we've developed a tuple based genotype encoder, a mapper from the genotype to the phenotype, a process based phenotype represen- tation, and an exoself program which runs outside the NN it's coupled to, capable of performing various assistive functions (though at this point exoself's only func- tion is the ability to tune the NN's weights, map between the genotype and pheno- type, and backup the NN's genotype to database). Finally, we also created a train- er program that generates and applies the NN systems to a problem specified by the researcher. The NNs generated by the trainer are all of the same topology, and the NNs are generated and applied to a problem in series, so that at any one time only a single NN is active. We now make our first leap towards neuroevolution.

Evolution is “the change over time of one or more inherited traits of individuals found in a population. ” When it comes to neuroevolution, the individuals of the population are neural network based systems. As we discussed in Chapter-4, a neuroevolutionary system performs the following steps:

1. Seed initial population of simple NNs.

2. Repeat:

3. Apply each NN in the population to some problem.

4. Calculate fitness score of each NN.

5. Using a selection algorithm, choose the most fit NNs of the population.

6. Let the fit NNs create offspring, where the offspring's genotype is generated through any of the following approaches:

DOI 10.1007/978-1-4614- 4463 - 3_8 , © Springer Science+Business Media New York 2013


– Mutation: by mutating the parent's genotype.

– Crossover: by somehow combining the genotypes of two or more fit parents. – Using a combination of mutation and crossover.

7. Create a new population composed of the fit parents and their offspring.

8. Until: A stopping condition (if any) is reached.

If we have been evolving the NNs for some particular problem or application, then once the stopping condition is reached, we can pick out the best performing NNs within the population, and count these NNs as solutions.

In the previous chapter, we have created a standardized method of training and applying NNs to problems through the use of scapes, which can gage the fitness of the NNs interfacing with them. Thus, we have the solution for step 3 (if we are to use more than one NN based system in parallel at any one time) and 4 of this loop. Step 5 requires us to develop a selection algorithm, a function which can pick out the most fit NNs in the population, and use them as the base from which to create the offspring for the next generation. This also means that since we will now deal with populations of NNs instead of a single NN, we will need to create some kind of database which can store these populations of genotypes. Step 6 requires that we create a function which can generate offspring that are based on, but genetical- ly differ from, their parents. This is done through mutation and crossover, and we will need to create modules that can perform these types of operations on the gen- otypes of fit NNs. For step 7 we will compose the new population by simply re- placing the unfit NNs in the population by the newly created offspring. For step 8, we can use the same approach to stopping conditions as we used in the trainer program. The evolutionary process will stop either when one of the NNs in the population has reached a level of fitness that we find high enough, or when there is innovation stagnation in the population. Meaning, the evolutionary process has stopped generating fitter organisms for a long enough time that makes us believe that a local or global optimum has been reached.

Whereas before the trainer program trained and dealt with a single NN at a time, we now need something that can monitor and supervise an entire population of NNs. Thus, we will remove the trainer program and develop a popula- tion_monitor program that can synchronize the evolutionary processes of a popu- lation of NN based intelligent agents. The population_monitor system shall be the one that will synchronize all these noted steps and functions, an independent pro- cess that constantly monitors the agents, and decides when and who creates off- spring... but because it is dependent on all these other parts, we will create and discuss in detail the population_monitor last.

                • Note********

Due to the source code heaviness of this chapter, it is essential that the comments within the presented source code be read. It is the comments that elaborate on, and explain how the pre- sented functions work, what they do, and how they do it.

8.1 The New Architecture

Before we begin putting together the neuroevolutionary platform, we will first create a diagram of the whole architecture, to try and visualize what new data structures we might need, as shown in Fig-8.1 .

Fig. 8.1 The architecture of a complete neuroevolutionary platform.

Figure 8.1 shows the diagram of the entire neuroevolutionary platform, and the manner in which the various modules are linked and related to each other. We are already familiar with NNs, private and public scapes, but the new elements of this architecture: database, population, species, Stat. Accumulator, and Error Logger, still need to be further explained.

We are now dealing with populations of NNs; we need a safe and secure way to store a large number of genotypes. This is done by using a stable and robust data- base, Mnesia. The database though does not start itself, so we need some kind of startup procedure for the whole neuroevolutionary platform so that when we start it, it starts mnesia and sets up all other types of global parameters and processes that might be necessary. In addition, as we discussed in the previous chapter, there are public and private scapes. The private scapes are summoned by each NN inde-


pendently, but the public scapes should already be running. These public scapes can be initiated and started during this initial startup procedure, indeed, these pub- lic scapes can belong, and be monitored by, some initial startup process. Also er- ror logging, and the gathering of, and accumulation of, statistics and system per- formance data, are all independent of the NNs and the evolutionary process, thus if we are to use these systems, they too should be started during the very start of the initialization of the neuroevolutionary platform. Thus we want to create a startup procedure that starts the Mnesia database, spawns the public scapes, the er- ror logger, and the statistics accumulator. This startup system basically creates the infrastructure needed for populations of NNs to exist, and for evolution to occur.

We need to give a name to the module in which this infrastructure will be cod- ed, and the process which in some sense represents this infrastructure. Let us call that module: polis, a Greek word that means an independent city state. Polis is the infrastructure necessary for the neuroevolutionary system to function, an infra- structure that brings together all the parts necessary for the NN based agents to ex- ist and evolve. Polis acts as a system in which all this functionality exists, it sum- mons the needed public scapes, and it is the top most monitoring element.

Once the infrastructure has been created, we can then start spawning popula- tions of NNs and begin applying them to some problem or simulation. The archi- tecture diagram of Fig-8.1 shows two independent populations, and that in itself requires an explanation. What is a population? A better question is, when do we need to create a population of NNs? We create a NN population when we are try- ing to solve a particular problem, or wish to apply our NNs to some simulation. Thus, a population of NNs is spawned for a particular purpose. Each simulation or application requires a specific set of morphologies (sensors, actuators, activation functions...), we can specify such constraints in the population data structure. The population element would at a high abstraction level dictate what the given group of NNs are allowed to interface with, what type of selection functions they are al- lowed to use, what activation functions they are allowed to use, what morpholo- gies make up the population … For this reason we should have multiple popula- tions, because we will be running multiple simulations and experiments using the same system, and we want to safely store these populations in the same database. If we allow the polis to have different populations, where each population is keep- ing track of its own NNs and the types of morphologies those NNs have access to, then we can run different experiments and simulations in parallel.

Furthermore, each population is composed of many evolving NNs. When NNs evolve, their genotypes change. When the difference of two genotypes is greater than some threshold X , then those genotypes belong to two different species. It would be useful for us to track speciation of NNs, and how these new species are formed, because speciation implies innovation. Thus we should also be able to group the NNs into different species through the use of a specie data structure. Each population should be broken down into species, where the number of species

8.2 The New Data Structures

and the size of each species, should be completely dynamic and depend on the NNs composing the population.

Finally, the Stat. Accumulator and Error Logger should be explained. We are now creating a rather large system, we need a way to keep track of errors, and a way to keep track of various statistics of the experiments we are running. For ex- ample we should keep track of how quickly the fitness is increasing on average, or how many evaluations it took to solve some problem, or how many generations, or what were the most and least fit NNs and when during evolution they came into ex- istence... or the running average fitness of some species... Keeping track of this will be the job of the Stat. Accumulator. Finally, the Error Logger is another pro- cess that should always be running, keeping track and catching any errors or alerts that occur during the time that the polis is online.

                • Note********

Technically an error logger already exists within the Erlang system itself. Thus we have a

choice of taking advantage of this already existing and robust system, or creating our own. Also the Stat. Accumulator can be implemented in many different ways, not all requiring it to be a completely separate process. For example the population_monitor will already have access to all the NNs belonging to a particular population. It will already have access to their sizes, fit- ness scores … and all other features, since it will be the one mutating them. Thus, the popula- tion_monitor can easily also perform the function of performance statistics accumulation and tracking.

Having now agreed on the new architecture of our system, we can start devis- ing the necessary data structures for our platform.


A population is a group of agents, in a neuroevolutionary system those agents are NN based systems. The genotypes of our NNs are represented as lists of rec- ords. Currently in our system, each NN genome is composed of a single cortex, one or more sensors, one or more actuators, and one or more neurons. Each ele- ment of the NN system knows what other elements it is connected to through ele- ment ids. But that is not the whole story, there is also meta-information that each NN should keep track off. For example, the NN topology that we specify during the genotype creation, such as: [1,2,3] , which specifies a NN with 1 neuron in the first layer, 2 neurons in the second, and 3 in the third, is an important piece of in- formation which can specify what specie this NN belongs to. Besides the NN to- pology, the following other features should be tracked by each NN:

1. id : The unique Id of the NN based agent by which the population_monitor can identify it, and contact it if necessary.


2. population_id : The Id of the population the NN belongs to.

3. specie_id : The Id of the specie the NN belongs to.

4. cx_id : The Id of the cortex of the NN, the cortex element which has the Ids of all the neurons, sensors and actuators of the NN.

5. fingerprint : The NN's particular “fingerprint ”, a tuple composed of the NN's topological structure, and the types of sensors and actuators it is using. Finger- prints can be used to calculate how much one NN system differs from another.

6. constraint : Constraint will define the NN's morphological type, what types of sensors, actuators, activation functions, and other features that the NN has access to during its evolution. Different species will have different constraints, and different constraints will define what different elements the NN and its off- spring can integrate during evolution. In essence, constraint keeps track of the available parameters, mutation operators, activation functions, morphologies, and various other parameters available to the NN as it evolves. It ensures that the NN using a particular constraint tuple produces offspring related to it in a particular manner, in a manner that ensures that the offspring too can be ap- plied to the same problem, or stay within certain specification constraints.

7. evo_hist : The evolutionary history of the NN, a list of mutation operators that were used on the seed NN system to evolve the current NN based system. It is the NN's evolutionary path from the simple topology it started with, to its cur- rent state. This way we can keep track of how the particular NN topology was reached, what path it took, and perhaps extract the why behind it all.

8. fitness : The NN's current fitness.

9. innovation_factor: The number of generations that have passed since the NN last increased in fitness.

10. pattern : The NN's topology.

11. generation : The generation to which this NN system belongs. The seed NN system has a generation of 0. When an offspring is created, its generation is that of its parent +1 .

Since the cortex already performs a specific function in the NN system, which is synchronizing the sensors, actuators, and neurons into a cohesive neurocomputational system, we should not overburden it with also having to track this new data. Instead what we will do is create a wrapper, another element to be added to the NN system's genotype. This new element will be part of the genotype and store this useful information. We will call this new element: agent , and it will store in itself these 11 noted features. Each NN based system will now also have its own id, the id of the agent element, an id by which it can be uniquely identified within a population. Finally, the reason why we name this new element agent, is because that is in essence what our NN based adaptive systems are, intelligent adaptive agents.

As we noted, each NN will belong to its own species , which is dependent on that NN's particular fingerprint. We thus also need to create a species data struc- ture. The species abstraction should keep track of the following information:


1.

2.

3.

4.

5.

6.

7.

8.

id : The species' unique Id.

fingerprint : The particular rough identification of the species, any NN with the same fingerprint belongs to this species.

agent_ids : The list of agent Ids which belong to this species.

champion_ids : A list of Ids of the best performing agents within the species, the species' champions.

avg_fitness : The average fitness of this species.

innovation_factor : Innovation factor is based on how long ago the average fit- ness of this species increased.

population_id : The Id of the population that this species belongs to. constraint : Constraint specifies the list of sensors, actuators, activation func- tions... that the agents belonging to this species have access to. And the name of the morphology of this species.

During one of our simulations we might want to start the experiment with many different species. Since the NNs depend on their morphologies, we can create a population with two different species, each with its own morphology. Then, when the NNs are created in those species, they would naturally start off with different sets of sensors and actuators, and the sensor and actuator sets available to them and belonging to the particular species they were seeded in. For example, this would be the case in the ALife simulation where we want to start the experi- ment with two separate species, predator and prey, each having its own different morphology, and access to its own set of different sensors and actuators.

An even higher level of abstraction is that of the population . A population con- tains all the NN based agents associated with some particular experiment or simu- lation. Thus if we wish to run multiple experiments, we don't want these NNs to intermingle; they belong to different worlds or simulation runs and experiments. To keep track of our populations, we will need to create a population abstraction. Through populations, we can for example stop an experiment at any time, at which point the NNs will be reverted back to their genotypes and stored in the da- tabase. We can then start another experiment with another population, which will have its own id. At some other point we might want to continue with our original simulation or experiment, and so all we would have to do is summon that original population, which would in return spawn all the NNs belonging to it, and the orig- inal simulation would continue from where it was left off. The population abstrac- tion should contain the following information:

1. id : A unique population Id or the id/name of the experiment.

2. specie_ids : The list of the species Ids that belong to this population.

3. avg_fitness : The average fitness of this population.

4. innovation_factor : Innovation factor is based on how long ago the average fit- ness of this population increased.

5. morphologies : List of available morphologies for this population. The list of morphologies defines the list of sensors and actuators available to the NNs in this population. Since the morphology defines the sensors and actuators of the


NN system, this list effectively defines the problem or simulation to which the evolving population of NN systems will be applied, and for what purpose the agents will be evolved.

Having now discussed what new data structures we need, it is time to begin de- veloping our neuroevolutionary platform. In the next section we will add the needed records to our records.hrl to accommodate these new data structures, and develop the polis module.

8.3 Developing the polis Module

The polis module should contain the functions that perform general, global tasks, and deal with initializing, starting, stopping and deleting the mnesia data- base used by the neuroevolutionary platform. Furthermore, it should also have the functions to initialize and start or spawn public scapes specified through their pa- rameters. Because there should be only one mnesia database per node, there needs to be only a single polis per node, representing a single neuroevolutionary plat- form. The following list summarizes the types of functions we want to be able to execute through the polis module:

1. Initialize new mnesia database.

2. Reset the mnesia database.

3. Start all the neuroevolutionary platform supporting processes (scapes, any error logging and statistics tracking programs...), so that the population_monitor sys- tems will have all the necessary infrastructure they need to apply evolutionary processes to their populations.

4. Stop and shut down the neuroevolutionary platform.

The following listing shows the polis module:

Listing-8.1: The polis module.

-module(polis).

%% API

-export([start/1,start/0,stop/0,init/2,create/0,reset/0,sync/0]).

%% gen_server callbacks

-export([init/1, handle_call/3, handle_cast/2, handle_info/2,terminate/2, code_change/3]). -behaviour(gen_server).

-include( “records.hrl ”).

%%=========================================== Polis Configuration Options

-record(state,{active_mods=[],active_scapes=[]}).

-record(scape_summary,{address,type,parameters=[]}).

-define(MODS,[]).

-define(PUBLIC_SCAPES,[ ]).


%The MODS list contains the names of the processes, functions, or other databases that also need to be executed and started when we start our neuroevolutionary platform. In the same manner, when we have created a new public scape, we can add a scape_summary tuple with this scape's information to the PUBLIC_SCAPES list, so that it is initialized and started with the system. The state record for the polis has all the elements needed to track the currently ac- tive mods and public scapes, which are either started during the startup of the neuroevolutionary platform, or spawned later, while the polis is already online.

%%=========================================== API

sync()->

make:all([load]).

% A sync/1 function can compile and reload all the modules pertaining to the project within the folder.

start() ->

case whereis(polis) of

undefined ->

gen_server:start(?MODULE, {?MODS,?PUBLIC_SCAPES}, []);

Polis_PId ->

io:format( “Polis:~p is already running on this node.~n ”,[Polis_PId])

end.

start(Start_Parameters) ->

gen_server:start(?MODULE, Start_Parameters, []).

init(Pid,InitState)->

gen_server:cast(Pid,{init,InitState}).

%The start/0 function first checks whether a polis process has already been spawned, by check- ing if one is registered. If it's not, then the start/1 function starts up the neuroevolutionary plat- form.

stop()->

case whereis(polis) of

undefined ->

io:format( “Polis cannot be stopped, it is not online~n ”);

Polis_PId ->

gen_server:cast(Polis_PId,{stop,normal})

end.

%The stop/0 function first checks whether a polis process is online. If there is an online polis process running on the node, then the stop function sends a signal to it requesting it to stop.

%%============================================ gen_server callbacks

init({Mods,PublicScapes}) ->

{A,B,C} = now(),

random:seed(A,B,C),

process_flag(trap_exit,true),

Chapter 8 Developing a Simple Neuroevolutionary Platform register(polis,self()),

io:format( “Parameters:~p~n ”,[{Mods,PublicScapes}]),

mnesia:start(),

start_supmods(Mods),

Active_PublicScapes = start_scapes(PublicScapes,[]),

io:format( “******** Polis: ##MATHEMA## is now online.~n ”),

InitState = #state{active_mods=Mods,active_scapes=Active_PublicScapes},

{ok, InitState}.

%The init/1 function first seeds random with a new seed, in the case a random number genera- tor will be needed. The polis process is then registered, the mnesia database is started, and the supporting modules, if any, are then started through the start_supmods/1 function. Then all the specified public scapes, if any, are activated. Having called our neuroevolutionary platform po- lis, we give this polis a name “MATHEMA ”, which is a Greek word for knowledge, and learn- ing. Finally we create the initial state, which contains the PIds of the currently active public scapes, and the names of the activated mods. Finally, the function then drops into the main gen_server loop.

handle_call({get_scape,Type},{Cx_PId,_Ref},S)->

Active_PublicScapes = S#state.active_scapes,

Scape_PId = case lists:keyfind(Type,3,Active_PublicScapes) of

false ->

undefined;

PS ->

PS#scape_summary.address

end,

{reply,Scape_PId,S};

handle_call({stop,normal},_From, State)->

{stop, normal, State};

handle_call({stop,shutdown},_From,State)->

{stop, shutdown, State}.

%At this point the polis only accepts a get_scape call, to which it replies with the PId or unde- fined message, and the two standard {stop,normal} and {stop,shutdown} calls.

handle_cast({init,InitState},_State)->

{noreply,InitState};

handle_cast({stop,normal},State)->

{stop, normal,State};

handle_cast({stop,shutdown},State)->

{stop, shutdown, State}.

%At this point the polis allows only for 3 standard casts: {init,InitState}, {stop,normal}, and {stop,shutdown}.

handle_info(_Info, State) ->

{noreply, State}.

%The handle_info/2 function is unused by the polis process at this time.


terminate(Reason, S) ->

Active_Mods = S#state.active_mods,

stop_supmods(Active_Mods),

stop_scapes(S#state.active_scapes),

io:format( “******** Polis: ##MATHEMA## is now offline, terminated with rea-

son:~p~n ”,[Reason]),

ok.

code_change(_OldVsn, State, _Extra) ->

{ok, State}.

%When polis is terminated, it first shuts down all the supporting mods by calling the stop_supmods/1 function, and then it shuts down all the public scapes by calling the stop_scapes/1 function.

%%--------------------------------------------------------------------

%%% Internal functions

%%--------------------------------------------------------------------

create()->

mnesia:create_schema([node()]),

mnesia:start(),

mnesia:create_table(agent,[{disc_copies, [node()]},{type,set},{attributes, rec-

ord_info(fields,agent)}]),

mnesia:create_table(cortex,[{disc_copies, [node()]},{type,set},{attributes, rec-

ord_info(fields,cortex)}]),

mnesia:create_table(neuron,[{disc_copies, [node()]},{type,set},{attributes, rec-

ord_info(fields,neuron)}]),

mnesia:create_table(sensor,[{disc_copies, [node()]},{type,set},{attributes, rec-

ord_info(fields,sensor)}]),

mnesia:create_table(actuator,[{disc_copies, [node()]},{type,set},{attributes, rec-

ord_info(fields,actuator)}]),

mnesia:create_table(population,[{disc_copies, [node()]},{type,set},{attributes, rec-

ord_info(fields,population)}]),

mnesia:create_table(specie,[{disc_copies, [node()]},{type,set},{attributes, rec-

ord_info(fields,specie)}]).

%The create/0 function sets up new mnesia database composed of the agent, cortex, neuron, sensor, actuator, polis, population, and specie tables.

reset()->

mnesia:stop(),

ok = mnesia:delete_schema([node()]),

polis:create().

%The reset/0 function deletes the schema, and recreates a fresh database from scratch.

%Start/Stop environmental modules: DBs, Environments, Network Access systems, and tools...


start_supmods([ModName|ActiveMods])->

ModName:start(),

start_supmods(ActiveMods);

start_supmods([])->

done.

%The start_supmods/1 function expects a list of module names of the mods that are to be start- ed with the startup of the neuroevolutionary platform. Each module must have a start/0 function that starts-up the supporting mod process.

stop_supmods([ModName|ActiveMods])->

ModName:stop(),

stop_supmods(ActiveMods);

stop_supmods([])->

done.

%The stop_supmods/1 expects a list of supporting mod names, the mod's name must be the name of its module, and that module must have a stop/0 function that stops the module. stop_supmods/1 goes through the list of the mods, and executes the stop() function for each one.

start_scapes([S|Scapes],Acc)->

Type = S#scape_summary.type,

Parameters = S#scape_summary.parameters,

{ok,PId} = scape:start_link({self(),Type,Parameters}),

start_scapes(Scapes,[S#scape_summary{address=PId}|Acc]);

start_scapes([],Acc)->

lists:reverse(Acc).

%The start_scapes/2 function accepts a list of scape_summary records, which specify the names of the public scapes and any parameters with which those scapes should be started. What specifies the scape which is going to be created by the scape module is the Type that is dropped into the function. Of course the scape module should already be able to create the Type of scape that is dropped into the start_link function. Once the scape is started, we record its PId in its scape_summary's record. Once all the public scapes have been started, the function returns a list of updated scape_summary records.

stop_scapes([S|Scapes])->

PId = S#scape_summary.address,

gen_server:cast(PId,{self(),stop,normal}),

stop_scapes(Scapes);

stop_scapes([])->

ok.

%The stop_scapes/1 function accepts a list of scape_summary records, and then stops all the scapes in the list. The function extracts a PId from every scape_summary in the list, and then requests the specified scapes to terminate themselves.


The polis process represents an interfacing point with the neuroevolutionary platform infrastructure. Through the polis module we can start and initialize the mnesia database that will support the evolutionary processes and store the geno- type of the NN based systems. Polis is the infrastructure and the system within which the database, the NN based agents, and the scapes they interface with, exist. It is for this reason that I gave this module the name polis, an independent and self governing city state of intelligent agents. Perhaps at some future time when multi- ple such systems are running on different nodes, each polis itself will have its own id, and each polis will concentrate on some particular goal towards which the neuroevolutionary system is aimed. It is only fitting to give this polis the name: “MATHEMA ”, which stands for knowledge and learning.

We saw in the previous listing that the create/0 function creates a new mnesia database composed of the following list of tables: sensor, actuator, neuron, cortex, agent, specie, and population. Our original records.hrl is still missing the follow- ing records to accommodate the listed tables: agent, specie, population, and polis. The updated records.hrl is shown in the following listing.

Listing-8.2: Updated contents of records.hrl

-record(sensor,{id,name,cx_id,scape,vl,fanout_ids=[]}).

-record(actuator,{id,name,cx_id,scape,vl,fanin_ids=[]}).

-record(neuron, {id, generation, cx_id, af, input_idps=[], output_ids=[], ro_ids=[]}). -record(cortex, {id, agent_id, sensor_ids=[], actuator_ids=[]}).

-record(agent,{id, generation, population_id, specie_id, cx_id,fingerprint, constraint,

evo_hist=[], fitness, innovation_factor, pattern=[]}).

-record(specie,{id,population_id, fingerprint, constraint, agent_ids=[], champion_ids=[], avg_fitness, innovation_factor}).

-record(population,{id,polis_id,specie_ids=[],morphologies=[],innovation_factor}).

As you've noted from the updated records.hrl file, we've also added the ele- ments: generation and ro_ids , to the neuron record. The generation element will track the number of generations that had passed since the last time the neuron was either mutated, or affected directly by some mutation which affects the NN's to- pology. In this way we can keep track of which parts of the NN system were most recently added to the network. The ro_ids (recurrent output ids) element keeps track of the recurrent output connections, and is a subset of the output_ids list. Thus, if a neuron A sends an output to a neuron B, and neuron B is located in the layer whose index is lower than A's layer, then B's id is entered not only into the output_ids list, but also into the ro_ids list. We need the ro_ids element because we need a way to track recurrent connections. Recurrent connections have to be treated differently than standard synaptic connections in a NN, as we will see and discuss in the following sections.

Before moving forward though, let's test the polis module and create a mnesia database. To do so, we first create the mnesia database by executing the function


polis:create() , and then test the polis starting and stopping functions by executing polis:start() and polis:stop() :

1> polis:create().

{atomic,ok}

2> polis:start().

Parameters:{[],[]}

                • Polis: ##MATHEMA## is now online.

{ok,<0.133.0>}

3> polis:stop().

ok

                • Polis: ##MATHEMA## is now offline, terminated with reason:normal

It works!. With the polis:create() function we created the necessary mnesia da- tabase tables, which we will need when we start testing other modules as we up- date and create them. The start function took the polis online successfully, and the stop function took the polis offline successfully. With this done, we can now move forward and begin updating the genotype module.

8.4 Updating the genotype Module

The genotype module encapsulates the NN based system creation and NN gen- otype access and storage. The move from ETS to Mnesia requires us to update the genotype access and storage functions. The addition of the agent record as part of the NN based system requires us to modify the NN system creation module. The new ability to add recursive connections to neurons will require us to rethink the way we represent the NN's topology, and the element's id structure.

Unlike in static NN based systems, topology and weight evolving artificial neu- ral network systems (TWEANNs) can modify the very topology and structure of a NN. We do not need to figure out what NN topology we should give to our NN system, because it will evolve the topology most optimal for the problem we give it. Plus, we never really know ahead of time what the most optimal NN topology needed to solve some particular problem anyway. The seed NN genotype should be the simplest possible, given the particular morphology of the agent, we let the neuroevolutionary process to complexify the topology of the NN system over time. Finally, because we will now use different kinds of activation functions, not only tanh but also sin, abs, sgn... we might wish for some species in the population to be started with a particular subset of these activation functions, and other spe- cies with another subset, to perhaps observe how and which evolutionary paths they take due to these different constraints. For this reason, we will also imple- ment a constraint record which the population_monitor can use when construct- ing agents. The constraint record specifies which morphology and which set of


activation functions the seed agent and its offspring should have access to during evolution. In the following subsections we discuss in more detail each of these factors, and then update our genotype module's source code.

8.4.1 Moving from ETS to Mnesia

Again, because we wrapped the retrieval and storage functions of the NN sys- tem elements inside their own functions within the genotype, the move from ETS to Mnesia will only necessitate the update of those specific functions. The main functions, read/2 and write/2, will need to be updated so that they use mnesia. The functions save_genotype/2, save_to_file/2, load_from_file/2, are no longer need- ed. Whereas the original save_genotype/2 function expected the genotype to be dropped in as a list, which it then entered into a table one record at a time, our new genotype will save each element as soon as it's created, instead of first forming a list of all the tuples and then saving them all at once. The save_to_file/2 and load_from_file/2 functions are ETS specific. Since mnesia is started and stopped by the polis, these two functions no longer have a role to play in our neuroevolutionary platform.

8.4.2 A NN Based Adaptive Agent

Our NN based adaptive system has been increasing in the number of elements, functions, and processes that it is composed of, and which define it. We now add another element to the genotype, the agent element. The agent element will store all the supporting information that the NN based system needs to keep track of the evolutionary history and other supplementary information not kept track of by the other elements. Neurons, Sensors, Actuators, and the Cortex elements keep track of only the data needed for their own specific functionality, whereas the agent el- ement will maintain all the global NN data (general topology of the NN, con- straints, specie id...). Thus, the complete NN based system genotype will now be composed of: one or more neuron records, one or more sensor records, one or more actuator records, a cortex record, and an agent record. Mirroring the geno- type, the phenotype of our NN based system is composed of: one or more neuron processes, one or more sensor processes, one or more actuator processes, the NN synchronizing cortex process, and the exoself process. In a way, the exoself pro- cess embodies the agent record of the genotype. The exoself process performs as- sistive services that might be needed by the NN system, it is the process that is ex- ternal to the NN, but is part of the whole NN based system, and it also has a PId that is unique to the agent, since an agent can have multiple sensors and actuators, but only a single exoself. It contains information about the entire NN based sys- tem; information that might be needed to restore and recover a NN, and perform


other numerous supportive tasks. This type of NN based system can also be re- ferred to as an adaptive agent, and thus for the remainder of the book, I will use the terms: “NN based system ”, “adaptive agent ”, and “agent ”, interchangeably.

8.4.3 A New Neuron Id Structure; Adding Recursive Connections

Neuroevolution allows for any neuron to make a connection to any other neu- ron, and that includes recursive connections. Our current neuron Id structure is a tuple containing the atom neuron (to identify the id as that which belongs to a neu- ron element), an integer based layer index (to specify where in the NN the neuron is located), and a unique Id (to ensure that every neuron id is unique): {{LayerIndex, Unique_Id}, neuron}. Using this neuron id structure, we can en- code both, feed forward, and recursive NNs... but there is a problem.

As shown in Fig-8.2 , imagine we have a NN which contains a neuron A in lay- er N and a neuron B in layer M. If we now apply a mutation to this NN, establish- ing a connection from neuron A to neuron B, and M > N, then the resulting NN is simply a feed forward connection, and everything works. If you we start with the same simple initial NN, and this time the mutation produces a connection from B to A, because M > N, the connection is recursive... everything still works ok. But what if during the mutation a new neural layer is added, right in the middle of lay- er N and M?

This scenario occurs when for example a new neuron is added to the NN, and it is not added to an existing layer, but instead it is added as an element of a new layer (thus increasing the depth of the NN), in the middle of two other existing neural layers. One of the possible mutation operators that produces this effect is the splice mutation operator. For example the splice mutation chooses a random neuron A in the NN, it then chooses a random output Id of neuron A to neuron B, disconnects neuron A from B, and then reconnects them through a newly created neuron C, placed in between them.


Fig. 8.2 Three types of mutations, showing the NN topologies before and after those muta- tions are applied. In A, a new link is created from neuron {1,A1} in layer 1, to neuron {2,B2} in layer 2. In B, a new recursive link is created from neuron {2,B2} in layer 2, to neuron {1,A2} in layer 1. In C, a splice is done, neurons {1,A2} and {2,B2} are un- linked/disconnected, and then relinked/connected through a new neuron, which is placed in a new layer between the two neurons. Due to a new layer, the Ids of B1 and B2 have to be adjusted, since the B1 and B2 neurons are moved from layer 2 to layer 3.

The problem is that when a mutation adds a new layer, it changes the topology, the layer indecies change for all the layers that come after the newly inserted lay- er. Since the layer index is a part of the neuron's id which is needed to keep track of whether the synaptic connections are feed forward or recursive, after such a mutation we must go in and update the ids of every neuron contained in the layers located after the newly inserted layer, and we need to update all the output id lists of the neurons which connect to these affected neurons. Since we don't know which neurons are affected and which neurons are connected to which ahead of time, after every such mutation, we have to access the NN genotype, go through


every neuron and update all the affected Ids. Then we would have to go through every sensor and actuator and update their fanout_ids and fanin_ids respectively as well, in case they are connected to any of the affected neurons. But there is a simpler solution.

We use layers only to keep track of whether the connections that are made be- tween the neurons are feed forward, or recurrent. Thus the most important part of the layer index is not its number, but the location on the number line, the order, mean- ing, whether a layer M is greater or smaller than layer N. So for example, if we have neurons in layer M=3 connected to neurons in layer N=4, and a mutation adds an- other layer between M and N, we can give this new layer an index of K = (N+M)/2, which is 3.5 in this case. Its value indicates that it is in the middle, thus if any neuron from layer M makes a connection to it, it will be feed forward. And if any neuron from layer N makes a connection to it, it will be recursive. All the necessary features are retained, using this method we can still properly track whether the connections are feedforward or recursive, and we do not have to update any of the already exist- ing ids when inserting a new layer, as shown in Fig-8.3 .

Eventually we will update our neuroevolutionary system to allow the NN based systems to modify their own topology, read their own NN topology using sensors, and change their own NN topology using actuators... Since the inputs and outputs of the NNs are usually normalized to be vectors containing values between -1 and 1 , we should think ahead and use a system where the layer indices are also all be- tween -1 and 1. This is easy, we simply decide that the sensors will be located at the -1 point, and so no neural layer can be located at -1. And we let the actuators be located at 1, and so no neural layer can be located at 1. Thus, when creating seed NN topologies, as discussed in the next section, we create them with that first initial neural layer at index 0, as shown in Fig-8.4 . Then, if a new layer is added after this initial layer, it is given an index at (0+1)/2 , if a new layer is added be- fore, then its index is set to (0+(-1))/2 . Once a new layer has been added after lay- er 0, we will now have the layers [0,0.5] composing the NN. If we need to add an- other layer after layer index 0.5, we follow the same pattern: (0.5 + 1)/2, thus the new layer index after 0.5, is 0.75. If on the other hand we need to add a new layer between 0.5 and 0, we give that layer an index of (0+0.5)/2, or 0.25. In this man- ner we can have infinitely many layers, and when adding or removing layers, none of the already existing neuron ids need to be modified or updated.

Fig. 8.3 A new way to designate neural layer indices, and the resulting ability to easily add and remove new layers without disrupting or having to update existing neural Ids.


8.4.4 Seed Computational Intelligence

We can neither predict nor derive ahead of time, not how large the NN should be nor what the topology of that NN should be, to optimally adapt to some envi- ronment, or solve some problem. The job of devising the proper topology, archi- tecture, functionality... everything, is that of evolution. Evolution alone decides what is fit to survive, and what is not. Evolution complexifies systems overtime, adding new features, elements, topological structures. Retaining what is useful, discarding what is not. Thus, we need only seed the minimal NN topologies, start with the simplest NN topologies and let evolution convert them into the more complex structures over time.

Fig. 8.4 Four types of minimalistic seed NN topologies are shown. Type A starts with a sin- gle Sensor and a single Actuator with an input vector length of 1, resulting in a NN with 1 neuron in layer 0 (the preferred seed topology). Type B starts with 1 sensor and 1 actuator whose input vl > 1. Type C starts with a single sensor and multiple actuators. And type D starts with multiple sensors and actuators.

The simplest seed NNs are composed of a single layer of neurons, connected from sensors, and connected to actuators, as shown in Fig-8.4 . The minimal start- ing topology depends on the total number of Sensors and Actuators the researcher decides to seed the population with. If the NN is set to start with P number of Sen- sors and one Actuator, where the actuator's input vector length is 1, then the seed NN starts with a single neuron connected from all the sensors and connected to a single actuator. If on the other hand the NN is initiated with P number of Sensors


and K number of actuators, the seed NNs will contain 1 layer of neurons, where each neuron is connected to a random subset of P Sensors, and to exactly 1 actua- tor. This neural layer will contain A 1 +...A k total Neurons, where A i is the size of the vector that is destined for each Actuator i . It is customary for the seed NNs to be initialized with a single Sensor and a single Actuator, letting the NN systems discover any other existing Sensors and Actuators through neuroevolution.

8.4.5 Constraints

The constraints should do just that, specify the set of general evolutionary guidelines for a particular organism, or neural network based system. The con- straints tuple specifies the morphology, and therefore the set of sensors and actua- tors that the NN system can draw its new sensor and actuator based elements from during evolution, and a set of neural activation functions: neural_afs . The idea be- hind constraints is that it allows us to start a population with multiple constraints, and therefore with multiple species and multiple fitness functions (one for each specie). For example, we can start a new population of size 100 with a list of two constraints, one whose morphology is prey , and the other whose morphology is predator . The prey and predator morphologies have different sets of sensors and actuators available for their species to draw from. The population monitor, seeing that there are two constraints, could then create the population composed of two species, each species of size 50, and each species using its own constraints. Since the constraints are inherited by the agent systems, the offspring an agent produces would be created based on its constraints. Constraints also allow us to perform the following experiment: Assume that we would like to see whether neural networks that use tanh and sin activation functions, or whether a neural networks that use sin, abs , and gauss activation functions, evolve a more profitable stock trading NN based agent. We can do this by starting a population with a list of two constraints which use the same morphology, but different neural_afs lists. This would divide the population into two seed species, each with its own set of neural_afs to draw from. We could even create two species of the same type/morphology, and release them into a public scape, where the only difference between the two species is their constraints with regards to neural_afs available to the evolving agents. This would allow us to see what set of neural_afs is better suited to evolve intelli- gent/adaptive systems faster and more effectively in the provided environment. It would even allow us to let the two species of the same type/morphology compete and fight each other, demonstrating which set of activation functions is more fit in that way. Finally, the constraint record also simply allows us to specify the mor- phology and neural_afs that we'd like to use during any particular simulation or experiment, the list of sensors, actuators, and activation functions that we would like our evolving NN based systems to incorporate their elements from, as they try to solve some particular problem. The constraint record we will add to our rec- ords.hrl file will use the following format:


-record(constraint,{morphology=[], neural_afs=[]}).

During the seed population creation, the function constructing the genotypes of the NN based systems would then be passed the specie id that the agents will be- long to, the id that the created agent should use, and the constraint record, set within the population_monitor beforehand.

8.4.6 The Updated genotype Module

Having now covered all the new features and properties of the genotype mod- ule, we can put these parts together: The construct_Agent function accepts as its parameters the specie_Id, agent_Id , and the constraint tuple. It then uses these pa- rameters to construct a seed population. The NNs in this seed population use min- imalistic topologies, connected to and from one of the sensors and actuators be- longing to the morphology specified within the constraints. Each neuron is created with a random set of weights but without a bias/threshold value, which can be lat- er incorporated through a mutation if needed. Finally, the activation function for the neuron is randomly chosen from the neural_afs list specified within the con- straint record.

Beyond these additions, we add to the genotype module three new functions: delete_Agent/1 , clone_Agent/2 , mutate/1 , and test/0 . The first three will be neces- sary when we begin deleting unfit NNs, and creating offspring based on the fit NNs. One of the ways to create an offspring based on a fit genotype in the population is by first cloning that genotype, and then mutating it, which is the approach we take for offspring creation in our neuroevolutionary platform. All mutation opera- tor functions will be kept in one module, while the mutate function wrapper is kept in the genotype module. This will allow us to keep the genome mutator mod- ule indifferent to where and how the genotype is stored, since the genotype mod- ule will keep track of that. In this manner, it will be mostly the genotype module which will need to deal with mnesia transactions, and be aware of how the ge- nome is stored. The fourth function we create will be test/0, we will use this func- tion to test whether our module can create a new agent, then clone it, then print the genotypes of the two agents (original and its clone) to console, and then finally de- lete them both.

The following listing shows the source code for the updated genotype module: Listing-8.3 The updated genotype module.

-module(genotype).

-compile(export_all).

-include( “records.hrl ”).


construct_Agent(Specie_Id,Agent_Id,SpecCon)->

random:seed(now()),

Generation = 0,

{Cx_Id,Pattern} = construct_Cortex(Agent_Id,Generation,SpecCon),

Fingerprint = create_fingerprint(Agent_Id),

Agent = #agent{

id = Agent_Id,

cx_id = Cx_Id,

specie_id = Specie_Id,

fingerprint = Fingerprint,

constraint = SpecCon,

generation = Generation,

pattern = Pattern,

evo_hist = []

},

write(Agent).

%The population monitor should have all the information with regards to the morphologies and species constraints under which the agent's genotype should be created. Thus the con- struct_Agent/3 is ran with the parameter Specie_Id to which this NN based system will belong, the Agent_Id that this NN based intelligent agent will have, and the SpecCon (specie con- straint) that will define the list of activation functions and other parameters from which the seed agent can choose its parameters. In this function, first the generation is set to 0, since the agent is just created, then the construct_Cortex/3 is ran, which creates the NN and returns its Cx_Id. Once the NN is created and the cortex's id is returned, we can fill out the information needed by the agent record, and then finally write it to the mnesia database

construct_Cortex(Agent_Id,Generation,SpecCon)->

Cx_Id = {{origin,generate_UniqueId()},cortex},

Morphology = SpecCon#constraint.morphology,

Sensors = [S#sensor{id={{-1,generate_UniqueId()},sensor},cx_id=Cx_Id}|| S <- morphology:get_InitSensors(Morphology)],

Actuators = [A#actuator{id={{1,generate_UniqueId()},actuator},cx_id=Cx_Id}||A<-

morphology:get_InitActuators(Morphology)],

N_Ids=construct_InitialNeuroLayer(Cx_Id,Generation,SpecCon,Sensors,Actuators,[],[]), S_Ids = [S#sensor.id || S<-Sensors],

A_Ids = [A#actuator.id || A<-Actuators],

Cortex = #cortex{

id = Cx_Id,

agent_id = Agent_Id,

neuron_ids = N_Ids,

sensor_ids = S_Ids,

actuator_ids = A_Ids

},

write(Cortex),

{Cx_Id,[{0,N_Ids}]}.


%construct_Cortex/3 generates a new Cx_Id, extracts the morphology from the constraint record passed to it in SpecCon, and then extracts the initial sensors and actuators for that mor- phology. After the sensors and actuators are extracted, the function calls con- struct_InitialNeuroLayer/7, which creates a single layer of neurons connected from the speci- fied sensors and to the specified actuators, and then returns the ids of the created neurons. Finally, the sensor and actuator ids are extracted from the sensors and actuators, and the cortex record is composed and written to the database.

construct_InitialNeuroLayer(Cx_Id,Generation,SpecCon,Sensors,[A|Actuators],

AAcc,NIdAcc)->

N_Ids = [{{0,Unique_Id},neuron}|| Unique_Id<-generate_ids(A#actuator.vl,[])],

U_Sensors=construct_InitialNeurons(Cx_Id,Generation,SpecCon,N_Ids,Sensors,A),

U_A = A#actuator{fanin_ids=N_Ids},

construct_InitialNeuroLayer(Cx_Id,Generation,SpecCon,U_Sensors,Actuators,

[U_A|AAcc],lists:append(N_Ids,NIdAcc));

construct_InitialNeuroLayer(_Cx_Id,_Generation,_SpecCon,Sensors,[],AAcc,NIdAcc)-> [write(S) || S <- Sensors],

[write(A) || A <- AAcc],

NIdAcc.

%construct_InitialNeuroLayer/7 creates a set of neurons for each Actuator in the actuator list. The neurons are initialized in the construct_InitialNeurons/6, where they are connected to the actuator, and from a random subset of the sensors passed to the function. The con- struct_InitialNeurons/6 function returns the updated sensors, some of which have now an up- dated set of fanout_ids which includes the new neuron ids they were connected to. The actua- tor's fanin_ids is then updated to include the neuron ids that were connected to it. Once all the actuators have been connected to, the sensors and the actuators are written to the database, and the set of neuron ids created within the function is returned to the caller.

construct_InitialNeurons(Cx_Id,Generation,SpecCon,[N_Id|N_Ids], Sensors,Actuator)-> case random:uniform() >= 0.5 of

true ->

S = lists:nth(random:uniform(length(Sensors)),Sensors),

U_Sensors = lists:keyreplace(S#sensor.id, 2, Sensors,

S#sensor{fanout_ids=[N_Id|S#sensor.fanout_ids]}),

Input_Specs = [{S#sensor.id,S#sensor.vl}];

false ->

U_Sensors = [S#sensor{fanout_ids=[N_Id|S#sensor.fanout_ids]} || S <-

Sensors],

Input_Specs=[{S#sensor.id,S#sensor.vl}||S<-Sensors]

end,

construct_Neuron(Cx_Id,Generation,SpecCon,N_Id,Input_Specs, [Actua-

tor#actuator.id]),

construct_InitialNeurons(Cx_Id,Generation,SpecCon,N_Ids, U_Sensors,Actuator);

construct_InitialNeurons(_Cx_Id,_Generation,_SpecCon,[],Sensors,_Actuator)->

Sensors.


%construct_InitialNeurons/6 accepts the list of sensors and a single actuator, connects each neuron to the actuator, and randomly chooses whether to connect it from all the sensors or a subset of the given sensors. Once all the neurons have been connected to the actuator and from the sensors, the updated sensors whose fanout_ids have been updated with the ids of the neu- rons, are returned to the caller.

construct_Neuron(Cx_Id,Generation,SpecCon,N_Id,Input_Specs,Output_Ids)->

Input_IdPs = create_InputIdPs(Input_Specs,[]),

Neuron=#neuron{

id=N_Id,

cx_id = Cx_Id,

generation=Generation,

af=generate_NeuronAF(SpecCon#constraint.neural_afs),

input_idps=Input_IdPs,

output_ids=Output_Ids,

ro_ids = calculate_ROIds(N_Id,Output_Ids,[])

},

write(Neuron).

create_InputIdPs([{Input_Id,Input_VL}|Input_IdPs],Acc) ->

Weights = create_NeuralWeights(Input_VL,[]),

create_InputIdPs(Input_IdPs,[{Input_Id,Weights}|Acc]);

create_InputIdPs([],Acc)->

Acc.

create_NeuralWeights(0,Acc) ->

Acc;

create_NeuralWeights(Index,Acc) ->

W = random:uniform()-0.5,

create_NeuralWeights(Index-1,[W|Acc]).

%Each neuron record is composed by the construct_Neuron/6 function. The con- struct_Neuron/6 creates the Input list from the tuples [{Id,Weights}...] using the vector lengths specified in the Input_Specs list. The create_InputIdPs/3 function uses create_NeuralWeights/2 to generate the random weights in the range of -0.5 to 0.5. The activation function that the neu- ron uses is chosen randomly from the neural_afs list within the constraint record passed to the construct_Neuron/6 function. construct_Neuron uses calculate_ROIds/3 to extract the list of re- cursive connection ids from the Output_Ids passed to it. Once the neuron record is filled in, it is saved to database.

generate_NeuronAF(Activation_Functions)->

case Activation_Functions of

[] ->

tanh;

Other ->

lists:nth(random:uniform(length(Other)),Other)


end.

%The generate_NeuronAF/1 accepts a list of activation function tags, and returns a randomly chosen one. If an empty list was passed as the parameter, the function returns the standard tanh tag.

calculate_ROIds(Self_Id,[Output_Id|Ids],Acc)->

case Output_Id of

{_,actuator} ->

calculate_ROIds(Self_Id,Ids,Acc);

Output_Id ->

{{TLI,_},_NodeType} = Self_Id,

{{LI,_},_} = Output_Id,

case LI =< TLI of

true ->

calculate_ROIds(Self_Id,Ids,[Output_Id|Acc]);

false ->

calculate_ROIds(Self_Id,Ids,Acc)

end

end;

calculate_ROIds(_Self_Id,[],Acc)->

lists:reverse(Acc).

%The function calculate_ROIds/3 accepts as input the Self_Id of the neuron, and the Out- put_Ids of the elements the neuron connects to. Since each element specifies its type and, in the case of neurons, the layer index it belongs to, the function checks if the Output_Id's layer index is lower than the Self_Id's layer index. If it is, the output connection is recursive and the Out- put_Id is added to the recursive output list. Once the recursive connection ids have been ex- tracted from the Output_Ids, the extracted id list is returned to the caller.

generate_ids(0,Acc) ->

Acc;

generate_ids(Index,Acc)->

Id = generate_UniqueId(),

generate_ids(Index-1,[Id|Acc]).

generate_UniqueId()->

{MegaSeconds,Seconds,MicroSeconds} = now(),

1/(MegaSeconds*1000000 + Seconds + MicroSeconds/1000000).

%The generate_UniqueId/0 creates a unique Id using current time, the Id is a floating point val- ue. The generate_ids/2 function creates a list of unique Ids.

create_fingerprint(Agent_Id)->

A = read({agent,Agent_Id}),

Cx = read({cortex,A#agent.cx_id}),

GeneralizedSensors = [(read({sensor,S_Id}))#sensor{id=undefined,cx_id=undefined} ||

S_Id<-Cx#cortex.sensor_ids],


GeneralizedActuators = [(read({sensor,A_Id}))#actuator{id=undefined,cx_id=undefined} ||

A_Id<-Cx#cortex.actuator_ids],

GeneralizedPattern = [{LayerIndex,length(LNIds)}||{LayerIndex,LNIds}<-A#agent.pattern],

GeneralizedEvoHist = generalize_EvoHist(A#agent.evo_hist,[]),

{GeneralizedPattern,GeneralizedEvoHist,GeneralizedSensors,GeneralizedActuators}. %create_fingerprint/1 calculates the fingerprint of the agent, where the fingerprint is just a tuple of the various general features of the NN based system, a list of features that play some role in distinguishing its genotype's general properties from those of other NN systems. Here, the fin- gerprint is composed of the generalized pattern (pattern minus the unique ids), generalized evo- lutionary history (evolutionary history minus the unique ids of the elements), a generalized sen- sor set, and a generalized actuator set of the agent in question.

generalize_EvoHist([{MO,{{ALI,_AUId},AType},{{BLI,_BUId},BType}, {{CLI,_CUId},

CType}}|EvoHist],Acc)->

generalize_EvoHist(EvoHist,[{MO,{ALI,AType},{BLI,BType}, {CLI,CType}}|Acc]);

generalize_EvoHist([{MO,{{ALI,_AUId},AType},{{BLI,_BUId},BType}}|EvoHist],Acc)->

generalize_EvoHist(EvoHist,[{MO,{ALI,AType},{BLI,BType}}|Acc]);

generalize_EvoHist([{MO,{{ALI,_AUId},AType}}|EvoHist],Acc)->

generalize_EvoHist(EvoHist,[{MO,{ALI,AType}}|Acc]);

generalize_EvoHist([],Acc)->

lists:reverse(Acc).

%generalize_EvoHist/2 generalizes the evolutionary history tuples by removing the unique el- ement ids. Two neurons which are using exactly the same activation function, located in exactly the same layer, and using exactly the same synaptic weights, will still have different unique ids. Thus, these ids must be removed to produce a more general set of tuples. There are 3 types of tuples in evo_hist list, with 3, 2 and 1 element ids. Once the evolutionary history list is general- ized, it is returned to the caller.

read(TnK)->

case mnesia:read(TnK) of

[] ->

undefined;

[R] ->

R

end.

%read/1 accepts the tuple composed of a table name and a key: {TableName,Key}, which it then uses to read from the mnesia database and return the record or the atom: undefined, to the caller.

write(R)->

mnesia:write(R).

% write/1 accepts a record and writes it to the database

delete(TnK)->

mnesia:delete(TnK).


% delete/1 accepts the parameter tuple: {TableName,Key}, and deletes the associated record from the table.

print(Agent_Id)->

A = read({agent,Agent_Id}),

Cx = read({cortex,A#agent.cx_id}),

io:format( “~p~n ”,[A]),

io:format( “~p~n ”,[Cx]),

[io:format( “~p~n ”,[read({sensor,Id})]) || Id <- Cx#cortex.sensor_ids],

[io:format( “~p~n ”,[read({neuron,Id})]) || Id <- Cx#cortex.neuron_ids],

[io:format( “~p~n ”,[read({actuator,Id})]) || Id <- Cx#cortex.actuator_ids].

%print/1 accepts an agent's id, finds all the elements composing the agent in question, and prints out the complete genotype of the agent.

delete_Agent(Agent_Id)->

A = read({agent,Agent_Id}),

Cx = read({cortex,A#agent.cx_id}),

[delete({neuron,Id}) || Id <- Cx#cortex.neuron_ids],

[delete({sensor,Id}) || Id <- Cx#cortex.sensor_ids],

[delete({actuator,Id}) || Id <- Cx#cortex.actuator_ids],

delete({cortex,A#agent.cx_id}),

delete({agent,Agent_Id}).

%delete_Agent/1 accepts the id of an agent, and then deletes that agent's genotype. This func- tion assumes that the id of the agent will be removed from the specie's agent_ids list, and any other needed clean up procedure will be performed by the calling function.

delete_Agent(Agent_Id,safe)->

F = fun()->

A = genotype:read({agent,Agent_Id}),

S = genotype:read({specie,A#agent.specie_id}),

Agent_Ids = S#specie.agent_ids,

write(S#specie{agent_ids = lists:delete(Agent_Id,Agent_Ids)}),

delete_Agent(Agent_Id)

end,

Result=mnesia:transaction(F),

io:format( “delete_agent(Agent_Id,safe):~p Result:~p~n ”,[Agent_Id,Result]). %delete_Agent/2 accepts the id of an agent, and then deletes that agent's genotype, but ensures that the species to which the agent belongs, has its agent_ids element updated. Unlike de- lete_Agent/1, this function updates the species' record.

clone_Agent(Agent_Id,CloneAgent_Id)->

F = fun()->

A = read({agent,Agent_Id}),

Cx = read({cortex,A#agent.cx_id}),

IdsNCloneIds = ets:new(idsNcloneids,[set,private]),


ets:insert(IdsNCloneIds,{threshold,threshold}),

ets:insert(IdsNCloneIds,{Agent_Id,CloneAgent_Id}),

[CloneCx_Id] = map_ids(IdsNCloneIds,[A#agent.cx_id],[]),

CloneN_Ids = map_ids(IdsNCloneIds,Cx#cortex.neuron_ids,[]),

CloneS_Ids = map_ids(IdsNCloneIds,Cx#cortex.sensor_ids,[]),

CloneA_Ids = map_ids(IdsNCloneIds,Cx#cortex.actuator_ids,[]),

clone_neurons(IdsNCloneIds,Cx#cortex.neuron_ids),

clone_sensors(IdsNCloneIds,Cx#cortex.sensor_ids),

clone_actuators(IdsNCloneIds,Cx#cortex.actuator_ids),

write(Cx#cortex{

id = CloneCx_Id,

agent_id = CloneAgent_Id,

sensor_ids = CloneS_Ids,

actuator_ids = CloneA_Ids,

neuron_ids = CloneN_Ids

}),

write(A#agent{

id = CloneAgent_Id ,

cx_id = CloneCx_Id

}),

ets:delete(IdsNCloneIds)

end,

mnesia:transaction(F).

%clone_Agent/2 accepts Agent_Id and CloneAgent_Id as parameters, and then clones the agent, giving the clone the CloneAgent_Id. The function first creates an ETS table to which it writes the ids of all the elements of the genotype and their correspondingly generated clone ids. Once all ids and clone ids have been generated, the function begins to clone the actual ele- ments. clone_Agent/2 first clones the neurons using clone_neurons/2, then the sensors using clone_sensors/2, and finally the actuators using clone_actuators. Once these elements are cloned, the function writes to database the clone versions of the cortex and the agent records, by writing to database the original records with updated clone ids.

map_ids(TableName,[Id|Ids],Acc)->

CloneId=case Id of

{{LayerIndex,_NumId},Type}-> %maps neuron and cortex ids. {{LayerIndex,generate_UniqueId()},Type};

{_NumId,Type}-> %maps sensor and actuator ids. {generate_UniqueId(),Type}

end,

ets:insert(TableName,{Id,CloneId}),

map_ids(TableName,Ids,[CloneId|Acc]);

map_ids(_TableName,[],Acc)->

Acc.


%map_ids/3 accepts the name of the ets table, and a list of ids as parameters. It then goes

through every id and creates a clone version of the id by generating a new unique id. The func- tion is able to generate new id structures for neuron, cortex, sensor, and actuator id types.

clone_sensors(TableName,[S_Id|S_Ids])->

S = read({sensor,S_Id}),

CloneS_Id = ets:lookup_element(TableName,S_Id,2),

CloneCx_Id = ets:lookup_element(TableName,S#sensor.cx_id,2),

CloneFanout_Ids =[ets:lookup_element(TableName,Fanout_Id,2)|| Fanout_Id <-

S#sensor.fanout_ids],

write(S#sensor{

id = CloneS_Id,

cx_id = CloneCx_Id,

fanout_ids = CloneFanout_Ids

}),

clone_sensors(TableName,S_Ids);

clone_sensors(_TableName,[])->

done.

%clone_sensors/2 accepts as input the name of the ets table and the list of sensor ids. It then

goes through every sensor id, reads the sensor from the database, and updates all its ids (id,

cx_id, and fanout_ids) from their original values to their clone version values stored in the ets table. Afterwards, the new version of the sensor is written to database, effectively cloning the original sensor.

clone_actuators(TableName,[A_Id|A_Ids])->

A = read({actuator,A_Id}),

CloneA_Id = ets:lookup_element(TableName,A_Id,2),

CloneCx_Id = ets:lookup_element(TableName,A#actuator.cx_id,2),

CloneFanin_Ids =[ets:lookup_element(TableName,Fanin_Id,2)|| Fanin_Id <-

A#actuator.fanin_ids],

write(A#actuator{

id = CloneA_Id,

cx_id = CloneCx_Id,

fanin_ids = CloneFanin_Ids

}),

clone_actuators(TableName,A_Ids);

clone_actuators(_TableName,[])->

done.

%clone_actuators/2 accepts as input the name of the ets table and the list of actuator ids. It then goes through every actuator id, reads the actuator from the database, and updates all its ids (id, cx_id, and fanin_ids) from their original values to their clone version values stored in the ets ta- ble. Afterwards, the new version of the actuator is written to database, effectively cloning the original actuator.

clone_neurons(TableName,[N_Id|N_Ids])->


N = read({neuron,N_Id}),

CloneN_Id = ets:lookup_element(TableName,N_Id,2),

CloneCx_Id = ets:lookup_element(TableName,N#neuron.cx_id,2),

CloneInput_IdPs = [{ets:lookup_element(TableName,I_Id,2),Weights}||

{I_Id,Weights} <- N#neuron.input_idps],

CloneOutput_Ids = [ets:lookup_element(TableName,O_Id,2)|| O_Id <-

N#neuron.output_ids],

CloneRO_Ids =[ets:lookup_element(TableName,RO_Id,2)|| RO_Id <-

N#neuron.ro_ids],

write(N#neuron{

id = CloneN_Id,

cx_id = CloneCx_Id,

input_idps = CloneInput_IdPs,

output_ids = CloneOutput_Ids,

ro_ids = CloneRO_Ids

}),

clone_neurons(TableName,N_Ids);

clone_neurons(_TableName,[])->

done.

%clone_neuron/2 accepts as input the name of the ets table and the list of neuron ids. It then goes through every neuron id, reads the neuron from the database, and updates all its ids (id, cx_id, output_ids, ro_ids, and input_idps) from their original values to their clone version val- ues stored in the ets table. Once everything is updated, the new (clone) version of the neuron is written to database.

test()->

Specie_Id = test,

Agent_Id = test,

CloneAgent_Id = test_clone,

SpecCon = #constraint{},

F = fun()->

construct_Agent(Specie_Id,Agent_Id,SpecCon),

clone_Agent(Specie_Id,CloneAgent_Id),

print(Agent_Id),

print(CloneAgent_Id),

delete_Agent(Agent_Id) ,

delete_Agent(CloneAgent_Id)

end,

mnesia:transaction(F).

%test/0 performs a test of the standard functions of the genotype module, by first creating a new agent, then cloning that agent, then printing the genotype of the original agent and its clone, and then finally deleting both of these agents.

create_test()->

Specie_Id = test,


Agent_Id = test,

SpecCon = #constraint{},

F = fun()->

case genotype:read({agent,test}) of

undefined ->

construct_Agent(Specie_Id,Agent_Id,SpecCon),

print(Agent_Id);

_ ->

delete_Agent(Agent_Id),

construct_Agent(Specie_Id,Agent_Id,SpecCon),

print(Agent_Id)

end

end,

mnesia:transaction(F).

%create_test/0 creates a simple NN based agent using the default constraint record. The func-

tion first checks if an agent with the id ‘test' already exists, if it does, the function deletes that agent and creates a new one. Otherwise, the function just creates a brand new agent with the id ‘test'.

Having now updated the genotype, let us compile and test it using the test/0 and create_test/0 functions. Since we already compiled and tested polis in the previous section, at this point we already have a mnesia database (if you have not yet com- piled polis and ran polis:create(), do so before testing the genotype module). To test the genotype module, we first take the polis online, then run the geno- type:test() function, and then take the polis offline, as is shown next:

1> polis:start().

Parameters:{[],[]}

                • Polis: ##MATHEMA## is now online.

{ok,<0.34.0>}

2> genotype:test().

{agent,test,0,undefined,test,

{{origin,7.522621162363539e-10},cortex},

{[{0,1}],

[],

[{sensor,undefined,xor_GetInput,undefined,

{private,xor_sim},

2

[{{0,7.522621162361355e-10},neuron}],

undefined}],

[{actuator,undefined,xor_SendOutput,undefined,

{private,xor_sim},

1

[{{0,7.522621162361355e-10},neuron}],

undefined}]},

Chapter 8 Developing a Simple Neuroevolutionary Platform {constraint,xor_mimic,[tanh,cos,gauss,abs]},

[],undefined,0,

[{0,[{{0,7.522621162361355e-10},neuron}]}]}

{cortex,{{origin,7.522621162363539e-10},cortex},

test,

[{{0,7.522621162361355e-10},neuron}],

[{{-1,7.522621162361434e-10},sensor}],

[{{1,7.522621162361411e-10},actuator}]}

{sensor,{{-1,7.522621162361434e-10},sensor},

xor_GetInput,

{{origin,7.522621162363539e-10},cortex},

{private,xor_sim},

2

[{{0,7.522621162361355e-10},neuron}],

undefined}

{neuron,{{0,7.522621162361355e-10},neuron},

0

{{origin,7.522621162363539e-10},cortex},

tanh,

[{{{-1,7.522621162361434e-10},sensor},

[-0.20275596630526205,0.14421756025063392]}],

[{{1,7.522621162361411e-10},actuator}],

[]}

{actuator,{{1,7.522621162361411e-10},actuator},

xor_SendOutput,

{{origin,7.522621162363539e-10},cortex},

{private,xor_sim},

1

[{{0,7.522621162361355e-10},neuron}],

undefined}

{agent,test_clone,0,undefined,test,

{{origin,7.522621162358474e-10},cortex},

{[{0,1}],

[],

[{sensor,undefined,xor_GetInput,undefined,

{private,xor_sim},

2

[{{0,7.522621162361355e-10},neuron}],

undefined}],

[{actuator,undefined,xor_SendOutput,undefined,

{private,xor_sim},

1

[{{0,7.522621162361355e-10},neuron}],

undefined}]},

{constraint,xor_mimic,[tanh,cos,gauss,abs]},


[],undefined,0,

[{0,[{{0,7.522621162361355e-10},neuron}]}]}

...

{atomic,ok}

3> polis:stop().

ok

                • Polis: ##MATHEMA## is now offline, terminated with reason:normal

It works! Though for the sake of brevity, the above console printout does not show the whole genotype of the test_clone agent (shown in boldface), we can still see that there were no errors, and that this test function created, cloned, printed, and then deleted both of the agents.

At some point in the future we might wish to test mutation operators on simple NN system genotypes, for this reason our genotype module also includes the cre- ate_test/0 function . This function, unlike the test/0 function, creates a default agent genotype with an id: test , using the default constraint stored in the records.hrl file. Furthermore, the function first checks whether such a test agent already exists, and if it does, the function deletes the test agent, and creates a brand new one. This function will become very handy when testing and experimenting with mutation operators, since it will allow us to create a simple NN topology, apply mutation operators to it as a test of some functionality, then print the mutated topology for manual analysis, and then recreate the test agent and apply a different set of muta- tion operators if needed... Being able to create individual NNs to test mutation op- erators on, and being able to map the test genotype to a phenotype and test the functionality of the same, is essential when developing and advancing a complex system like this. We now test the create_test/0 function by starting the polis, exe- cuting the create_test/0 function twice, and then stopping the polis:

1> polis:start().

Parameters:{[],[]}

                • Polis: ##MATHEMA## is now online.

{ok,<0.34.0>}

2> genotype:create_test().

{agent,test,0,undefined,test,

{{origin,7.588472966671075e-10},cortex},

undefined,

{constraint,xor_mimic,[tanh,sin,abs]},

[],undefined,undefined,

[{0,[{{0,7.588472966664959e-10},neuron}]}]}

...

{atomic,{atomic,[ok]}}

3> genotype:create_test().

{agent,test,0,undefined,test,

{{origin,7.588472880658339e-10},cortex},

Chapter 8 Developing a Simple Neuroevolutionary Platform undefined,

{constraint,xor_mimic,[tanh,sin,abs]},

[],undefined,undefined,

[{0,[{{0,7.588472880658257e-10},neuron}]}]}

...

4> polis:stop().

                • Polis: ##MATHEMA## is now offline, terminated with reason:normal

ok

It works! The create_test/0 created a brand new test agent, and then when the create_test/0 was executed again, it deleted the old test agent, and created a new one. As before, for the sake of brevity, not the whole genotypes that were printed to console are shown, as they are very similar to the one shown in the test/0 func- tion earlier.

Having now developed a genotype module with all the necessary features to support a neuroevolutionary platform, we can move forward and begin working on the genotype mutator module.

8.5 Developing the genotype_mutator

We already have a mechanism used by the exoself to mutate/perturb the synap- tic weights during the NN tuning phase. But for a NN to grow, become more complex, and for a population to evolve, we need mutation operators that modify the topologies and architectures of the NN systems belonging to a population. The mutation operators should be able to add new neurons to the NN system, add new sensors, new actuators, and be able to modify other features of the NN based sys- tem. By having the ability to modify NN topologies, the evolutionary process can take hold and generate new species (topologically different organisms) within the population, and just as in the biological case, produce ever more complex NN based agents, more adept to their environment, and more fit with regards to the problem they are evolving to solve.

We need to have a set of mutation operators which are flexible enough so that any NN topology A can be turned into a NN topology B, by applying the available mutation operators to the NN system in some sequence. It is only then that our neuroevolutionary system will have the necessary tools and flexibility to evolve any type of NN based system given enough time and the appropriate fitness func- tion.

In this section we will concentrate on developing such a flexible and general set of mutation operators (MOs). The following set of MOs are required by evolution such that it has the ability to do both, complexify and/or prune one NN topology into any other by applying the below listed MOs in some order:


1. add_bias :

Choose a random neuron A , check if it has a bias in its weights list, if it does not, add the bias value. If the neuron already has a bias value, do nothing.

2. remove_bias :

Choose a random neuron A , check if it has a bias in its weights list, if it does, remove it. If the neuron does not have a bias value, do nothing.

3. mutate_weights :

Choose a random neuron A , and perturb each weight in its weight list with a probability of 1/sqrt(length(weights)) , with the perturbation intensity randomly chosen between -Pi/2 and Pi/2 .

4. reset_weights :

Choose a random neuron A , and reset all its synaptic weights to random values ranging between -Pi/2 and Pi/2 .

5. mutate_af :

Choose a random neuron A, and change its activation function to a new random activation function chosen from the af_list in the constraint record belonging to the NN.

6. add_inlink :

Choose a random neuron A , and an element B , and then add a connection from element B (possibly an existing sensor) to neuron A.

7. add_outlink :

Choose a random neuron A , and an element B , and then add a connection from neuron A to element B (possibly an existing actuator). The difference between this mutation operator and the add_inlink mutation operator, is that in one we choose a random neuron and then choose a random element from which we make a connection to the chosen neuron. While in the other we choose a ran- dom neuron, and then choose a random element to which the neuron makes a connection. The first (add_inlink) is capable of making links to sensors, while the second (add_outlink) is capable of potentially making links to actuators.

8. add_neuron :

Create a new neuron A, giving it a unique id and positioning it in a randomly selected layer of the NN. Then give the neuron A a randomly chosen activation function. Then choose a random neuron B in the NN and connect neuron A's inport to the neuron B's outport. Then choose a random neuron C in the NN and connect neuron A's outport to the neuron C's inport.

9. splice: There are 2 versions of this mutation operator, outsplice, and insplice:

outsplice : Create a new neuron A with a random activation function. Then choose a random neuron B in the NN. Randomly select neuron B's outport leading to some element C's (neuron or actuator) inport. Then disconnect neuron B from element C, and reconnect them through the newly created neuron A.

insplice : Create a new neuron A with a random activation function. Then choose a random neuron B in the NN. Randomly select neuron B's inport from some element C's (neuron or sensor) outport. Then disconnect neu-


ron B from element C, and reconnect them through the newly created neu- ron A. The reason for having an outsplice and an insplice, is that the outsplice can insert a new neuron between some random element and an actuator, while the insplice can insert a new neuron between an element and a sensor.

10. add_sensorlink :

Compared to the number of neurons, there are very few sensors, and so the probability of the add_inlink connecting a neuron to a sensor is very low. To increase the probability that the NN connects to a sensor, we can create the add_sensorlink mutation operator. This mutation operator first chooses a ran- dom existing sensor A, it then chooses a random neuron B to which A is not yet connected, and then connects A to B.

11. add_actuatorlink :

As in add_sensorlink, when compared to the number of neurons, there are very few actuators, and so the probability of the add_outlink connecting a neuron to an actuator is very low. Thus, we can implement the add_actuatorlink to in- crease the probability of connecting a neuron to an actuator. In this mutation operator, first a random actuator A is chosen which is connected to less neurons than its vl element dictates (an incompletely connected actuator). Then a ran- dom neuron B is chosen to which the actuator is not yet connected. Then A is connected from B.

12. remove_sensorlink :

First a random sensor A is chosen. From the sensor's fanout_ids list, a random neuron id is chosen, and then the sensor is disconnected from the corresponding neuron.

13. remove_actuatorlink :

First a random actuator A is chosen. From the actuator's fanin_ids list, a ran- dom neuron id is chosen, and then the actuator is disconnected from the corre- sponding neuron.

14. add_sensor :

Choose a random sensor from the sensor list belonging to the NN's morpholo- gy, but which is not yet used. Then connect the sensor to a random neuron A in the NN, thus adding a new sensory organ to the NN system.

15. add_actuator :

Choose a random actuator from the actuator list belonging to the NN's mor- phology, but which is not yet used. Then connect a random neuron A in the NN to this actuator, thus adding a new morphological feature to the NN that can be used to interact with the world.

16. remove_inlink :

Choose a random neuron A, and disconnect it from a randomly chosen element in its input_idps list.

17. remove_outlink :

Choose a random neuron A, and disconnect it from a randomly chosen element in its output_ids list.


18. remove_neuron :

Choose a random neuron A in the NN, and remove it from the topology. Then fix the presynaptic neuron B's and postsynaptic neuron C's outports and inports respectively to accommodate the removal of the connection with neuron A.

19. desplice: There are 2 versions of this operator, deoutspolice, and deinsplice:

deoutsplice : Choose a random neuron B in the NN, such that B's outport is connected to an element (neuron or actuator) C through some neuron A. Then delete neuron A and reconnect neuron B and element C directly. deintsplice : Choose a random neuron B in the NN, such that B's inport is connected to by an element (neuron or sensor) C through some neuron A. Then delete neuron A and connect neuron B and element C directly.

20. remove_sensor :

If a NN has more than one sensor, choose a random sensor belonging to the NN, and remove it by first disconnecting it from the neurons it is connected to, and then removing the tuple representing it from the genotype altogether.

21. remove_actuator :

If a NN has more than one actuator, choose a random actuator belonging the NN, and remove it by first disconnecting it from the neurons it is connected from, and then removing the tuple representing it from the genotype altogether.

Note that when choosing random neurons to connect to, we do not specify whether that neuron should be in the next layer, or whether that neuron should be in the previous layer. These mutations allow for both, feedforward, and recurrent connections to be formed.

Technically, we do not need every one of these mutation operators, the follow- ing list will be enough for a highly versatile complexifying topology and weight evolving artificial neural network (TWEANN) system: mutate_weights, add_bias, remove_bias, mutate_af, add_neuron, splice (just one of them), add_inlink, add_outlink, add_sensorlink, add_actuatorlink, add_sensor, and add_actuator. Note that this combination of MOs can convert any NN topology A into a NN to- pology B, given that A is contained (smaller, and simpler in a sense) within B. The add_inlink, add_outlink, add_sensorlink, add_actuatorlink mutation operators al- low for neurons to form new connections to neurons, sensors and actuators. The add_sensor and add_actuator, can add/integrate the new sensor and actuator pro- grams into the NN system. The add_neuron will add new neurons in parallel with other neurons in a layer, while outsplice will create new layers, increasing the depth of the NN system, and form new connections in series. The weight perturba- tions will be performed by the exoself, in a separate phase from the topological mutation phase, which will effectively make our system a memetic algorithm based TWEANN. On the other hand, if we also add mutate_weights operator to the mutation phase, and remove the exoself's weight tuning/perturbing ability in its separate phase, then our system will become a standard genetic algorithm based TWEANN.


In this section we will only create these 12 mutation operators. The deletory (except for the remove_bias, which does not delete or simplify a NN, but modifies the processing functionality of the neuron, by biasing and unbiasing its activation function) operators can be added later, because they share very similar logic to their complexifying mutator counterparts. Due to the exoself, we can easily switch between genetic and memetic TWEANN approaches by simply turning the exoself's tuning ability on or off. We can easily implement both, the genetic and the memetic approaches in our system. We will be able to turn the tuning on and off, and see the difference in the TWEANN's efficiency and robustness when us- ing the two different methods.

In the following subsections we will discuss each mutation operator, and de- velop that operator's source code. Once all the operators have been discussed, and their algorithms implemented, we will put it all together into a single geno- type_mutator module.

8.5.1 The Precursor Connection Making and Breaking Functions

Almost every mutation operator that is discussed next, relies on elements being connected and disconnected. For this purpose we create dedicated functions that can link and unlink any two elements. When connecting two elements, there are three possible connection types: a sensor-to-neuron, a neuron-to-neuron, and a neuron-to-actuator. The same for when we are disconnecting one element from another, we can disconnect: a sensor-from-neuron, a neuron-from-neuron, and a neuron-from-actuator.

When we establish a connection from element A to element B, we first use the ids of the connecting elements to deduce which type of connection is going to be made, and then dependent on that, establish the link between the two elements. If element A is a neuron and element B is a neuron, we perform the following set of steps to create a link from the presynaptic Neuron A to the postsynaptic Neuron B:

1. Read neuron A from the database.

2. Add neuron B's id to neuron A's output_ids list, if a connection is recursive, add neuron B's id to the ro_ids list as well.

3. Write the updated neuron A to database.

4. Read neuron B from the database.

5. Append to neuron B's input_idps list a new tuple with neuron A's id, and a new randomly generated weight: {NeuronA_Id,[Weight]}.

6. Write the updated neuron B to database.

If element A is a sensor, and the element B is a neuron, then to create a link from the presynaptic Sensor A to postsynaptic Neuron B, we perform the follow- ing set of steps:


1. Read sensor A from the database.

2. Add neuron B's id to sensor A's fanout_ids list.

3. Write the updated sensor A to database.

4. Read neuron B from the database.

5. Append to neuron B's input_idps list a new tuple with sensor A's id, and a weights list of length vl, where vl is the output vector length of sensor A.


Finally, if element A is a neuron, and element B is an actuator, then to create the link From Neuron A to Actuator B, we perform the following steps:

1. Read neuron A from database.

2. Add actuator B's id to neuron A's output_ids list.


4. Read actuator B from database.

5. If the number of neurons connected to the actuator is less than the actuator's vl, then add neuron A's id to actuator B's fanin_ids list. Otherwise exit with error to stop the mutation. This is done to prevent unnecessary connections, since if the actuator can only use a vl number of signals as parameters for executing its action function, there is no need to add any more connections to the actuator than that. The calling function can choose an actuator that still has space for connections, or even create a completely new actuator.

6. Write the updated actuator B to database.

The source code for the function that establishes the connection from element A to element B, is shown in the following listing.

Listing 8.4: The implementation of the link_FromElementToElement(Agent_Id, FromElement_Id, ToElement_Id) function.

link_FromElementToElement(Agent_Id,From_ElementId,To_ElementId)->

case {From_ElementId,To_ElementId} of

{{_FromSId,neuron},{_ToSId,neuron}} ->

link_FromNeuronToNeuron(Agent_Id,From_ElementId,To_ElementId);

{{_FromSId,sensor},{_ToSId,neuron}} ->

link_FromSensorToNeuron(Agent_Id,From_ElementId,To_ElementId);

{{_FromNId,neuron},{_ToAId,actuator}} ->

link_FromNeuronToActuator(Agent_Id,From_ElementId,To_ElementId)

end.

%The function link_FromElementToElement/3 first calculates what type of link is going to be established (neuron to neuron, sensor to neuron, or neuron to actuator), and then calls the spe- cific linking function based on that.

link_FromNeuronToNeuron(Agent_Id,From_NeuronId,To_NeuronId)->

A = genotype:read({agent,Agent_Id}),

Generation = A#agent.generation,


%From Part

FromN = genotype:read({neuron,From_NeuronId}),

U_FromN = link_FromNeuron(FromN,To_NeuronId,Generation),

genotype:write(U_FromN),

%To Part

ToN = genotype:read({neuron,To_NeuronId}), %We read it afterwards, in the case that it's the same Element. Thus we do not overwrite the earlier changes.

FromOVL = 1,

U_ToN = link_ToNeuron(From_NeuronId,FromOVL,ToN,Generation),

genotype:write(U_ToN).

%link_FromNeuronToNeuron/3 establishes a link from neuron with id = From_NeuronId, to a neuron with an id = To_NeuronId. The function then calls link_FromNeuron/4, which estab- lishes the link on the From_NeuronId's side. The updated neuron associated with the From_NeuronId is then written to database. To decide how long the weight list that is going to be added to the To_NeuronId's input_idps should be, the function calculates From_NeuronId's output vector length. Since the connection is from a neuron, FromOVL is set to 1. link_ToNeuron/4 is then called, and the link is established on the To_NeuronId's side. Finally, the updated neuron associated with the To_NeuronId is written to database. The order of read- ing the FromN and ToN neuron records from the database is important. It is essential that ToN is read after the U_FromN is written to database, in the case that From_NeuronId and To_NeuronId refer to the same neuron (a recurrent connection from the neuron to itself). If both neurons are read at the same time, for example before the links are established, then the link es- tablished in the U_FromN will be overwritten when the U_ToN is written to file. Thus, order is important in this function.

link_FromNeuron(FromN,ToId,Generation)->

{{FromLI,_},_} = FromN#neuron.id,

{{ToLI,_},_} = To_NeuronId,

FromOutput_Ids = FromN#neuron.output_ids,

FromRO_Ids = FromN#neuron.ro_ids,

case lists:member(ToId, FromOutput_Ids) of

true ->

exit( “******** ERROR:add_NeuronO[cannot add O_Id to Neuron]: ~p

already a member of ~p~n ”,[ToId,FromN#neuron.id]);

false ->

{U_FromOutput_Ids,U_FromRO_Ids} = case FromLI >= ToLI of

true ->

{[ToId|FromOutput_Ids],[ToId|FromRO_Ids]};

false ->

{[ToId|FromOutput_Ids],FromRO_Ids}

end,

FromN#neuron{

NPIds randomly from all the NPIds com- posing the NN, we now use NIds, because during the selection function, the way we compute the Spread value a neuron should use, is by analyzing that neuron's age, and its generation. Thus, once the list of selected neurons is composed, we use the IdsNPIds ets table which maps ids to pids and back, to convert the NIds to NPIds, and send each of the selected NPIds the noted message. Finally, the exoself then reactivates the cortex, and drops back into its main loop. The updated source

ompared, and the ToId is either added only to the FromN's output_ids list, or if the con- nection is recursive, when ToLayerIndex =< FromLayerIndex, it is added to output_ids and ro_ids lists. The FromN's generation is updated to the value Generation, which is the current, most recent generation, since this neuron has just been modified. Finally, the updated neuron record is returned to the caller. If ToId, which is the id of the element to which the connection is being established, is already a member of the FromN's output_ids list, then the function exits with error.

link_ToNeuron(FromId,FromOVL,ToN,Generation)->

ToInput_IdPs = ToN#neuron.input_idps,

case lists:keymember(FromId, 1, ToInput_IdPs) of

true ->

exit( “ERROR:add_NeuronI::[cannot add I_Id]: ~p already a member of

~p~n ”,[FromId,ToN#neuron.id]);

false ->

U_ToInput_IdPs = [{FromId, geno-

type:create_NeuralWeights(FromOVL,[])}|ToInput_IdPs],

ToN#neuron{

input_idps = U_ToInput_IdPs,

generation = Generation

}

end.

%link_ToNeuron/4 updates the record of ToN, so that it is prepared to receive a connection from the element FromId. The link emanates from element with the id FromId, whose output vector length is FromOVL, and the connection is made to the neuron ToN, the record which is updated in this function. The ToN's input_idps is updated with the tuple {FromId, [W_1... W_FromOVL]}, then the neuron's generation is updated to Generation (the current, most recent generation), and the updated ToN's record is returned to the caller. If FromId is already part of the ToN's input_idps list, which means that the link already exists between the neuron ToN, and element FromId, then the function exits with an error.

link_FromSensorToNeuron(Agent_Id,From_SensorId,To_NeuronId)->

A = genotype:read({agent,Agent_Id}),

Generation = A#agent.generation,

%From Part

FromS = genotype:read({sensor,From_SensorId}),

U_FromS = link_FromSensor(FromS,To_NeuronId),

genotype:write(U_FromS),

%To Part

Chapter 8 Developing a Simple Neuroevolutionary Platform ToN = genotype:read({neuron,To_NeuronId}),

FromOVL = FromS#sensor.vl,

U_ToN = link_ToNeuron(From_SensorId,FromOVL,ToN,Generation),

genotype:write(U_ToN).

%The function link_FromSensorToNeuron/3 establishes a connection from the sensor with id From_SensorId, to the neuron with id To_NeuronId. First the sensor record is updated with the connection details using the function link_FromSensor, and the updated sensor record is written to database. Then the record of the neuron to which the link is being established is updated us- ing the function link_ToNeuron/4, after which the updated neuron is written to database.

link_FromSensor(FromS,ToId)->

FromFanout_Ids = FromS#sensor.fanout_ids,

case lists:member(ToId, FromFanout_Ids) of

true ->

exit( “******** ERROR:link_FromSensor[cannot add ToId to Sensor]:

~p already a member of ~p~n ”,[ToId,FromS#sensor.id]);

false ->

FromS#sensor{fanout_ids = [ToId|FromFanout_Ids]}

end.

%The function link_FromSensor/2 updates the record of the sensor FromS, from whom the link emanates towards the element with id ToId. First the function ensures that there is no connec- tion that is already established between FromS and ToId, if a connection between these two el- ements already exists, then the function exits with error. If there is no connection between the two elements, then ToId is added to the sensor's fanout_ids list, and the updated record of the sensor is returned to the caller.

link_FromNeuronToActuator(Agent_Id,From_NeuronId,To_ActuatorId)->

A = genotype:read({agent,Agent_Id}),

Generation = A#agent.generation,

%From Part

FromN = genotype:read({neuron,From_NeuronId}),

U_FromN = link_FromNeuron(FromN,To_ActuatorId,Generation),

genotype:write(U_FromN),

%To Part

ToA = genotype:read({actuator,To_ActuatorId}),

Fanin_Ids = ToA#actuator.fanin_ids,

case length(Fanin_Ids) >= ToA#actuator.vl of

true ->

exit( “******** ERROR:link_FromNeuronToActuator:: Actuator already fully

connected ”);

false ->

U_Fanin_Ids = [From_NeuronId|Fanin_Ids],

genotype:write(ToA#actuator{fanin_ids = U_Fanin_Ids})

end.


%The function Link_FromNeuronToActuator/4 establishes a link emanating from the neuron with an id From_NeuronId, to an actuator with the id To_ActuatorId. First the From_NeuronId's record is updated using the function link_FromNeuron/3, after which the up- dated neuron record is written to database. Then the function checks whether the actuator to which the neuron is establishing the link, still has space for that link (length(Fanin_Ids) is less than the actuator's vector length, vl). If there is no more room, then the function exits with er- ror, if there is room, then the actuator's fanin_ids list is updated by appending to it the id of the neuron's id. Finally, then the updated actuator is written to database.

Though at this point our neuroevolutionary system will only perform mutations that add to the NN system's topology, the splice mutation operator does require a function that disconnects one element from another. For this reason, we also create the functions needed to cut the links between two elements. As before, there are three types of links that exist and can be cut: 1. From Element A and To Element B, where A & B are both neurons, 2. From Element A is a sensor, and To Element B is a neuron, and finally 3. From Element A is a neuron and To Element B is an Actuator. The following listing shows the implementation of the functions that cut the link between some “From Element A ” and “To Element B ”.

Listing 8.5: The cutlink_FromElementToElement(Agent_Id,FromElement_Id,ToElement_Id)

function.

cutlink_FromElementToElement(Agent_Id,From_ElementId,To_ElementId)->

case {From_ElementId,To_ElementId} of

{{_FromId,neuron},{_ToId,neuron}} ->

cutlink_FromNeuronToNeuron(Agent_Id,From_ElementId,To_ElementId);

{{_FromId,sensor},{_ToId,neuron}} ->

cutlink_FromSensorToNeuron(Agent_Id,From_ElementId,To_ElementId);

{{_FromId,neuron},{_ToId,actuator}} ->

cutlink_FromNeuronToActuator(Agent_Id,From_ElementId,To_ElementId)

end.

%cutlink_FromElementToElement/3 first checks which of the three types of connections exists between From_ElementId and To_ElementId (neuron to neuron, sensor to neuron, or neuron to actuator), and then disconnects the two elements using one of the three specialized cutlink_... functions.

cutlink_FromNeuronToNeuron(Agent_Id,From_NeuronId,To_NeuronId)->

A = genotype:read({agent,Agent_Id}),

Generation = A#agent.generation,

%From Part

FromN = genotype:read({neuron,From_NeuronId}),

U_FromN = cutlink_FromNeuron(FromN,To_NeuronId,Generation),

genotype:write(U_FromN),

%To Part

ToN = genotype:read({neuron,To_NeuronId}),


U_ToN = cutlink_ToNeuron(From_NeuronId,ToN,Generation),

genotype:write(U_ToN).

%The cutlink_FromNeuronToNeuron/3 function disconnects the connection from the From_NeuronId to the To_NeuronId. The function first disconnects the neuron associated with From_NeuronId by calling the cutlink_FromNeuron/3, and then writes to database the updated neuron record. The function then disconnects the neuron associated with the To_NeuronId from the connection using the cutlink_ToNeuron/3, and writes to database the updated ToN record. If the From_NeuronId and the To_NeuronId are ids of the same neuron, then it is important to first write U_FromN to database, before reading the ToN neuron from the database, so as not to lose the update made by the cutlink_FromNeuron/3, before reading the updated neuron from the database and calling the cutlink_ToNeuron. Thus, this order of reading and writing the neu- rons from the database is essential to cover the corner cases.

cutlink_FromNeuron(FromN,ToId,Generation)->

FromOutput_Ids = FromN#neuron.output_ids,

FromRO_Ids = FromN#neuron.ro_ids,

case lists:member(ToId, FromOutput_Ids) of

true ->

U_FromOutput_Ids = FromOutput_Ids--[ToId],

U_FromRO_Ids = FromRO_Ids--[ToId], %Does nothing if not recursive. FromN#neuron{

output_ids = U_FromOutput_Ids,

ro_ids = U_FromRO_Ids,

generation = Generation};

false ->

exit( “ERROR::cutlink_FromNeuron [cannot remove O_Id]: ~p not a

member of ~p~n ”,[ToId,FromN#neuron.id])

end.

%cutlink_FromNeuron/3 cuts the connection on the FromNeuron (FromN) side. The function first checks if the ToId is a member of the output_ids list. If it's not, then the function exits with an error. If the ToId is a member of the output_ids list, then the function removes the ToId from the FromOutput_Ids list and from the FromRO_Ids list. Even if the ToId is not a recursive con- nection, we still try to remove it from ro_ids list, in which case the result returns the original FromRO_Ids, and no change is made to it. Once the lists are updated, the updated neuron rec- ord of FromN is returned to the caller.

cutlink_ToNeuron(FromId,ToN,Generation)->

ToInput_IdPs = ToN#neuron.input_idps,

case lists:keymember(FromId, 1, ToInput_IdPs) of

true ->

U_ToInput_IdPs = lists:keydelete(FromId,1,ToInput_IdPs),

ToN#neuron{

input_idps = U_ToInput_IdPs,

generation = Generation};

false ->


exit( “ERROR[cannot remove I_Id]: ~p not a member of

~p~n ”,[FromId,ToN#neuron.id])

end.

%cutlink_ToNeuron/3 cuts the connection on the ToNeuron (ToN) side. The function first

checks if the FromId is a member of the ToN's input_idps list, if it's not, then the function exits with error. If FromId is a member, then that tuple is removed from the ToInput_IdPs list, and

the updated ToN record is returned to the caller.

cutlink_FromSensorToNeuron(Agent_Id,From_SensorId,To_NeuronId)->

A = genotype:read({agent,Agent_Id}),

Generation = A#agent.generation,

%From Part

FromS = genotype:read({sensor,From_SensorId}),

U_FromS = cutlink_FromSensor(FromS,To_NeuronId,Generation),

genotype:write(U_FromS),

%To Part

ToN = genotype:read({neuron,To_NeuronId}),

U_ToN = cutlink_ToNeuron(From_SensorId,ToN,Generation),

genotype:write(U_ToN).

%The cutlink_FromSensorToNeuron/3 cuts the connection from the From_SensorId to To_NeuronId. The function first cuts the connection on the From_SensorId side using the cutlink_FromSensor/3 function, and writes the updated sensor to database. The function then cuts the connection on the To_NeuronId side using the cutlink_ToNeuron/3 function, and writes the updated neuron record to database.

cutlink_FromSensor(FromS,ToId,Generation)->

FromFanout_Ids = FromS#sensor.fanout_ids,

case lists:member(ToId, FromFanout_Ids) of

true ->

U_FromFanout_Ids = FromFanout_Ids--[ToId],

FromS#sensor{

fanout_ids = U_FromFanout_Ids,

generation=Generation};

false ->

exit( “ERROR::cutlink_FromSensor [cannot remove ToId]: ~p not a

member of ~p~n ”,[ToId,FromS#sensor.id])

end.

%The cutlink_FromSensor/3 function first checks whether ToId is a member of the sensor's FromS fanout_ids list. If it is not, then the function exits with an error. If ToId is a member of FromS's fanout_ids list, then it is removed from the list, and the updated sensor record of FromS is returned to the caller.

cutlink_FromNeuronToActuator(Agent_Id,From_NeuronId,To_ActuatorId)->

A = genotype:read({agent,Agent_Id}),

Generation = A#agent.generation,


%From Part

FromN = genotype:read({neuron,From_NeuronId}),

U_FromN = cutlink_FromNeuron(FromN,To_ActuatorId,Generation),

genotype:write(U_FromN),

%To Part

ToA = genotype:read({actuator,To_ActuatorId}),

U_ToA = cutlink_ToActuator(From_NeuronId,ToA,Generation),

genotype:write(U_ToA).

%cutlink_FromNeuronToActuator/3 cuts the connection from the From_NeuronId to To_ActuatorId. The function first cuts the connection on the From_NeuronId side using the cutlink_FromNeuron/3 function, and writes the updated U_FromN to database. Then the con- nection on the To_ActuatorId is cut using the cutlink_ToActuator/3 function, after which the updated actuator record is written to database.

cutlink_ToActuator(FromId,ToA,Generation)->

ToFanin_Ids = ToA#actuator.fanin_ids,

case lists:member(FromId, ToFanin_Ids) of

true ->

U_ToFanin_Ids = ToFanin_Ids--[FromId],

ToA#actuator{

fanin_ids = U_ToFanin_Ids,

generation=Generation};

false ->

exit( “ERROR::cutlink_ToActuator [cannot remove FromId]: ~p not a

member of ~p~n ”,[FromId,ToA])

end.

%The cutlink_ToActuator/3 function cuts the connection on the ToActuator's side. The func- tion first checks if the FromId is a member of the actuator ToA's fanin_ids list. If it is not, the function exits with an error. If FromId is a member of the actuator's fanin_ids list, then the id is removed from the list, and the updated actuator record is returned to the caller.

8.5.2 mutate_weights

This is one of the simplest mutation operators. We first access the cx_id from the agent's record. After reading the cortex tuple from the database, we then choose a random id from the neuron_ids list, and using this id, read the neuron record from the database. Once we have the record, we access the neuron's in- put_idps list. Then calculate the total number of weights belonging to the neuron by adding the weight list lengths of each idp . We will have the probability of a weight in the input_idps list being mutated, set to 1/sqrt(tot_weights) . Once the mutation probability has been calculated, we go through every weight in the in- put_idps list, and mutate it with the probability of the calculated mutation proba- bility. Thus on average, a total of (1/sqrt(tot_weights))*tot_weights number of


weights in the list will be perturbed/mutated, sometimes less, and sometimes more. We will also set the weight perturbation intensity to be between -Pi and Pi . Once the weights have been perturbed, we write the updated neuron record back to mnesia with its updated (perturbed) input_idps. The code for this mutation opera- tor is shown in the following listing.

Listing – 8.6: The implementation of the mutate_weights mutation operator. mutate_weights(Agent_Id)->

A = genotype:read({agent,Agent_Id}),

Cx_Id = A#agent.cx_id,

Cx = genotype:read({cortex,Cx_Id}),

N_Ids = Cx#cortex.neuron_ids,

N_Id = lists:nth(random:uniform(length(N_Ids)),N_Ids),

N = genotype:read({neuron,N_Id}),

Input_IdPs = N#neuron.input_idps,

U_Input_IdPs = perturb_IdPs(Input_IdPs),

U_N = N#neuron{input_idps = U_Input_IdPs},

EvoHist = A#agent.evo_hist,

U_EvoHist = [{mutate_weights,N_Id}|EvoHist],

U_A = A#agent{evo_hist = U_EvoHist},

genotype:write(U_N),

genotype:write(U_A).

%The mutate_weights/1 function accepts the Agent_Id parameter, extracts the NN's cortex, and then chooses a random neuron belonging to the NN with a uniform distribution probability.

Then the neuron's input_idps list is extracted, and the function perturb_IdPs/1 is used to per- turb/mutate the weights. Once the Input_IdPs have been perturbed, the agent's evolutionary his- tory: EvoHist, is updated to include the successfully applied mutate_weights mutation operator. Then the updated Agent and the updated neuron are written back to database.

perturb_IdPs(Input_IdPs)->

Tot_Weights=lists:sum([length(Weights) || {_Input_Id,Weights}<-Input_IdPs]),

MP = 1/math:sqrt(Tot_Weights),

perturb_IdPs(MP,Input_IdPs,[]).

perturb_IdPs(MP,[{Input_Id,Weights}|Input_IdPs],Acc)->

U_Weights = perturb_weights(MP,Weights,[]),

perturb_IdPs(MP,Input_IdPs,[{Input_Id,U_Weights}|Acc]);

perturb_IdPs(_MP,[],Acc)->

lists:reverse(Acc).

%perturb_IdPs/1 accepts the Input_IdPs list of format: [{Id,Weights}...], calculates the total

number of weights in the Input_IdPs, and then calculates the mutation probability MP, using

the equation: 1/sqrt(Tot_Weights). Once the mutation probability is calculated, each weight in

the Input_IdPs list has a chance of MP to be perturbed/mutated. Once all the weights in the In- put_IdPs list had a chance of being perturbed, the updated Input_IdPs is returned to the caller.

Chapter 8 Developing a Simple Neuroevolutionary Platform perturb_weights(MP,[W|Weights],Acc)->

U_W = case random:uniform() < MP of

true->

sat((random:uniform()-0.5)*?DELTA_MULTIPLIER+W,-

?SAT_LIMIT,?SAT_LIMIT);

false ->

W

end,

perturb_weights(MP,Weights,[U_W|Acc]);

perturb_weights(_MP,[],Acc)->

lists:reverse(Acc).

%perturb_weights/3 is called with the mutation probability MP, a weights list, and an empty list [] to be used as an accumulator. The function goes through every weight, where every weight

has a chance of MP to be mutated/perturbed. The perturbations have a random intensity be-

tween -Pi and Pi. Once all the weights in the weights list had a chance of being perturbed, the updated weights list is reversed back to its original order, and returned back to the caller.

sat(Val,Min,Max)->

if

Val < Min -> Min;

Val > Max -> Max;

true -> Val

end.

%The sat/3 function calculates whether Val is between Min and Max. If it is, then Val is re-

turned as is. If Val is less than Min, then Min is returned. If Val is greater than Max, then Max

is returned.

8.5.3 add_bias & remove_bias

These mutation operators are applied to a randomly chosen neuron in a NN. First we select a random neuron id from neuron_ids list, then read the neuron record, and then search the input_idps to see if it already has a bias value. If input_idps has a bias, we exit the mutation with an error, and try another mutation. If the input_idps list does not yet use a bias, we append the bias tuple to the list's end. The remove_bias mutation operator is very similar to the add_bias, but uses the lists:keydelete/3 to remove the bias if one is present in the input_idps list of a randomly chosen neuron. The following listing shows the implementation for these two mutation operators.


Listing – 8.7: The implementation of the add_bias & remove_bias mutation operators. add_bias(Agent_Id)->

A = genotype:read({agent,Agent_Id}),

Cx_Id = A#agent.cx_id,

Cx = genotype:read({cortex,Cx_Id}),

N_Ids = Cx#cortex.neuron_ids,

N_Id = lists:nth(random:uniform(length(N_Ids)),N_Ids),

Generation = A#agent.generation,

N = genotype:read({neuron,N_Id}),

Input_IdPs = N#neuron.input_idps,

case lists:keymember(bias, 1, Input_IdPs) of

true ->

exit( “********ERROR:add_bias:: This Neuron already has a bias part. ”); false ->

U_Input_IdPs = lists:append(Input_IdPs,[{bias,[random:uniform()-0.5]}]),

U_N = N#neuron{

input_idps = U_Input_IdPs,

generation = Generation},

EvoHist = A#agent.evo_hist,

U_EvoHist = [{add_bias,N_Id}|EvoHist],

U_A = A#agent{evo_hist=U_EvoHist},

genotype:write(U_N),

genotype:write(U_A)

end.

%The add_bias/1 function is called with the Agent_Id parameter. The function first extracts the neuron_ids list from the cortex element and chooses a random neuron from the id list. The neu- ron is then read from the database and its input_idps list is checked for the bias element. If the neuron's input_idps list already has a bias tuple, then the function is exited. If the input_idps list does not have the bias tuple, then the bias is added and the agent's evolutionary history EvoHist is updated. Finally, the updated neuron and agent are written back to mnesia.

remove_bias(Agent_Id)->

A = genotype:read({agent,Agent_Id}),

Cx_Id = A#agent.cx_id,

Cx = genotype:read({cortex,Cx_Id}),

N_Ids = Cx#cortex.neuron_ids,

N_Id = lists:nth(random:uniform(length(N_Ids)),N_Ids),

Generation = A#agent.generation,

N = genotype:read({neuron,N_Id}),

Input_IdPs = N#neuron.input_idps,

case lists:keymember(bias, 1, Input_IdPs) of

false ->


exit( “********ERROR:remove_bias:: This Neuron does not have a bias

part. ”);

true ->

U_Input_IdPs = lists:keydelete(bias,1,Input_IdPs),

U_N = N#neuron{

input_idps = U_Input_IdPs,

generation = Generation},

EvoHist = A#agent.evo_hist,

U_EvoHist = [{remove_bias,N_Id}|EvoHist],

U_A = A#agent{evo_hist=U_EvoHist},

genotype:write(U_N),

genotype:write(U_A)

end.

%The remove_bias/1 function is called with the Agent_Id parameter. The function first extracts the neuron_ids list from the cortex element and chooses a random neuron id from it. The neuron is then read from the database and its input_idps list checked for a bias element. If the neuron's input_idps list has a bias tuple, it is removed and the agent's evolutionary history list is updated with the tuple {remove_bias,N_Id}, and the updated neuron and agent records are then written to database. If the input_idps list does not have the bias tuple, the function exits with an error stating so.

8.5.4 mutate_af

To execute this mutation operator, we first choose a random neuron A in the NN. This neuron keeps the tag/name of the activation function it uses in an atom form, stored in its record's af element. We retrieve this activation function tag, and then randomly choose a new activation function tag from the list of available acti- vation functions, which are specified within the specie's constraint tuple. To en- sure that the mutate_af chooses a new activation function, the currently used acti- vation function tag is first subtracted from the available activations list, and the new af tag is then chosen from the remaining tags. If the remaining activation function list is empty, then the neuron is assigned the standard tanh activation function. The implementation of the mutate_af function is shown in the following listing.

Listing 8.8: The implementation of the mutate_af mutation operator.

mutate_af(Agent_Id)->

A = genotype:read({agent,Agent_Id}),

Cx_Id = A#agent.cx_id,

Cx = genotype:read({cortex,Cx_Id}),


N_Ids = Cx#cortex.neuron_ids,

N_Id = lists:nth(random:uniform(length(N_Ids)),N_Ids),

Generation = A#agent.generation,

N = genotype:read({neuron,N_Id}),

AF = N#neuron.af,

Activation_Functions = (A#agent.constraint)#constraint.neural_afs -- [AF],

NewAF = genotype:generate_NeuronAF(Activation_Functions),

U_N = N#neuron{af=NewAF,generation=Generation},

EvoHist = A#agent.evo_hist,

U_EvoHist = [{mutate_af,Agent_Id}|EvoHist],

U_A = A#agent{evo_hist=U_EvoHist},

genotype:write(U_N),

genotype:write(U_A).

%The mutate_af/1 function chooses a random neuron, and then changes its currently used acti- vation function into another one available from the neural_afs list of the agent's constraint rec- ord.

8.5.5 add_outlink

To apply the add_outlink mutation operator to a NN, we first choose a random neuron A. This neuron keeps track of whom it sends its signals to using its out- put_ids list. We then extract the NN's neuron_ids and actuator_ids list from the cortex record, and then subtract the neuron's output_ids list from the neu- ron_ids++actuator_ids , which gives us a list of element (neuron and actuator) ids that this neuron is not yet connected to. If this list is empty, we exit the mutation and try another one. If the list is not empty, we randomly select the id from this list, and connect the neuron to this randomly selected element B . There are two types of elements we could have randomly chosen, a neuron element, and an actu- ator element. Let's look at each possible outlink (an outgoing link) connection in turn.

If this randomly chosen element B is a neuron, we perform the following steps to form a connection from neuron A to neuron B:

1. Modify neuron A: We first add B's id to the A's output_ids list, then we check if B's layer is equal to or less than A's layer, if so we add B's id to A's ro_ids list as well. We then set A's generation to that of the agent, the current genera- tion, and write the updated neuron record to database.

2. Modify neuron B: We first add a new input_idp of the form: {NeuronA_Id, Weights}, to the B's input_idps list, where Weights is a list composed of a sin- gle weight generated randomly between -Pi/2 and Pi/2. We then update B's generation, and write the updated neuron record to database.


On the other hand, if this randomly chosen element B is an actuator, we per- form the following steps to form a connection from neuron A to actuator B:

1. We first check if actuator B's length(fanin_ids) is lower than its vl. If it is, then this actuator can accept another connection. If it's not, then we exit the muta- tion operator and try another one. Let us assume that the actuator can still ac- cept new connections.

2. Modify neuron A: We add B's id to A's output_ids list. B is an actuator, so we do not need to check its layer, we know that it's the last one, with index 1 . We then update A's generation to that of the agent's, and write the updated neuron record to the database.

3. Modify actuator B: We add A's id to B's fanin_ids list. We then write the up- dated actuator to the database.

Having updated the elements of both, record A and B, the connection between them is formed. Having now performed a successful mutation operator, the agent's evo_hist list is updated. For this type of mutation, we form the tuple of the form: {MutationOperator,FromId,ToId}, which in this case is: {add_outlink, ElementA_Id,ElementB_Id}, and then append it the to evo_hist list. Then the up- dated agent record is stored to database. Finally, the add_outlink function returns control back to the caller. A few variations of how this mutation operator can modify a NN's topology is shown in Fig-8.5 .

Fig. 8.5 Applying add_outlink mutation operator to a NN system.


The source code for the add_outlink, is reliant on the link_FromElementTo Element/3 function, covered earlier. The implementation of the add_outlink func- tion, is shown in the following listing.

Listing-8.9: The implementation of the add_outlink mutation operator.

add_outlink(Agent_Id)->

A = genotype:read({agent,Agent_Id}),

Cx_Id = A#agent.cx_id,

Cx = genotype:read({cortex,Cx_Id}),

N_Ids = Cx#cortex.neuron_ids,

A_Ids = Cx#cortex.actuator_ids,

N_Id = lists:nth(random:uniform(length(N_Ids)),N_Ids),

N = genotype:read({neuron,N_Id}),

Output_Ids = N#neuron.output_ids,

case lists:append(A_Ids,N_Ids) -- Output_Ids of

[] ->

exit( “********ERROR:add_outlink:: Neuron already connected to all ids ”); Available_Ids ->

To_Id = lists:nth(random:uniform(length(Available_Ids)),Available_Ids),

link_FromElementToElement(Agent_Id,N_Id,To_Id),

EvoHist = A#agent.evo_hist,

U_EvoHist = [{add_outlink,N_Id,To_Id}|EvoHist],

U_A = A#agent{evo_hist=U_EvoHist},

genotype:write(U_A)

end.

%The add_outlink/1 function reads the cortex record from the database based on the cortex id extracted from the agent record. The function then selects a random neuron from the neuron_ids stored in the cortex record. The function then subtracts the neuron's output_ids from the com- bined list of the actuator and neuron ids belonging to the neural network to get a list of ids be- longing to the elements to which this neuron is not yet connected. If this list is empty, the func- tion exits with error. If the list is not empty, it is given the name Available_Ids, from which a random id is chosen, and the neuron is then connected to it. Finally, the agent's evo_hist list is updated, and the updated agent record is written to the database.

8.5.6 add_inlink

Similarly to the mutation operator: add_outlink, when performing the mutation add_inlink , we first choose a random neuron A from the agent's neuron_ids list. After reading the neuron record, we extract the ids from neuron A's input_idps list. Once the input ids are extracted, we subtract that list from the agent's neu- ron_ids++sensor_ids , to acquire a list of elements that neuron A has not yet been connected from. If this list is empty, we exit the mutation and try another one. If


the list is not empty, we randomly select the id of B from this list, and connect from it , to neuron A. There are two types of elements we could have randomly chosen, a neuron element, and a sensor element. Let's look at each possible inlink connection in turn.

If this randomly chosen element B is a neuron, we perform the same set of steps as we did in the add_outlink section, but with the two neuron ids reversed. Thus, we perform the following steps to form a connection from neuron B to neu- ron A:

1. Modify neuron B: We first add A's id to the B's output_ids list, then we check if A's layer is equal to or less than B's layer, if so we add A's id to B's ro_ids list as well. We then reset B's generation to 0, and write the updated neuron record to database.

2. Modify neuron A: We first add a new input_idp of the form: {NeuronB_Id, Weights}, to the A's input_idps list, where Weights is a list composed of a sin- gle weight generated randomly between -Pi/2 and Pi/2. We then reset A's gen- eration to that of the agent, and write the updated neuron record to database.

If this randomly chosen element B is a sensor, we perform the following steps to form a connection from sensor B to neuron A:

1. We first check if A's id is already in sensor B's fanout_ids list. If it is, then we exit the mutation operator and try another one. Let us assume that sensor B is not yet connected to neuron A.

2. Modify sensor B: We add A's id to B's fanout_ids list. We then write the up- dated sensor to the database.

3. Modify neuron A: We first add a new input_idp of the form: {SensorB_Id,Weights} to the A's input_idps list, where the length of the Weights list is dependent on the sensor B's vl (output vector length). The weights list is composed of values generated randomly to be between -Pi/2 and Pi/2 each. We then reset A's generation to that of the agent, and write the updated neuron record to database.

As in the add_outlink function, having now updated the elements of both, rec- ord A and B, the connection between them is formed. And having now performed a successful mutation operator, the agent's evo_hist list is updated. For this type of mutation, we form the tuple of the form: {MutationOperator, FromId, ToId}, which in this case is: {add_inlink, ElementB_Id, ElementA_Id}, and then append it to the evo_hist list. Finally, the updated agent record is stored to data- base and the add_inlink function returns control back to the caller. A few varia- tions of how this mutator can modify a NN's topology is shown in Fig-8.6.


Fig. 8.6 Applying add_inlink mutation operator to a NN system.

Listing-8.10: The implementation of the add_inlink mutation operator. add_inlink(Agent_Id)->

A = genotype:read({agent,Agent_Id}),

Cx_Id = A#agent.cx_id,

Cx = genotype:read({cortex,Cx_Id}),

N_Ids = Cx#cortex.neuron_ids,

S_Ids = Cx#cortex.sensor_ids,

N_Id = lists:nth(random:uniform(length(N_Ids)),N_Ids),

N = genotype:read({neuron,N_Id}),

{I_Ids,_WeightLists} = lists:unzip(N#neuron.input_idps),

case lists:append(S_Ids,N_Ids) -- I_Ids of

[] ->

exit( “********ERROR:add_INLink:: Neuron already connected from all

ids ”);

Available_Ids ->

From_Id = lists:nth(random:uniform(length(Available_Ids)),Available_Ids),

link_FromElementToElement(Agent_Id,From_Id,N_Id),

EvoHist = A#agent.evo_hist,

U_EvoHist = [{add_inlink,From_Id,N_Id}|EvoHist],

genotype:write(A#agent{evo_hist=U_EvoHist})

end.


%The add_inlink/1 function extracts the list of neuron ids within the NN, and chooses a random id from the list. We extract the ids from the input_Idps list, forming the “I_Ids ” list. We then subtract the I_Ids from the combined neuron and sensor ids belonging to the NN (neuron_ids and sensor_ids lists extracted from the cortex's record). The result is a list of presynaptic ele- ment ids from which the neuron is not yet connected. If this list is empty, the function exits with an error. Otherwise, the function chooses a random id from this list and establishes a con- nection between the neuron and this randomly selected presynaptic element. Finally, the agent's evo_hist list is updated, and the updated agent is written to database.

8.5.7 add_sensorlink

To apply the add_sensorlink mutation operator to the NN, first a random sensor id: S_Id, is chosen from the cortex's sensor_ids list. Then the sensor associated with S_Id is read from the database, and the sensor's fanout_ids is subtracted from the cortex's neuron_ids. The resulting list is that of neurons which are not yet connected from S_Id. A random neuron id: N_Id, is chosen from that list, and then a connection from S_Id to N_Id is established. The source code for this mutation operator is shown in the following listing.

Listing-8.11: The implementation of the add_sensorlink mutation operator. add_sensorlink(Agent_Id)->

A = genotype:read({agent,Agent_Id}),

Cx_Id = A#agent.cx_id,

Cx = genotype:read({cortex,Cx_Id}),

N_Ids = Cx#cortex.neuron_ids,

S_Ids = Cx#cortex.sensor_ids,

S_Id = lists:nth(random:uniform(length(S_Ids)),S_Ids),

S = genotype:read({sensor,S_Id}),

case N_Ids -- S#sensor.fanout_ids of

[] ->

exit( “********ERROR:add_sensorlink:: Sensor already connected to all

N_Ids ”);

Available_Ids ->

N_Id = lists:nth(random:uniform(length(Available_Ids)),Available_Ids),

link_FromElementToElement(Agent_Id,S_Id,N_Id),

EvoHist = A#agent.evo_hist,

U_EvoHist = [{add_sensorlink,S_Id,N_Id}|EvoHist],

genotype:write(A#agent{evo_hist=U_EvoHist})

end.


%The function add_sensorlink/1 randomly selects a sensor id: S_Id, from the cortex's sen- sor_ids list, and then establishes from that sensor a connection to a neuron still unlinked to this sensor, randomly selected from the cortex's neuron_ids list. To perform this, the function first selects a random sensor id S_Id from the cortex's sensor_ids list. Then a list of N_Ids to which S_Id is not yet connected is calculated by subtracting from the N_Ids the S_Id's fanout_ids list. If the resulting list is empty, then the function exits with an error since there are no other neu- rons to which the sensor can establish a new connection. If the list is not empty, then a random neuron id, N_Id, is selected from this list, and a connection is established from S_Id to N_Id. Finally, the agent's evo_hist is then updated and written to database.

A possible topological mutation scenario when applying the add_sensorlink mutation operator to a neural network, is shown in the following figure.

Fig. 8.7 Applying the add_sensorlink mutation operator to a neural network.

8.5.8 add_actuatorlink

To apply the add_actuatorlink mutation operator to a NN, first a random actua- tor id A_Id is chosen from the cortex's actuator_ids list. Then the actuator associ- ated with the A_Id is read from database. Then we check whether the length of A_Id's fanin_ids list is less than that of its vl . If it is, it would imply that it has not yet been fully connected, and that some of the parameters for controlling its action function are still using some default values within the actuator functions, and that the actuator should be connected from more neurons. If on the other hand the


length of the fanin_ids is equal to vl, then the actuator does not need to be con- nected from any more neurons, and the mutation operator function exits with er- ror. If the actuator can still be connected from new neurons, then its fanin_ids list is subtracted from the cortex's neuron_ids. The resulting list is that of the neurons which are not yet connected to A_Id. A random neuron id, N_Id, is selected from the list, and a connection is then established from N_Id to A_Id. The source code for this mutation operator is shown in the following listing.

Listing-8.12: The implementation of the add_actuatorlink mutation operator. add_actuatorlink(Agent_Id)->

Agent = genotype:read({agent,Agent_Id}),

Cx_Id = Agent#agent.cx_id,

Cx = genotype:read({cortex,Cx_Id}),

N_Ids = Cx#cortex.neuron_ids,

A_Ids = Cx#cortex.actuator_ids,

A_Id = lists:nth(random:uniform(length(A_Ids)),A_Ids),

A = genotype:read({actuator,A_Id}),

case N_Ids -- A#actuator.fanin_ids of

[] ->

exit( “********ERROR:add_actuatorlink:: Neuron already connected from all

ids ”);

Available_Ids ->

N_Id = lists:nth(random:uniform(length(Available_Ids)),Available_Ids),

link_FromElementToElement(Agent_Id,N_Id,A_Id),

EvoHist = Agent#agent.evo_hist,

U_EvoHist = [{add_actuatorlink,N_Id,A_Id}|EvoHist],

genotype:write(Agent#agent{evo_hist=U_EvoHist})

end.

%The add_actuatorlink/1 selects a random actuator id A_Id from the cortex's actuator_ids list, and then connects to A_Id a randomly selected neuron from which A_Id is not yet connected.

To accomplish this, the function first selects a random actuator id A_Id from the cortex's actua- tor_ids list. Then the function creates a list of neuron ids from which it is not yet connected,

done by subtracting the actuator's fanin_ids list from the cortex's neuron_ids list. If the result- ing id pool is empty, then the function exits with error. If the resulting id pool is not empty, a neuron id N_Id is randomly chosen from this id list, and the actuator is connected to this ran- domly chosen neuron. Finally, the agent's evo_hist is updated, and the updated agent is written to database.

A possible topological mutation scenario when applying the add_actuatorlink mutation operator to a neural network based system is shown in the following fig- ure.


Fig. 8.8 Applying the add_actuatorlink mutation operator to a neural network.

8.5.9 add_neuron

To apply the add_neuron mutation operator to a NN, we first read selected agent's pattern list, which specifies the general topological pattern of its NN. The topological pattern list has the following structure: [...{LayerIndex(n), LayerNeuron_Ids(n)}...] , where the LayerIndex variable specifies the index, and the LayerNeuron_Ids is a list of the ids that belong to this layer. Next, we random- ly (with uniform distribution) select a tuple from this list. This tuple specifies to which layer we will add the new neuron.

Having now decided on the neural layer, we then create a new neuron Id, with the layer index specified by the LayerIndex value. Next we construct a new neu- ron K using the construct_Neuron/6 function from the genotype module, with the following parameters: construct_Neuron(Cx_Id, Generation=CurrentGen, SpecCon, N_Id, Input_Specs=[], Output_Ids=[]). The N_Id is the one just created. Cx_Id is retrieved from the agent record, and the specie constraint (SpecCon) is acquired by first getting the Specie_Id from the agent record, reading the specie record, and then retrieving its constraint parameter.


This new neuron is created completely disconnected, with an empty input_idps, and output_ids lists. For the presynaptic element from which the new neuron K will be connected, we combine the neuron_ids and sensor_ids lists to form the FromId_Pool list, and then randomly choose an element A from it. We then ran- domly choose an id from the ToId_Pool , composed from the union of neuron_ids and actuator_ids, designating that element B, the one to which the new neuron K will connect. Finally, we use the functions add_outlink and add_inlink to connect A to K, and to connect K to B, respectively.

Having formed the connections, we then add N_Id to the agent's neuron_ids list, and update its evo_hist list by appending to it the tuple: {add_neuron, ElementA_Id, N_Id, ElementB_Id}. We then write the updated agent, element A, element B, and the newly created neuron's (K) record, to the database. A few var- iations of how this mutator can modify a NN's topology is shown in Fig-8.9 .

Fig. 8.9 Applying add_neuron mutation operator to a NN system.

Listing-8.13: The implementation of the add_neuron mutation operator. add_neuron(Agent_Id)->

A = genotype:read({agent,Agent_Id}),

Generation = A#agent.generation,

Pattern = A#agent.pattern,

Cx_Id = A#agent.cx_id,

Cx = genotype:read({cortex,Cx_Id}),

N_Ids = Cx#cortex.neuron_ids,

S_Ids = Cx#cortex.sensor_ids,


A_Ids = Cx#cortex.actuator_ids,

{TargetLayer,TargetNeuron_Ids} = lists:nth(random:uniform(length(Pattern)),Pattern),

NewN_Id = {{TargetLayer,genotype:generate_UniqueId()},neuron},

U_N_Ids = [NewN_Id|N_Ids],

U_Pattern = lists:keyreplace(TargetLayer, 1, Pattern, {TargetLayer, [NewN_Id| TargetNeuron_Ids]}),

SpecCon = A#agent.constraint,

genotype:construct_Neuron(Cx_Id,Generation,SpecCon,NewN_Id,[],[]),

FromId_Pool = N_Ids++S_Ids,

ToId_Pool = N_Ids ++ A_Ids,

From_ElementId = lists:nth(random:uniform(length(FromId_Pool)), FromId_Pool),

To_ElementId = lists:nth(random:uniform(length(ToId_Pool)),ToId_Pool),

link_FromElementToElement(Agent_Id,From_ElementId,NewN_Id),

link_FromElementToElement(Agent_Id,NewN_Id,To_ElementId),

U_EvoHist = [{add_neuron,From_ElementId,NewN_Id,To_ElementId}|A#agent.evo_hist],

genotype:write(Cx#cortex{neuron_ids = U_N_Ids}),

genotype:write(A#agent{pattern=U_Pattern,evo_hist=U_EvoHist}).

%The function add_neuron/1 creates a new neuron, and connects it to a randomly selected ele- ment in the NN, and from a randomly selected element in the NN. The function first reads the agent's pattern list, selects a random layer from the pattern, and then creates a new neuron id for that layer. Then a new unconnected neuron is created with that neuron id. The function then extracts the neuron_ids and the sensor_ids lists from the cortex. A random id, From_ElementId, is then chosen from the union of the sensor_ids and neuron_ids lists. Then a random id, To_ElementId, is chosen from the union of neuron_ids and actuator_ids (can be the same id as the From_ElementId). The function then establishes a connection from the neuron to To_ElemenId, and a connection to the neuron from From_ElementId. Finally, the cortex's neu- ron_ids list is updated with the id of the newly created neuron, the agent's evo_hist is updated, and finally, the updated cortex and agent records are written to database.

8.5.10 outsplice

The splice mutation operator increases the depth of the NN system through the side effect of adding a new neural layer when adding a new neuron to the NN. This mutation operator chooses a random neuron A in the NN, then chooses a ran- dom id in its output_ids list, which we designate as the id of element B. Finally, the mutation operator creates a new neuron K, disconnects A from B, and then re- connects them through K. If element B is in the layer directly after A's layer, then a new layer must be created, into which the new neuron K is inserted. It is through this that the depth of the NN is increased. However, to create neuron K, we first have to create K's id, and to do so we perform the following steps:


1. Retrieve the agent's pattern list (the NN's topology).

2. From pattern, extract the LayerIndex that is between A and B. If there is no layer separating A and B (for example, B's layer comes right after A's), then create a new layer whose layer index is LayerIndex_K = (LayerIdex_A + LayerIndex_B)/2 . If A is in the last neural layer, then the next layer is the one that belongs to the actuators, LayerIndex 1, and so a new layer can still be in- serted: LayerIndex_K = (LayerIdex_A + 1)/2 .

3. K's id is then: {{LayerIndex_K,Unique_Id},neuron} .

Once K's id is created and neuron K is constructed, we insert tuple {LayerIndex_K, [NeuronK_Id]} into the agent's Pattern list, unless the pattern al- ready has LayerIndex_K, in which case we add the NeuronK_Id to the list of ids belonging to the existing LayerIndex_K. We then update the agent's evo_hist list by appending to it the tuple: {splice, ElementA_Id, NeuronK_Id, ElementB_Id} , and then finally write the updated agent to file. Having updated the agent record, we update the A, B, and K elements.

The first step is to cut the connection from A to B using the cutlink_From ElementToElement/3 function, and then depending on whether B is another neu- ron or actuator, the following one of the two possible approaches is taken:

If element B is a neuron:

1. Delete B's id from A's output_ids list.

2. Use the function link_FromElementToElement/3 to create a connection from A to K.

3. Delete A's input_idp tuple from B's input_idps list.

4. Use the function link_FromElementToElement/3 to create a connection from K to B.

5. Reset A's, B's, and K's generation values to that of the most current generation (the one of agent's).

6. Write to database the updated neurons A, B, and K.

If element B is an actuator:


3. Delete A's id from B's fanin_ids list.


5. Reset the generation parameter for neuron A, and K.

6. Write to database the updated elements A, B, and K.

We made a good choice in isolating the linking and link cutting functionality within their own respective functions. We have used these two functions, link_FromElementToElement/3 and cutlink_FromElementToElement/3 , in almost


every mutation operator we've implemented thus far. Fig-8.10 demonstrates a few variations of how the splice mutator can modify a NN's topology.

Fig. 8.10 Applying the outsplice mutation operator to a NN system.

Listing-8.14: The implmenetnation of the outsplice mutation operator. outsplice(Agent_Id)->

A = genotype:read({agent,Agent_Id}),

Generation = A#agent.generation,

Pattern = A#agent.pattern,

Cx_Id = A#agent.cx_id,

Cx = genotype:read({cortex,Cx_Id}),

N_Ids = Cx#cortex.neuron_ids,

N_Id = lists:nth(random:uniform(length(N_Ids)),N_Ids),

N = genotype:read({neuron,N_Id}),

{{LayerIndex,_UId},neuron} = N_Id,

%Compose a feedforward Id pool, to create the splice from.

O_IdPool = case [{{TargetLayerIndex,TargetUId},TargetType} || {{TargetLayerIndex,TargetUId},TargetType} <- N#neuron.output_ids, TargetLayerIndex > LayerIndex] of

[] ->

exit( “********ERROR:outsplice:: O_IdPool== [] ”);

Ids ->

Ids

end,

%Choose a random neuron in the output_ids for splicing.


O_Id = lists:nth(random:uniform(length(O_IdPool)),O_IdPool),

{{OutputLayerIndex,_Output_UId},_OutputType} = O_Id,

%Create a new Layer, or select an existing one between N_Id and the O_Id, and create the new unlinked neuron.

NewLI = get_NewLI(LayerIndex,OutputLayerIndex,next,Pattern),

NewN_Id={{NewLI,genotype:generate_UniqueId()},neuron},

SpecCon = A#agent.constraint,

genotype:construct_Neuron(Cx_Id,Generation,SpecCon,NewN_Id,[],[]),

%Update pattern.

U_Pattern=case lists:keymember(NewLI,1,Pattern) of

true->

{NewLI,InLayerIds}=lists:keyfind(NewLI, 1, Pattern),

lists:keyreplace(NewLI, 1, Pattern, {NewLI,[NewN_Id|InLayerIds]});

false ->

lists:sort([{NewLI,[NewN_Id]}|Pattern])

end,

%Disconnect N_Id from the O_Id, and then reconnect them through NewN_Id cutlink_FromElementToElement(Agent_Id,N_Id,O_Id),

link_FromElementToElement(Agent_Id,N_Id,NewN_Id),

link_FromElementToElement(Agent_Id,NewN_Id,O_Id),

%Updated agent

EvoHist = A#agent.evo_hist,

U_EvoHist = [{outsplice,N_Id,NewN_Id,O_Id}|EvoHist],

U_Cx = Cx#cortex{neuron_ids = [NewN_Id|Cx#cortex.neuron_ids]}, genotype:write(U_Cx),

genotype:write(A#agent{pattern=U_Pattern,evo_hist=U_EvoHist}).

%The function outsplice/1 chooses a random neuron id from the cortex's neuron_ids list, dis- connects it from a randomly chosen id in its output_ids list, and then reconnects it to the same element through a newly created neuron. The function first chooses a random neuron N with the neuron id N_Id from the cortex's neuron_ids list. Then the neuron N's output_ids list is ex- tracted, and a new id list O_IdPool is created from the ids in the output_ids list which are locat- ed in the layers after the N_Id's layer (the ids of elements to whom the N_Id forms a feed for- ward connection). From this subset of the output_ids list, a random O_Id is chosen (if the sublist is empty, then the function exits with an error). First, N_Id is disconnected from the O_Id. The function then creates or extracts a new layer index, NewLI, located between N_Id and O_Id. If there exists a layer between N_Id and O_Id, NewLI is simply that layer. If on the other hand O_Id's layer comes immediately after N_Id's, then a new layer is created between O_Id and N_Id, whose layer index is in the middle of the two elements. A new unconnected neuron is then created in that layer, with a neuron id NewN_Id. The neuron NewN_Id is then connected to the O_Id, and from the N_Id, thus establishing a path from N_Id to O_Id through the NewN_Id. The cortex's neuron_ids is updated with the NewN_Id, and the agent's evo_hist list is updated with the new mutation operator tuple {outsplice,N_Id,Newn_Id,O_Id}. Finally, the updated cortex and agent are written to database.

get_NewLI(LI,LI,_Direction,_Pattern)->


exit( “******** ERROR: get_NewLI FromLI == ToLI ”); get_NewLI(FromLI,ToLI,Direction,Pattern)->

NewLI = case Direction of

next ->

get_NextLI(Pattern,FromLI,ToLI);

prev ->

get_PrevLI(lists:reverse(Pattern),FromLI,ToLI)

end,

NewLI.

%get_NewLI/4 calculates or creates a new layer index located between FromLI and ToLI. This function calls get_NextLI/3 or get_PrevLI/3, depending on whether the direction of the connec- tion is from sensors towards actuators (Direction = next), or from actuators towards sensors (Direction = prev), which is the case when executing an insplice/1 function, which calculates or creates a new layer between the N_Id and one of the ids in its input_idps list. If the FromLI == ToLI, the function exits with an error.

get_NextLI([{FromLI,_LastLayerNIds}],FromLI,ToLI)->

(FromLI+ToLI)/2;

get_NextLI([{LI,_LayerNIds}|Pattern],FromLI,ToLI)->

case LI == FromLI of

true ->

[{NextLI,_NextLayerNIds}|_] = Pattern,

case NextLI == ToLI of

true ->

(FromLI + ToLI)/2;

false ->

NextLI

end;

false ->

get_NextLI(Pattern,FromLI,ToLI)

end.

%get_NextLI checks whether the ToLI comes directly after FromLI, or whether there is another layer between them. If there is another layer between them, then that layer is returned, and the splice neuron is put into it. If there is no layer between FromLI and ToLI, then a new layer is created in the middle. Such a new layer index has the value of (FromLI+ToLI)/2.

get_PrevLI([{FromLI,_FirstLayerNIds}],FromLI,ToLI)->

(FromLI+ToLI)/2;

get_PrevLI([{LI,_LayerNIds}|Pattern],FromLI,ToLI)->

case LI == FromLI of

true ->

[{PrevLI,_PrevLayerNIds}|_] = Pattern,

case PrevLI == ToLI of

true ->

(FromLI + ToLI)/2;


false ->

PrevLI

end;

false ->

get_PrevLI(Pattern,FromLI,ToLI)

end.

%get_PrevLI checks whether the layer index ToLI, comes directly before FromLI, or whether there is another layer in between them. If there is another layer, then the function returns that layer, if no such layer is found, the function creates a new layer index with value: (FromLI+ToLI)/2.

8.5.11 add_sensor

The add_sensor mutation operator (MO) modifies the agent's architecture, its morphology in a sense, by adding new sensory “organs ” to the NN based system. If the NN system was started with just a camera sensor, but its specie morphology provides for a larger list of sensory organs, then it is through the add_sensor muta- tion operator that the NN system can also acquire pressure and radiation sensors (if available for that agent's morphology), for example. By acquiring new sensory organs slowly, through evolution, the NN based system has the chance to evolve connections to only the most useful sensors in the environment the agent inhabits, and work best with the agent's morphology and NN topology.

An agent can have multiple sensors of the same type, as long as they differ in at least some specification. For example, some sensors also specify parameters, which can vary between sensors of the same name. Assume that the agent controls a robot, there can be numerous sensors available to the robot that the NN based system can make use of to improve its performance within the world. But the same type of sensors, let's say camera sensor, can also be installed at different lo- cations on the same real, or simulated robot. This installation location can be spec- ified within the parameters, thus there can be a large list of the same type of sen- sors which simply differ in their coordinate parameters.

To apply this mutation to the NN based system, first the agent's morphology name is retrieved. Using the agent's morphology name we then access the mor- phology module to get the list of all the available sensors for that morphology. The function morphology:get_sensors/1 returns the list of all such available sen- sors. From this list we subtract the list of sensors that the agent is already using, and thus create a Sensor_Pool . Finally, we select a random sensor from this list, let us call that sensor, A.

Afterwards, Sensor A's id is created and then added to the cortex's sensor_ids list. Then a random neuron B is chosen from the cortex's neuron_ids list, and us- ing the function link_FromElementToElement, the connection from A to B is


established. Agent's evo_hist list is updated by adding to it a new tuple: {add_sensor,SensorA_Id,NeuronB_Id} . Then the updated agent, cortex, sensor, and neuron tuples are written to the database. Fig-8.11 demonstrates a few varia- tions of how this MO can modify a NN's topology.

Fig. 8.11 Applying the add_sensor mutation operator to a NN based system.

Listing-8.15: The implementation of the add_sensor mutation operator. add_sensor(Agent_Id)->

Agent = genotype:read({agent,Agent_Id}),

Cx_Id = Agent#agent.cx_id,

Cx = genotype:read({cortex,Cx_Id}),

S_Ids = Cx#cortex.sensor_ids,

SpeCon = Agent#agent.constraint,

Morphology = SpeCon#constraint.morphology,

case morphology:get_Sensors(Morphology)--[(genotype:read({sensor, S_Id}))#sensor{ id=undefined, cx_id=undefined,fanout_ids=[]} || S_Id<-S_Ids] of

[] ->

exit( “********ERROR:add_sensor(Agent_Id):: NN system is already using

all available sensors ”);

Available_Sensors ->io:format( “Available_Sensors ”),

NewS_Id = {{-1,genotype:generate_UniqueId()},sensor},

NewSensor = (lists:nth(random:uniform(length(Available_Sensors)),

Available_Sensors))#sensor{id=NewS_Id,cx_id=Cx_Id},

genotype:write(NewSensor),

N_Ids = Cx#cortex.neuron_ids,


N_Id = lists:nth(random:uniform(length(N_Ids)),N_Ids),

link_FromElementToElement(Agent_Id,NewS_Id,N_Id),

EvoHist = Agent#agent.evo_hist,

U_EvoHist = [{add_sensor,NewS_Id,N_Id}|EvoHist],

U_Cx = Cx#cortex{sensor_ids=[NewS_Id|S_Ids]},

genotype:write(U_Cx),

genotype:write(Agent#agent{evo_hist=U_EvoHist})

end.

%The add_sensor/1 function adds and connects a new sensor to the neural network, a sensor from which the NN is not yet connected. After retrieving the morphology name from the con- straint record retrieved from the agent, the complete set of available sensors is retrieved using the morphology:get_Sensors/1 function. From this complete sensor list we subtract the sensor tuples used by the NN based system. But before we can do so, we first revert the sensor id and cx_id of each used sensor, back to undefined, since that is what their initial state within the sen- sor tuples is. With the NN's sensor ids and cx_ids reverted back to undefined, they can be sub- tracted from the complete set of the sensors available to the given morphology. If the resulting list is empty, then the function exits with an error. On the other hand, if the resulting list is not empty, then there are still sensors which the NN is not yet using (though it does not mean that using the new sensors would make the NN better, these sensors might be simply useless, and hence not previously incorporated during evolution). From this resulting list we then select a random sensor, and create for it a unique sensor id: NewS_Id. A random neuron id: N_Id, is then selected from the cortex's neuron_ids list, and a connection is established from NewS_Id to N_Id. The cortex's sensor_ids is updated with the new sensor's id, and the agent's evo_hist is updated with the new tuple. Finally, the updated cortex and agent records are then written to database.

8.5.12 add_actuator

Similarly to the add_sensor mutation operator, the add_actuator MO modifies the agent's architecture, by adding to it new morphological element which it can then use to interact with the world. Just as with any other new addition to the NN's topology, or architecture (when adding sensors and actuators), some NN based systems will not integrate well with the newly added element, while others will. Some will not get an advantage in the environment, while others will. Those that do successfully integrate a new element into their architecture, and those that gain benefit from that new element, will have an advantage over those that do not have such an element integrated.

The add_actuator mutator first accesses the agent's morphology name and re- trieves the list of all currently available actuators through the execution of the morphology:get_Actuators/1 function. An Actuator_Pool is then formed by sub- tracting from this actuator list the list of actuators already used by the NN based agent. A random actuator A is then chosen from this Actuator_Pool. From the cor-


tex's neuron_ids list, a random id of a neuron B is retrieved. Then, using the link_FromElementToElement/3 function, a connection from B to A is established. Finally, A's id is added to the cortex's actuator_ids list, agent's evo_hist is updat- ed by appending to it the tuple: {add_actuator, NeuronB_Id, ActuatorA_Id} , and the updated neuron, actuator, agent, and cortex records are written to database. Fig-8.12 demonstrates a few variations of how this mutation operator can modify a NN's topology.

Fig. 8.12 Applying the add_actuator mutation operator to a NN system.

Listing-8.14: The implementation of the add_actuator mutation operator. add_actuator(Agent_Id)->

Agent = genotype:read({agent,Agent_Id}),

Cx_Id = Agent#agent.cx_id,

Cx = genotype:read({cortex,Cx_Id}),

A_Ids = Cx#cortex.actuator_ids, %TODO: Should we fill in all the fanin_ids locations, or

just 1? and let evolution fill the rest? We can go either way, or compare the performance of one implementation against the other. In this implementation, we take the second approach.

SpeCon = Agent#agent.constraint,

Morphology = SpeCon#constraint.morphology,

case morphology:get_Actuators(Morphology)--[(genotype:read({actuator,

A_Id}))#actuator{cx_id =undefined, id=undefined,fanin_ids=[]} || A_Id<-A_Ids] of

[] ->

exit( “********ERROR:add_actuator(Agent_Id):: NN system is already using

all available actuators ”);

Chapter 8 Developing a Simple Neuroevolutionary Platform Available_Actuators ->

NewA_Id = {{1,genotype:generate_UniqueId()},actuator},

NewActuator=(lists:nth(random:uniform(length(Available_Actuators)),

Available_Actuators))#actuator{id=NewA_Id,cx_id=Cx_Id},

genotype:write(NewActuator),

N_Ids = Cx#cortex.neuron_ids,

N_Id = lists:nth(random:uniform(length(N_Ids)),N_Ids),

link_FromElementToElement(Agent_Id,N_Id,NewA_Id),

EvoHist = Agent#agent.evo_hist,

U_EvoHist = [{add_actuator,N_Id,NewA_Id}|EvoHist],

U_Cx = Cx#cortex{actuator_ids=[NewA_Id|A_Ids]},

genotype:write(U_Cx),

genotype:write(Agent#agent{evo_hist=U_EvoHist})

end.

%The add_actuator/1 function adds and connects a new actuator to the neural network, an actu- ator type to which the NN is not yet connected to. After we extract the morphology name from

the agent's constraint record, we execute the function: morphology:get_Actuators(Morphology) to get the list of actuators available to the NN system. From that list the function then removes

the actuators the NN is already connected to, after the ids and cx_ids of those actuators are set

to undefined. The resulting list is the list of actuators to which the NN is not yet connected. A random actuator is chosen from that list, and then a random neuron id N_Id from cortex's neu- ron_ids is chosen, and connected to the new actuator. Finally, the cortex's actuator_ids list is updated with the id of the newly created actuator, the agent's evo_hist is updated with the new tuple, and then both, the updated cortex and the agent are written to database.

8.5.13 Planning the Remaining Few Details of the Genotype Mutator Module

Having now discussed the functionality of every mutation operator, we can start creating the actual mutator module. There are three remaining issues that are in need of a resolution:

1. When creating a mutant clone from some fit genotype, how many mutation op- erators in sequence should be applied to this clone to produce the final mutant? And what should that number depend on?

I have came to the conclusion that the total number of Mutation Operators (MOs) that should be applied to a clone of a NN to create a mutant offspring, is a value that should in some manner depend on the size of the parent NN system. To see why, consider the following: Imagine you have a NN system composed of a single neuron. We now apply a mutation to this NN system, any mutation operator applied will result in this very simple single neuron NN to function very different- ly. Now assume that we have a NN system composed of one million neurons,


interconnected to a great extent. If we apply a single mutation operator to this NN system, then the effect on its functionality will not be as drastic. So then, when compared to smaller NNs, the larger NN systems require a larger number of muta- tions to function differently.

But, sometimes drastic changes are not needed, sometimes we only wish to tune the NN system, just add a neuron here, or a single connection there... We do not know ahead of time how many mutation operators need to be applied to a NN system to produce a functional offspring that will be able to jump out of a local optima, or reach a higher fitness level. Thus, not only should the number of muta- tion operators applied be proportional to the size of the NN system, but it should also be chosen randomly. Thus, offspring should be created by applying a random number of mutation operators, within the range of 1 and some maximum number which is dependent on the NN's size. This will result in a system that creates off- spring by at times applying a large number of mutation operators when creating an offspring, and at times a small number of mutation operators. The larger the NN, the greater the spread from which the number of MOs to be applied is chosen.

In our system, the number of mutation operators to be applied should be a ran- dom number, chosen with uniform distribution, between 1 and sqrt(Tot_Neurons), where Tot_Neurons is the variable containing the total number of neurons within the NN system. Thus, by increasing the range of the possible number of MOs ap- plied to a cloned NN in proportion to the size of the parent NN, allows us to make the mutation intensity significant enough to allow the mutant offspring to continue producing innovations in its behavior when compared to its parent, and thus ex- plore the fitness landscape far and wide. At the same time, some offspring will on- ly acquire a few MOs and differ topologically only slightly, and thus have a chance to tune and explore the local topological areas on the topological fitness landscape. Let us give this approach a name, let us call it a “Random Intensity Mutation ”, (RIM) . We will call this method: Random Intensity Mutation, because the intensity, which in this case is the range of the number of the MOs applied, is randomly chosen. In the same way that we chose randomly the number of weight perturbations to be applied to the NN by the exoself, which too can be considered to have been an application of RIM.

I think, and we will benchmark this later to test the theory, that RIM provides an advantage over those neuroevolutionary systems that only apply a static num- ber of mutation operators when generating offspring. Also, note that the system we are now creating does not use recombination (the crossover of two or more genotypes to create an offspring), and so the use of RIM is an approach to getting just as much of genetical change as would the use of recombination give. But RIM allows us to do this in a controlled manner. Plus, when we are creating a new gen- otype by combining two different genotypes in some manner, there is very little chance that the two parts of two different genotypes whose phenotypes process in- formation in completely different manner, or completely the same (when those genotypes are very closely or completely related), will yield a fit or an improved


agent, respectively. The use of RIM allows us to slowly complexify the NN sys- tems, to build on top of fit genotypes. Using the RIM approach, sometimes we fine tune the genotype, to see if the genotype can be made more effective. While at other times, the genotypical space is explored far and wide, as we try to free the NN system from some local optima that it is stuck in.

                • Note********

Sometimes, progress can only be made when multiple mutation operators are applied in se- quence, in one go and before fitness evaluation, to create a new offspring. Evolution is such that a fitness function, especially one that considers that the more concise genotypes are fitter, does not allow for certain combination of mutations to take place over multiple generations. For ex- ample consider the scenario shown in Fig-8.13 . Assume that in this scenario, an agent inhabits a flatland world, it has a camera sensor, a differential drive actuator [1], and a single neuron neu- ral network, where the neuron uses a tanh activation function. In this flatland, the organism would greatly benefit if it also had a radiation sensor that connects to a neuron that uses sin ac- tivation function, which itself is connected to neuron which uses tanh activation function. While at the same time, this world might not be a good place for agents that either have NN topologies composed of a camera sensor connected to two neurons in layer 0, where one neuron uses a tanh and the other uses a sin activation function. This world might also not be favorable to NN systems that have two sensors, a camera and a radiation sensor, both connected to a neuron us- ing a tanh activation function, which then connects to the differential drive actuator. If our neuroevolutionary system only creates offspring by applying a single mutation operator to a parent clone, then we would only generate NNs that are unfavorable in the flatland. These off- spring would not get a chance to live on to the next generation and create their own offspring, generated through the mutation resulting in a NN composed of the two neuron two sensor based topology. It is only when our neuroevolutionary system is able to apply at least two mutation operators in series, in a single go, that we have a chance of generating this favorable NN based mutant system. It is this mutant offspring composed of two neurons and two sensors, that has a chance to jump out of the local optima.


Fig. 8.13 Climbing out of a local optima using RIM.

2. When adding a new actuator to a NN, and this new actuator has vl = X , should we randomly connect to it X number of neurons, or just one?

When adding a new actuator, the simplest and best approach would be to con- nect to it only a single neuron in the NN. We can let the mutation operators like add_outlink, add_actuatorlink, and add_neuron, be the ones to establish the re- maining connections between the NN and the new actuator, over multiple genera- tions as the agent evolves. We can create the actuator functions such that they use default values for all elements in their input vector that are not yet fed from real neurons. After all, there might be some actuators whose vl is larger than the total number of neurons in the NN at the time of being added. Yet it would still be a good idea to connect it to the NN, letting evolution slowly link the right neurons (the ones that lead to a fitter NN system) to the actuator, over multiple genera- tions. Eventually, the NN would integrate such an actuator into itself fully, con- necting to it vl number of neurons.

Having decided on these two remaining details of the mutator module, we can now implement it.

8.5.14 Implementing the genotype_mutator Module

We now construct the genotype_mutator module using everything we've dis- cussed in this section. The mutate function accepts the parameter Agent_Id, and


mutates that agent's NN by applying X number of successful mutation operators in series, where X depends on the size of the NN. At this point, the specie the agent belongs to will be the same that its parent belongs to. Thus, the species mak- ing up the population will be based on the constraint specification. The constraints are specified when creating the population, thus if one creates 3 different con- straints, then the population will be composed of 3 different species. The agents belonging to a particular specie will only have access to the features specified within the constraint of that specie.

Though we wished to keep all but the genotype module blind to what genotype storage method is used, we need to use the mnesia's transaction function to benefit from the atomic transactions offered by it. We execute the mutation operators within a transaction , and in this manner ensure that if any part of the MO fails, any modifications made to the topology within that failed MO, are reverted. Thus, we will let the genotype_mutator module know that mnesia is being used, and let it use the transactions to take full advantage of the mnesia database. The following listing shows the source code for the entire genotype_mutator module, with the al- ready presented mutation operator functions truncated with “... ”.

Listing-8.17: The implementation of the genotype_mutator module.

-module(genome_mutator).

-compile(export_all).

-include( “records.hrl ”).

-define(DELTA_MULTIPLIER,math:pi()*2).

-define(SAT_LIMIT,math:pi()*2).

-define(MUTATORS,[

mutate_weights,

add_bias,

remove_bias,

add_outlink,

% remove_outLink,

add_inlink,

% remove_inlink,

add_sensorlink,

add_actuatorlink,

add_neuron,

% remove_neuron,

outsplice,

% insplice,

add_sensor,

% remove_sensor,

add_actuator

% remove_actuator

]).


-define(ACTUATORS,morphology:get_Actuators(A#agent.morphology)).

-define(SENSORS,morphology:get_Sensors(A#agent.morphology)).

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

test()->

Result = genotype:mutate(test),

case Result of

{atomic,_} ->

io:format( “******** Mutation Succesful.~n ”);

_->

io:format( “******** Mutation Failure:~p~n ”,[Result])

end.

%The test/1 function simply tests the mutate/1 function using a random mutation operator, on

an agent whose id is ‘test'.

test(Agent_Id,Mutator)->

F = fun()->

genome_mutator:Mutator(Agent_Id)

end,

mnesia:transaction(F).

%test/2 function tests the mutation operator “Mutator ” on the agent with an id Agent_Id. mutate(Agent_Id)->

random:seed(now()),

F = fun()->

A = genotype:read({agent,Agent_Id}),

OldGeneration = A#agent.generation,

NewGeneration = OldGeneration+1,

genotype:write(A#agent{generation = NewGeneration}),

apply_Mutators(Agent_Id)

end,

mnesia:transaction(F).

%The mutate/1 function applies a random available mutation operator to an agent with an id: Agent_Id.

apply_Mutators(Agent_Id)->

A = genotype:read({agent,Agent_Id}),

Cx = genotype:read({cortex,A#agent.cx_id}),

TotNeurons = length(Cx#cortex.neuron_ids),

TotMutations = random:uniform(round(math:pow(TotNeurons,1/2))),

io:format( “Tot neurons:~p Performing Tot mutations:~p

on:~p~n ”,[TotNeurons,TotMutations,Agent_Id]),

apply_Mutators(Agent_Id,TotMutations).

%apply_Mutators/1 chooses a random number X between 1 and math:sqrt(Tot_Neurons)),

where Tot_Neurons is the total number of neurons in the neural network, and then applies X number of randomly chosen mutation operators to the NN. The function first calculates the


length of the neuron_ids list, from which it then calculates the TotMutations value by choosing the random number between 1 and sqrt of the length of neuron_ids list. Having now chosen how many mutation operators to apply to the NN based system, the function apply_Mutators/1 calls apply_Mutators/2.

apply_Mutators(_Agent_Id,0)->

done;

apply_Mutators(Agent_Id,MutationIndex)->

Result = apply_NeuralMutator(Agent_Id),

case Result of

{atomic,_} ->

apply_Mutators(Agent_Id,MutationIndex-1);

Error ->

io:format( “******** Error:~p~nRetrying with new Muta-

tion...~n ”,[Error]),

apply_Mutators(Agent_Id,MutationIndex)

end.

%apply_Mutators/2 applies the set number of successful mutation operators to the Agent. If a mutation operator exits with an error, the function tries another mutation operator. It is only af- ter a successful mutation operator is applied that the MutationIndex is decremented.

apply_NeuralMutator(Agent_Id)->

F = fun()->

Mutators = ?MUTATORS,

Mutator = lists:nth(random:uniform(length(Mutators)),Mutators),

io:format( “Mutation Operator:~p~n ”,[Mutator]),

genome_mutator:Mutator(Agent_Id)

end,

mnesia:transaction(F).

%apply_NeuralMutator/1 applies the actual mutation operator to the NN. Because the genotype is stored in mnesia, if the mutation operator function exits with an error, the database made changes are retracted, and a new mutation operator can then be applied to the agent, as if the previous unsuccessful mutation operator was never applied. The mutation operator to be ap- plied to the agent is chosen randomly from the mutation operator list: ?MUTATORS.

mutate_weights(Agent_Id)->

...

add_bias(Agent_Id)->

remove_bias(Agent_Id)->

mutate_af(Agent_Id)->


link_FromElementToElement(Agent_Id,From_ElementId,To_ElementId)->

...

cutlink_FromElementToElement(Agent_Id,From_ElementId,To_ElementId)->

...

add_outlink(Agent_Id)->

...

add_inlink(Agent_Id)->

...

add_sensorlink(Agent_Id)->

...

add_actuatorlink(Agent_Id)->

...

add_neuron(Agent_Id)->

...

outsplice(Agent_Id)->

insplice(Agent_Id)->

add_sensor(Agent_Id)->

...

add_actuator(Agent_Id)->

...

Having now created the polis, genotype, and the mutator module, we have all the modules and functions necessary to perform a few simple mutation tests using the genome_mutator's test/2 function. The test/2 function will allow us to use the genotype module to create a test agent, apply specific mutation operators to it, and then check whether they executed correctly by analyzing the genotype printed to screen using the print/1 function. Since we know that a seed NN with a single neu- ron is created without a threshold element, we will first test the simplest mutator, add_threshold , as shown below. For the sake of brevity, only the neuron record is shown when print/1 is executed.


1> polis:start().

Parameters:{[],[]}

                • Polis: ##MATHEMA## is now online.

{ok,<0.34.0>}

2> genotype:create_test().

...

{neuron,{{0,7.588454795555494e-10},neuron},

0

{{origin,7.588454795561199e-10},cortex},

abs,

[{{{-1,7.588454795555557e-10},sensor},

[0.03387578757696197,-0.35293313204412424]}],

[{{1,7.588454795555509e-10},actuator}],

[]}

...

{atomic,{atomic,[ok]}}

3> genome_mutator:test(test,add_bias).

{atomic,ok}

4> genotype:print(test).

...

{neuron,{{0,7.588454795555494e-10},neuron},

0

{{origin,7.588454795561199e-10},cortex},

abs,

[{{{-1,7.588454795555557e-10},sensor},

[0.03387578757696197,-0.35293313204412424]},

{bias,[0.28866429973097785]} ],

[{{1,7.588454795555509e-10},actuator}],

[]}

...

{atomic,[ok]}

It works! If we compare the neuron record before and after the add_bias muta- tion operator was applied, we can see that a bias tuple was added to the neuron's input_idps (shown in bold face in the above console printout). Let's test a muta- tion operator that is a bit more complex, let's test the outsplice MO. Before we ex- ecute the function, let us think of what should happen, and then test that theory. We know that the default agent is created with a single sensor, a neuron in layer 0, and a single actuator. An outsplice adds a new neuron between the neuron and one of the ids in its output_ids list. Since this test agent's neuron is only connected to the actuator, only the actuator's id should be in the neurons output_ids list. Thus, the outsplice function should add a new neuron in its own new layer: 0.5 = (0+1)/2 . Let us now test our prediction, and again for the sake of brevity, I will not show the agent and cortex records, because this mutation operator only affects the NN topology, the sensors, neurons, and actuators:


1> polis:start().

Parameters:{[],[]}

                • Polis: ##MATHEMA## is now online.

{ok,<0.34.0>}

2> genotype:create_test().

...

{sensor,{{-1,7.588451156728372e-10},sensor},

xor_GetInput,

{{origin,7.588451156734038e-10},cortex},

{private,xor_sim},

2

[{{0,7.588451156728286e-10},neuron}],

undefined}

{neuron,{{0,7.588451156728286e-10},neuron},

0

{{origin,7.588451156734038e-10},cortex},

abs,

[{{{-1,7.588451156728372e-10},sensor},

[-0.38419731227432274,0.2612339422607457]}],

[{{1,7.588451156728309e-10},actuator}],

[]}

{actuator,{{1,7.588451156728309e-10},actuator},

xor_SendOutput,

{{origin,7.588451156734038e-10},cortex},

{private,xor_sim},

1

[{{0,7.588451156728286e-10},neuron}],

undefined}

{atomic,{atomic,[ok]}}

3> genome_mutator:test(test,outsplice).

{atomic,ok}


...

{sensor,{{-1,7.588451156728372e-10},sensor},

xor_GetInput,

{{origin,7.588451156734038e-10},cortex},

{private,xor_sim},

2

[{{0,7.588451156728286e-10},neuron}],

undefined}

{neuron,{{0.5,7.58845109269042e-10},neuron},

0

{{origin,7.588451156734038e-10},cortex},

sin,

[{{{0,7.588451156728286e-10},neuron},[0.3623116220700018]}],

Chapter 8 Developing a Simple Neuroevolutionary Platform [{{1,7.588451156728309e-10},actuator}],

[]}

{neuron,{{0,7.588451156728286e-10},neuron},

0

{{origin,7.588451156734038e-10},cortex},

abs,

[{{{-1,7.588451156728372e-10},sensor},

[-0.38419731227432274,0.2612339422607457]}],

[{{0.5,7.58845109269042e-10},neuron}],

[]}

{actuator,{{1,7.588451156728309e-10},actuator},

xor_SendOutput,

{{origin,7.588451156734038e-10},cortex},

{private,xor_sim},

1

[{{0.5,7.58845109269042e-10},neuron}],

0}

{atomic,[ok]}

It works! No errors. And from the output to console we can compare the geno- types of the NN systems before and after the mutation. As we predicted, when the genotype printed at 4> is compared to one printed at 2>, we see a new neuron with id: {{0.5,7.58845109269042e-10},neuron} added to the system, shown in bold- face in the above console printout. The new neuron is inserted into its own layer with index 0.5, connected from the first neuron, and to the actuator. The NN pat- tern, which specifies the number of layers composing the NN, and the layer densi- ty, was also changed. The initial, and the final patterns, stored in the agent record, are as follows:

Initial topology: [{0,[{{0,7.588451156728286e-10},neuron}]}]

Final topology: [{0,[{{0,7.588451156728286e-10},neuron}]},

{0.5,[{{0.5,7.58845109269042e-10},neuron}]}]

In this way we can continue going through each mutation operator, applying it to a test agent, and then confirming manually that the MO works as designed. Catching errors in mutation operators can be difficult, and in general catching small errors in evolutionary systems is even more so because if the error does not crash the program and only slightly affects the system, the evolved organisms will evolve around such problems. If for example the add_inlink operator constantly produces damaged links, the agents will still adapt and solve the problem they are applied to, by using the remaining MOs to compensate. It would require a manual analysis of the topology and functionality of the NN based system to finally real- ize that there is some minor problem with a mutation operator.


A long while back when experimenting with different ways a neuron can pro- cess incoming data, I was testing a theory which required me to filter all neuron incoming signals through a tanh function, before calculating a dot product of those signals with their corresponding synaptic weights, and then sent through the acti- vation function. By mistake, I forgot to remove that modification after the tests. I continued applying the system to real world problems, and the system continued to work. It was only a few months later, when I was going over the neuron module implementation, that I realized I left it in. The system continued performing well, evolution routed around this small problem. But the point is that errors can go un- noticed because the system will continue functioning relatively well, even with a few flaws or bugs. This is even more so when it comes in developing such systems in Erlang, because it is even more difficult to crash an Erlang based system.

We have now covered all the genotype related issues of our neuroevolutionary system. In the next section we discuss and develop the new population_monitor module.

8.5.15 Developing the population_monitor

The population_monitor module is the only remaining large module we need to create before we have a functional topology and weight evolving artificial neural network system. The remaining modules (exoself, cortex, & neuron), need merely be slightly updated. The population_monitor process will have to perform a num- ber of complex functions, keeping track of an entire population of agents, select- ing fit, removing the unfit, creating offspring from the fit agents through cloning and mutation, and then finally reapplying the new generation of agents to the problem or simulation set by the researcher.

After the polis is started and the public scapes are spawned, we can generate a seed population from which a solution, or simply an ever more intelligent popula- tion of agents, will evolve. Towards what the neuroevolution is applied depends and is specified by the scape, which itself is a simulation of some physical or mathematical space. The goal of neuroevolution is to generate and guide the popu- lation towards higher fitness. Where, how, and when, the fitness points are distrib- uted to the neural networks, and under what conditions, is determined by the scape itself. Once all agents have finished interacting with the scapes, or scape, we are left with a population of agents, each of whom has a fitness score. How and when the agent finishes, or dies, or terminates, is once again dependent on the scape and the problem. The agent “finishes ” when for example it has gone through every el- ement in the database, or dies of “old age ” when the agent controls an avatar with- in some simulated environment... At this point, the population_monitor uses a se- lection function to select a certain percentage of the population as fit or valid, while marking the rest as unfit, or invalid. We will not implement crossover ap- proaches for offspring generation, since the topological RIM system will create


enough variation, and do so much faster and safer than any crossover algorithm can.

The offspring are created through cloning and mutation. Not all fit agents are equal, some are more equal than others, some have a higher fitness level. Though all the fit agents will survive to the next generation, the number of offspring each agent creates will depend on that agent's fitness. The population_monitor will de- cide how many offspring to allocate to each agent. The offspring will be created by first cloning the fit agent, and then by mutating the clone to produce a varia- tion, a mutant, of it. The clone, with its own unique agent id, is assigned to the same specie that its parent belongs to. Once all the offspring are created, where “all ” means the same number as was deleted during the selection process, the new generation of agents is then released back into the scape, or applied again to the problem. Then, the evolutionary cycle repeats.

The way we specify what to apply the neuroevolutionary system and the par- ticular population to, is through the constraint record. The constraint records specify the morphologies into which the population will be subdivided, where each morphology has its own sensors and actuators from which the agent will draw its sensors and actuators from during evolution. The scapes define the prob- lem, and the sensors and actuators specify what scapes the agent can interact with, and how it can interface with them. The constraints also specify activation func- tions available to the evolving adaptive agents, and thus we can further specify and try out the same morphologies but with different activation function sets. Thus, the seed population of agents that the population_monitor tracks will, from the very start, be subdivided into multiple species, where the number of species depends on the number of constraint records in the list dropped as a parameter into the population initialization function.

Based on this narrative, we can see that the population_monitor program has to do quite a few things, let us break it down into steps, and then compose the algo- rithm for each step:

1. Create a Seed_Species list, composed of constraint records as follows: Seed_Species=[#constraint{morphology=A,neural_afs=[...]},#constraint{

morphology=B}...]

2. Specify seed population size, and max population size. The seed population size specifies the number of seed agents to create, which will compose the seed population. The max population size specifies the maximum number of agents that the population can sustain.

3. Divide the total number of X seed agents by length(Seed_Species) , letting the resulting number Y specify the number each seed specie will start with. Thus, each Seed_Specie will have Y number of agents, and the seed population will have X = Y*length(Seed_Species) number of agents in total.

4. Spawn the agents belonging to the population by starting up their exoselfs. Each exoself is started with the Agent_Id, and PopulationMonitor_PId as


parameters. The exoself of the agent bootstraps itself, converting its genotype into its phenotype.

5. Each agent then interacts with the scape, until it dies, or finishes its session with the scape, at which point the agent's exoself notifies the popula- tion_monitor of its fitness score, and that it is done. The agent then terminates itself by terminating all the processes that it is composed of.

6. The population_monitor waits for the fitness score from every agent in the population it is monitoring. Once all the agents have finished and terminated, the population_monitor runs a selection algorithm on the agent list, to seperate the fit from the unfit.

7. The population_monitor composes a list of tuples: [{TrueFitness, Agent_Id}...], where: TrueFitness = Fitness/math:pow(TotN,?EFF,) and where TotN is the total number of neurons, and ?EFF is a researcher specified efficiency index, usually set to 0.1 . What this does is make the agent's fitness dependent on the agent's NN size. For example, if two agents have the same fitness, but one is composed of 2 neurons, and the other of 20, then we should choose the 2 neu- ron based agent because it is that much more efficient. How much weight is given to the NN size is specified through the ?EFF parameter, the Efficiency parameter.

8. At this point, we apply a selection algorithm, an algorithm that removes unfit agents, and allows the fit ones to produce offspring. I have developed a selec- tion algorithm which I dubbed competition , and which has yielded excellent re- sults in the past. The steps of this algorithm are as follows, and which I will ex- plain in more detail afterwards:

1. Calculate the average energy cost of a Neuron in the population using the following steps:

TotEnergy = Agent(1)_Fitness + Agent(2)_Fitness...

TotNeurons = Agent(1)_TotNeurons + Agent(2)_TotNeurons... NeuronEnergyCost = TotEnergy/TotNeurons

2. Sort the Agents in the population based on TrueFitness.

3. Remove the bottom 50% of the population.

4. Calculate the number of allotted offspring for each Agent(i): AllotedNeurons = (AgentFitness/NeuronEnergyCost),

AllotedOffsprings(i) = round(AllotedNeurons(i)/Agent(i)_TotNeurons) 5. Calculate total number of offspring being produced for the next generation:

TotalNewOffsprings = AllotedOffsprings(1)+...AllotedOffsprings(n).

6. Calculate PopulationNormalizer, to keep the population within a certain limit:

PopulationNormalizer = TotalNewOffsprings/PopulationLimit

7. Calculate the normalized number of offspring allotted to each NN based system (Normalized Allotted Offsprin = NAO):

NAO(i)= round(AllotedOffsprings(i)/PopulationNormalizer(i))

8. If NAO == 1, then the agent is allowed to survive to the next generation without offspring, if NAO > 1, then the agent is allowed to produce (NAO

9.


-1) number of mutated copies of itself, if NAO = 0, then the agent is re- moved from the population and deleted.

Then the Topological Mutation Phase is initiated, and the mutator program passes through the database creating the appropriate NAO number of mu- tant clones of the surviving fit agents.

9. Go to 4, until stopping condition is reached, where the following stopping con- ditions are available:

The best fitness of the population is not increased some X number of times, where X is set by the researcher.

The goal fitness level is reached by one of the agents in the population. The preset maximum number of generations has passed.

The preset maximum number of evaluations has passed.

From the fitness score modification (making it dependent on the NN size as well) and the competition selection algorithm, it can be seen that it becomes very difficult for bloated NNs to survive when smaller systems produce better or simi- lar results. Yet when a large NN produces significantly better results justifying its complexity and size, it can begin to compete and push out the smaller NN sys- tems. This selection algorithm takes into account that a NN composed of 2 Neu- rons is double the size of a 1 Neuron NN, and thus should also have an increased fitness if it wants to produce just as many offspring. On the other hand, a NN of size 101 is only slightly larger than a NN of size 100, and thus should pay only slightly more per offspring. This selection algorithm has proven excellent when it comes to keeping neural network bloating to a minimum. At the same time, this does not mean that the system will be too greedy, after all, we allow for the top 50% to survive, as long as they show some level of competitiveness. Because of the application of RIM during the mutation phase, we can expect that if there is a path towards greater complexity and fitness, eventually one of the mutant off- spring will find it.

Having now discussed all the steps and features the population_monitor needs to make and have respectively, we can begin developing the source code. The fol- lowing listing shows the population_monitor module.

Listing-8.18: The implementation of the population_monitor module. -module(population_monitor).

-include( “records.hrl ”).

%% API

-export([start_link/1,start_link/0,start/1,start/0,stop/0,init/2]).

%% gen_server callbacks

-export([init/1, handle_call/3, handle_cast/2, handle_info/2, terminate/2, code_change/3, cre- ate_MutantAgentCopy/1,test/0, create_specie/3, continue/2, continue/3,init_population/1, ex- tract_AgentIds/2,delete_population/1]).

-behaviour(gen_server).


%%%%%%%%%%% Population Monitor Options & Parameters -define(SELECTION_ALGORITHM,competition).

-define(EFF,0.05).

-define(INIT_CONSTRAINTS,[#constraint{morphology=Morphology,neural_afs

=Neural_AFs} || Morphology<-[xor_mimic],Neural_AFs<-tanh]).

-define(SURVIVAL_PERCENTAGE,0.5).

-define(SPECIE_SIZE_LIMIT,10).

-define(INIT_SPECIE_SIZE,10).

-define(INIT_POPULATION_ID,test).

-define(OP_MODE,gt).

-define(INIT_POLIS,mathema).

-define(GENERATION_LIMIT,100).

-define(EVALUATIONS_LIMIT,100000).

-define(DIVERSITY_COUNT_STEP,500).

-define(GEN_UID,genotype:generate_UniqueId()).

-define(CHAMPION_COUNT_STEP,500).

-define(FITNESS_GOAL,inf).

-record(state,{op_mode,population_id,activeAgent_IdPs=[],agent_ids=[],tot_agents,agents_left,

op_tag,agent_summaries=[],pop_gen=0,eval_acc=0,cycle_acc=0,time_acc=0,step_size,

next_step,goal_status,selection_algorithm}).

%%=============================================================API

%%--------------------------------------------------------------------

%% Function: start_link() -> {ok,Pid} | ignore | {error,Error}

%% Description: Starts the server

%%--------------------------------------------------------------------

start_link(Start_Parameters) ->

gen_server:start_link(?MODULE, Start_Parameters, []).

start(Start_Parameters) ->

gen_server:start(?MODULE, Start_Parameters, []).

start_link() ->

gen_server:start_link(?MODULE, [], []).

start() ->

gen_server:start(?MODULE, [], []).

stop() ->

gen_server:cast(monitor,{stop,normal}).

init(Pid,InitState)->

gen_server:cast(Pid,{init,InitState}).

%%=================================================gen_server callbacks

%%--------------------------------------------------------------------


%% Function: init(Args) -> {ok, State} |

%% {ok, State, Timeout} |

%% ignore |

%% {stop, Reason}

%% Description: Initiates the server

%%--------------------------------------------------------------------

init(Parameters) ->

process_flag(trap_exit,true),

register(monitor,self()),

io:format( “******** Population monitor started with parameters:~p~n ”,[Parameters]), State = case Parameters of

{OpMode,Population_Id,Selection_Algorithm}->

Agent_Ids = extract_AgentIds(Population_Id,all),

ActiveAgent_IdPs = summon_agents(OpMode,Agent_Ids),

  1. state{op_mode=OpMode,

population_id = Population_Id,

activeAgent_IdPs = ActiveAgent_IdPs,

tot_agents = length(Agent_Ids),

agents_left = length(Agent_Ids),

op_tag = continue,

selection_algorithm = Selection_Algorithm}

end,

{ok, State}.

%In init/1 the population_monitor process registers itself with the node under the name moni- tor, and sets all the needed parameters within its #state record. The function first extracts all the Agent_Ids that belong to the population using the extract_AgentIds/2 function. Each agent is then spawned/activated, and converted from genotype to phenotype in the summon_agents/2 function. The summon_agents/2 function summons the agents and returns to the caller a list of tuples with the following format: [{Agent_Id,Agent_PId}...]. Finally, once the state record's parameters have been set, the function drops into the main gen_server loop.

%%--------------------------------------------------------------------

%% Function: %% handle_call(Request, From, State) -> {reply, Reply, State} |

%%

%%

%%

%%

%%

{reply, Reply, State, Timeout} | {noreply, State} |

{noreply, State, Timeout} | {stop, Reason, Reply, State} | {stop, Reason, State}

%% Description: Handling call messages %%--------------------------------------------------------------------

handle_call({stop,normal},_From, S)->

ActiveAgent_IdPs = S#state.activeAgent_IdPs,

[Agent_PId ! {self(),terminate} || {_DAgent_Id,Agent_PId}<-ActiveAgent_IdPs], {stop, normal, S};

handle_call({stop,shutdown},_From,State)->


{stop, shutdown, State}.

%If the population_monitor process receives a {stop,normal} call, it checks if there are any agents that are still active. If there are any, it terminates them, and then itself terminates.

%%--------------------------------------------------------------------

%% Function: handle_cast(Msg, State) -> {noreply, State} |

%%

%%

{noreply, State, Timeout} |

{stop, Reason, State}

%% Description: Handling cast messages %%--------------------------------------------------------------------

handle_cast({Agent_Id,terminated,Fitness,AgentEvalAcc,AgentCycleAcc,AgentTimeAcc},S) when S#state.selection_algorithm == competition ->

Population_Id = S#state.population_id,

OpTag = S#state.op_tag,

AgentsLeft = S#state.agents_left,

OpMode = S#state.op_mode,

U_EvalAcc = S#state.eval_acc+AgentEvalAcc,

U_CycleAcc = S#state.cycle_acc+AgentCycleAcc,

U_TimeAcc = S#state.time_acc+AgentTimeAcc,

case (AgentsLeft-1) =< 0 of

true ->

mutate_population(Population_Id,?SPECIE_SIZE_LIMIT,

S#state.selection_algorithm),

U_PopGen = S#state.pop_gen+1,

io:format( “Population Generation:~p Ended.~n~n~n ”,[U_PopGen]),

case OpTag of

continue ->

Specie_Ids = (genotype:dirty_read({population,

Population_Id}))#population.specie_ids,

SpecFitList=[(genotype:dirty_read({specie,

Specie_Id}))#specie.fitness || Specie_Id <- Specie_Ids],

BestFitness=lists:nth(1,lists:reverse(lists:sort([MaxFitness ||

{_,_,MaxFitness,_} <- SpecFitList]))),

case (U_PopGen >= ?GENERATION_LIMIT) or

(S#state.eval_acc >= ?EVALUATIONS_LIMIT) or (BestFitness > ?FITNESS_GOAL) of

true -> %termination condition reached

Agent_Ids = extract_AgentIds(Population_Id,all),

TotAgents=length(Agent_Ids),

U_S=S#state{agent_ids=Agent_Ids, tot_agents

=TotAgents,agents_left=TotAgents,pop_gen=U_PopGen,eval_acc=U_EvalAcc, cycle_acc

=U_CycleAcc,time_acc=U_TimeAcc},

{stop,normal,U_S};

false -> %in progress

Agent_Ids = extract_AgentIds(Population_Id,all),


U_ActiveAgent_IdPs =summon_agents(OpMode,

Agent_Ids),

TotAgents=length(Agent_Ids),

U_S=S#state{activeAgent_IdPs

=U_ActiveAgent_IdPs, tot_agents=TotAgents,agents_left=TotAgents, pop_gen=U_PopGen,

eval_acc=U_EvalAcc,cycle_acc=U_CycleAcc, time_acc=U_TimeAcc},

{noreply,U_S}

end;

done ->

io:format( “Shutting down Population Monitor~n ”),

U_S = S#state{agents_left = 0,pop_gen=U_PopGen, eval_acc

=U_EvalAcc, cycle_acc=U_CycleAcc,time_acc=U_TimeAcc},

{stop,normal,U_S};

pause ->

io:format( “Population Monitor has paused.~n ”),

U_S = S#state{agents_left=0,pop_gen=U_PopGen, eval_acc

=U_EvalAcc, cycle_acc=U_CycleAcc,time_acc=U_TimeAcc},

{noreply,U_S}

end;

false ->

ActiveAgent_IdPs = S#state.activeAgent_IdPs,

U_ActiveAgent_Ids = lists:keydelete(Agent_Id,1,ActiveAgent_IdPs),

U_S = S#state{activeAgent_IdPs = U_ActiveAgent_Ids,agents_left =

AgentsLeft-1,eval_acc=U_EvalAcc,cycle_acc=U_CycleAcc,time_acc=U_TimeAcc},

{noreply,U_S}

end;

%This clause accepts the cast signals sent by the agents which terminate after finishing with their evaluations. The clause specializes in the “competition ” selection algorithm, which is a generational selection algorithm. As a generational selection algorithm, it waits until the entire population has finished being evaluated, and only then selects the fit from the unfit, and com- poses an updated population for the next generation. The OpTag can be set from the outside to shutdown the population_monitor by setting it to done. Once a stopping condition is reached, either through a generation limit, an evaluations limit, or fitness goal, the population_monitor exits normally. If the stopping condition is not reached, the population_monitor spawns the new generation of agents, and waits again for all the agents in the population to complete their eval- uations. If the OpTag is set to pause, it does not generate a new population, and instead goes in- to a waiting mode, during which it waits to be either restarted or terminated.

handle_cast({op_tag,pause},S) when S#state.op_tag == continue ->

U_S = S#state{op_tag = pause},

{noreply,U_S};

%The population_monitor process can accept a pause command cast. When it receives it, it goes into pause mode after all the agents have completed with their evaluations. The process can only go into a pause mode if it is currently in the continue mode (its op_tag is set to contin- ue).


handle_cast({op_tag,continue},S) when S#state.op_tag == pause ->

Population_Id = S#state.population_id,

OpMode = S#state.op_mode,

Agent_Ids = extract_AgentIds(Population_Id,all),

U_ActiveAgent_IdPs=summon_agents(OpMode,Agent_Ids),

TotAgents=length(Agent_Ids),

U_S=S#state{activeAgent_IdPs=U_ActiveAgent_IdPs,tot_agents=TotAgents,agents_left

=TotAgents,op_tag=continue},

{noreply,U_S};

%The population_monitor process can accept a continue command if its current op_tag is set to pause. When it receives a continue command, it summons all the agents in the population, and continues with its neuroevolution synchronization duties.

handle_cast({init,InitState},_State)->

{noreply,InitState};

handle_cast({stop,normal},State)->

{stop, normal,State};

handle_cast({stop,shutdown},State)->

{stop, shutdown, State}.

%%--------------------------------------------------------------------

%% Function: handle_info(Info, State) -> {noreply, State} |

%% {noreply, State, Timeout} |

%% {stop, Reason, State}

%% Description: Handling all non call/cast messages

%%--------------------------------------------------------------------

handle_info(_Info, State) ->

{noreply, State}.

terminate(Reason, S) ->

case S of

[] ->

io:format( “******** Population_Monitor shut down with Reason:~p, with

State: []~n ”,[Reason]);

_ ->

Population_Id = S#state.population_id,

OpTag = S#state.op_tag,

OpMode = S#state.op_mode,

io:format( “******** Population_Monitor:~p shut down with Reason:~p

OpTag:~p, while in OpMode:~p~n ”,[Population_Id,Reason,OpTag,OpMode]),

io:format( “******** Tot Agents:~p Population Generation:~p Eval_Acc:~p

Cycle_Acc:~p Time_Acc:~p~n ”,[S#state.tot_agents,S#state.pop_gen,S#state.eval_acc,

S#state.cycle_acc,S#state.time_acc])

end.


%When the population_monitor process terminates, it states so, notifies with what op_tag and op_mode it terminated, all the stats gathered, and then shuts down.

%%--------------------------------------------------------------------

%% Func: code_change(OldVsn, State, Extra) -> {ok, NewState}

%% Description: Convert process state when code is changed %%--------------------------------------------------------------------

code_change(_OldVsn, State, _Extra) ->

{ok, State}.

%%--------------------------------------------------------------------

%% Internal functions

%%--------------------------------------------------------------------

extract_AgentIds(Population_Id,AgentType)->

P = genotype:dirty_read({population,Population_Id}),

Specie_Ids = P#population.specie_ids,

case AgentType of

champions ->

extract_ChampionAgentIds(Specie_Ids,[]);

all ->

extract_AllAgentIds(Specie_Ids,[])

end.

%The extract_AgentIds/2 function accepts the Population_Id and a parameter which specifies what type of agents (all agent, or just champions) to extract from the population, after which it extracts the ids of those agents. Depending on the AgentType parameter, the function either calls extract_ChampionAgentIds/2 or extract_AllAgentIds/2, which return the list of agent ids to the caller.

extract_ChampionAgentIds([Specie_Id|Specie_Ids],Acc)->

S = genotype:dirty_read({specie,Specie_Id}),

ChampionAgent_Ids = S#specie.champion_ids,

extract_ChampionAgentIds(Specie_Ids,lists:append(ChampionAgent_Ids,Acc));

extract_ChampionAgentIds([],Acc)->

Acc.

%extract_ChampionAgentIds/2 accumulates the ids of champion agents from every specie in the Specie_Ids list, and then returns that list to the caller.

extract_AllAgentIds([Specie_Id|Specie_Ids],Acc)->

extract_AllAgentIds(Specie_Ids,lists:append(extract_SpecieAgentIds(Specie_Id),

Acc));

extract_AllAgentIds([],Acc)->

Acc.

%extract_AllAgentIds/2 accumulates and returns to the caller an id list of all the agents belong- ing to all the species in the Specie_Ids list.


extract_SpecieAgentIds(Specie_Id)->

S = genotype:dirty_read({specie,Specie_Id}),

S#specie.agent_ids.

%extract_SpecieAgentIds/1 returns a list of agent ids belonging to some particular specie, back

to the caller.

summon_agents(OpMode,Agent_Ids)->

summon_agents(OpMode,Agent_Ids,[]).

summon_agents(OpMode,[Agent_Id|Agent_Ids],Acc)->

Agent_PId = exoself:start(Agent_Id,self()),

summon_agents(OpMode,Agent_Ids,[{Agent_Id,Agent_PId}|Acc]);

summon_agents(_OpMode,[],Acc)->

Acc.

%The summon_agents/2 and summon_agents/3 functions spawns all the agents in the

Agent_ids list, and return to the caller a list of tuples of the following form: [{Agent_Id,Agent_PId}...].

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

test()->

init_population({?INIT_POPULATION_ID,?INIT_CONSTRAINTS,?OP_MODE,

?SELECTION_ALGORITHM}).

%The test/0 function starts the population monitor through init_population/1 with a set of de- fault parameters specified through the macros of this module.

init_population({Population_Id,Specie_Constraints,OpMode,Selection_Algorithm})->

random:seed(now()),

F = fun()->

case genotype:read({population,Population_Id}) of

undefined ->

create_Population(Population_Id,Specie_Constraints);

_ ->

delete_population(Population_Id),

create_Population(Population_Id,Specie_Constraints)

end

end,

Result = mnesia:transaction(F),

case Result of

{atomic,_} ->

population_monitor:start({OpMode,Population_Id,Selection_Algorithm});

Error ->

io:format( “******** ERROR in PopulationMonitor:~p~n ”,[Error])

end.

%The function init_population/1 creates a new population with the id Population_Id, composed of length(Specie_Constraints) species, where each specie uses the particular specie constraint specified within the Specie_Constraints list. The function first checks if a population with the


noted Population_Id already exists. If a population with such an id does already exist, then the function first deletes it, and then creates a new one. Since the ids are usually generated with the genotype:create_UniqueId/0, the only way an already existing Population_Id is dropped into the function as a parameter is if it is intended by the researcher. When performing benchmarks or running other tests on the system, the Population_Id is set to test: Population_Id = test.

create_Population(Population_Id,Specie_Constraints)->

SpecieSize = ?INIT_SPECIE_SIZE,

Specie_Ids = [create_specie(Population_Id,SpecCon,origin,SpecieSize) || SpecCon <-

Specie_Constraints],

Population = #population{

id = Population_Id,

specie_ids = Specie_Ids

},

genotype:write(Population).

create_specie(Population_Id,SpeCon,Fingerprint)->

Specie_Id = genotype:generate_UniqueId(),

create_specie(Population_Id,Specie_Id,0,[],SpeCon,Fingerprint).

create_specie(Population_Id,SpeCon,Fingerprint,SpecieSize)->

Specie_Id = genotype:generate_UniqueId(),

create_specie(Population_Id,Specie_Id,SpecieSize,[],SpeCon,Fingerprint).

create_specie(Population_Id,Specie_Id,0,IdAcc,SpeCon,Fingerprint)->

io:format( “Specie_Id:~p Morphology:~p~n ”,[Specie_Id,

SpeCon#constraint.morphology]),

Specie = #specie{

id = Specie_Id,

population_id = Population_Id,

fingerprint = Fingerprint,

constraint = SpeCon,

agent_ids = IdAcc

},

genotype:write(Specie),

Specie_Id;

create_specie(Population_Id,Specie_Id,Agent_Index,IdAcc,SpeCon,Fingerprint)->

Agent_Id = {genotype:generate_UniqueId(),agent},

genotype:construct_Agent(Specie_Id,Agent_Id,SpeCon),

create_specie(Population_Id,Specie_Id,Agent_Index-1,[Agent_Id|IdAcc],

SpeCon,Fingerprint).

%The create_Population/3 generates length(Specie_Constraints) number of species, where each specie is composed of ?INIT_SPECIE_SIZE number of agents. The function uses the cre- ate_specie/4 to generate the species. The create_specie/3 and create_specie/4 functions are sim- plified versions which use default parameters to call the create_specie/6 function. The cre- ate_specie/6 function constructs the agents using the genotype:construct_Agent/3 function, accumulating the Agent_Ids in the IdAcc list. Once all the agents have been created, the func-


tion creates the specie record, fills in the required elements, writes the specie record to data- base, and then returns the Specie_Id to the caller.

continue(OpMode,Selection_Algorithm)->

Population_Id = test,

population_monitor:start({OpMode,Population_Id,Selection_Algorithm}).

continue(OpMode,Selection_Algorithm,Population_Id)->

population_monitor:start({OpMode,Population_Id,Selection_Algorithm}).

%The function continue/2 and continue/3 are used to summon an already existing population with Population_Id, and continue with the experiment using the chosen Selection_Algorithm.

mutate_population(Population_Id,KeepTot,Selection_Algorithm)->

NeuralEnergyCost = calculate_EnergyCost(Population_Id),

F = fun()->

P = genotype:read({population,Population_Id}),

Specie_Ids = P#population.specie_ids,

[mutate_Specie(Specie_Id,KeepTot,NeuralEnergyCost,Selection_Algorithm) ||

Specie_Id <- Specie_Ids]

end,

{atomic,_} = mnesia:transaction(F).

%The function mutate_population/3 mutates the agents in every specie in its specie_ids list, maintaining each specie within the size of KeepTot. The function first calculates the average cost of each neuron, and then mutates each species separately using the calculated NeuralEnergyCost and Selection_Algorithm as parameters.

mutate_Specie(Specie_Id,PopulationLimit,NeuralEnergyCost,Selection_Algorithm)->

S = genotype:dirty_read({specie,Specie_Id}),

{AvgFitness,Std,MaxFitness,MinFitness} = calculate_SpecieFitness({specie,S}),

Agent_Ids = S#specie.agent_ids,

AgentSummaries = construct_AgentSummaries(Agent_Ids,[]),

io:format( “Selection Algorirthm:~p~n ”,[Selection_Algorithm]),

case Selection_Algorithm of

competition ->

TotSurvivors =

round(length(AgentSummaries)*?SURVIVAL_PERCENTAGE),

SDX=lists:reverse(lists:sort([{Fitness/math:pow(TotN,?EFF), {Fitness,

TotN,Agent_Id}}||{Fitness,TotN,Agent_Id}<-AgentSummaries])),

ProperlySorted_AgentSummaries = [Val || {_,Val}<-SDX], Valid_AgentSummaries=lists:sublist(ProperlySorted_AgentSummaries,

TotSurvivors),

Invalid_AgentSummaries=AgentSummaries--Valid_AgentSummaries,

{_,_,Invalid_AgentIds} = lists:unzip3(Invalid_AgentSummaries), [genotype:delete_Agent(Agent_Id) || Agent_Id <- Invalid_AgentIds], io:format( “Valid_AgentSummaries:~p~n ”,[Valid_AgentSummaries]),


io:format( “Invalid_AgentSummaries:~p~n ”, [Inva-

lid_AgentSummaries]),

TopAgentSummaries = lists:sublist(Valid_AgentSummaries,3),

{_TopFitnessList,_TopTotNs,TopAgent_Ids} =

lists:unzip3(TopAgentSummaries),

io:format( “NeuralEnergyCost:~p~n ”,[NeuralEnergyCost]),

NewGenAgent_Ids = competition(Valid_AgentSummaries,

PopulationLimit,NeuralEnergyCost);

top3 ->

TotSurvivors = 3,

ProperlySorted_AgentSummaries =

lists:reverse(lists:sort(AgentSummaries)),

Valid_AgentSummaries= lists:sublist(ProperlySorted_AgentSummaries,

TotSurvivors),

Invalid_AgentSummaries=AgentSummaries--Valid_AgentSummaries,

{_,_,Invalid_AgentIds} = lists:unzip3(Invalid_AgentSummaries),

{_,_,Valid_AgentIds} = lists:unzip3(Valid_AgentSummaries), [genotype:delete_Agent(Agent_Id) || Agent_Id <- Invalid_AgentIds], io:format( “Valid_AgentSummaries:~p~n ”,[Valid_AgentSummaries]),

io:format( “Invalid_AgentSummaries:~p~n ”, [Inva-

lid_AgentSummaries]),

TopAgentSummaries = lists:sublist(Valid_AgentSummaries,3),

{_TopFitnessList,_TopTotNs,TopAgent_Ids} =

lists:unzip3(TopAgentSummaries),

io:format( “NeuralEnergyCost:~p~n ”,[NeuralEnergyCost]),

NewGenAgent_Ids = top3(Valid_AgentIds,PopulationLimit-

TotSurvivors,[])

end,

{FList,_TNList,_AgentIds}=lists:unzip3(ProperlySorted_AgentSummaries),

[TopFitness|_] = FList,

U_InnovationFactor = case TopFitness > S#specie.innovation_factor of

true ->

0;

false ->

S#specie.innovation_factor-1

end,

genotype:write(S#specie{

agent_ids = NewGenAgent_Ids,

champion_ids = TopAgent_Ids,

fitness = {AvgFitness,Std,MaxFitness,MinFitness},

innovation_factor = U_InnovationFactor}).

%The function mutate_Specie/4 uses the selection algorithm of type Selection_Algorithm to separate the fit from the unfit agents within the same species, and then mutates the fit agents to produce the final mutant offspring, maintaining the total specie size within PopulationLimit. The function first creates a list of agent summaries, which is a list of the format: [{Fit-


ness,TotNeurons,Agent_Id}...]. The function then modifies the fitness scores to be proportional

to the agent's efficiency, which is based on the number of neurons it took the agent to produce this fitness (the NN's size). The function then sorts the updated summaries, and then splits the sorted summary list into a valid (fit) and invalid (unfit) lists of agents. The invalid agents are deleted, and the valid agents are used to create offspring using the Selection_Algorithm with which the function was called. The agent ids belonging to the next generation (the valid agents and their offspring) are then produced by the selection function. Afterward, the innovation factor (the last time the specie's top fitness improved) is updated. Finally, the ids of the top 3 agents within the specie are noted (these are the champion agents, best performing agents within the specie), and the updated specie record is written to database. The above function shows two types of selection algorithms, the ‘competition' selection algorithm, and the ‘top3' selection al- gorithm.

construct_AgentSummaries([Agent_Id|Agent_Ids],Acc)->

A = genotype:dirty_read({agent,Agent_Id}),

construct_AgentSummaries(Agent_Ids,[{A#agent.fitness,

length((genotype:dirty_read({cortex, A#agent.cx_id}))#cortex.neuron_ids),Agent_Id}|Acc]);

construct_AgentSummaries([],Acc)->

Acc.

%The construct_AgentSummaries/2 reads the agents in the Agent_Ids list, and composes a list of tuples with the following format: [{AgentFitness,AgentTotNeurons,Agent_Id}...]. This list

of tuples is referred to as: AgentSummaries. Once the AgentSummaries list is created, it is re- turned to the caller.

competition(Sorted_AgentSummaries,PopulationLimit,NeuralEnergyCost)->

{AlotmentsP,NextGenSize_Estimate} = calculate_alotments(Sorted_AgentSummaries,

NeuralEnergyCost,[],0),

Normalizer = NextGenSize_Estimate/PopulationLimit,

io:format( “Population size normalizer:~p~n ”,[Normalizer]), gather_survivors(AlotmentsP,Normalizer,[]).

%The competition/3 is part of the selection algorithm called ‘competition'. This function first executes calculate_alotments/4 to calculate the number of offspring allotted to each agent in the AgentSummaries list. The function then calculates the Normalizer value, which is used to nor- malize the allotted number of offspring for each agent, to ensure that the final species size is within PopulationLimit. The function then drops into the gather_survivors/3 function, which uses the normalized offspring allotment values to create the actual mutant offspring for each agent.

calculate_alotments([{Fitness,TotNeurons,Agent_Id}|Sorted_AgentSummaries],

NeuralEnergyCost,Acc,NewPopAcc)->

NeuralAlotment = Fitness/NeuralEnergyCost,

MutantAlotment = NeuralAlotment/TotNeurons,

U_NewPopAcc = NewPopAcc+MutantAlotment,

calculate_alotments(Sorted_AgentSummaries,NeuralEnergyCost, [{MutantAlotment,

Fitness,TotNeurons,Agent_Id}|Acc],U_NewPopAcc);


calculate_alotments([],_NeuralEnergyCost,Acc,NewPopAcc)->

io:format( “NewPopAcc:~p~n ”,[NewPopAcc]),

{Acc,NewPopAcc}.

%The calculate_alotments/4 function accepts the AgentSummaries list and for each agent, us- ing the NeuralEnergyCost, calculates how many offspring that agent can produce by using the agent's Fitness, TotNeurons, and NeuralEnergyCost parameters. The function first calculates how many neurons the agent is allotted, based on the agent's fitness and the cost of each neuron (which itself was calculated based on the average performance of the population). From the number of neurons allotted to the agent, the function then calculates how many offspring the agent should be allotted, by dividing the number of neurons it is allotted by the agent's NN size. The function also keeps track of how many offspring will be created from all these agents in general, by adding up all the offspring allotments. The calculate_alotments/4 function does this for each tuple in the AgentSummaries, and then returns the calculated allotment list and NewPopAcc to the caller.

gather_survivors([{MutantAlotment,Fitness,TotNeurons,Agent_Id}|AlotmentsP],

Normalizer, Acc)->

Normalized_MutantAlotment = round(MutantAlotment/Normalizer),

io:format( “Agent_Id:~p Normalized_MutantAlotment:~p~n ”, [Agent_Id,

Normalized_MutantAlotment]),

SurvivingAgent_Ids = case Normalized_MutantAlotment >= 1 of

true ->

MutantAgent_Ids = case Normalized_MutantAlotment >= 2 of

true ->

[create_MutantAgentCopy(Agent_Id)|| _ <-

lists:seq(1,Normalized_MutantAlotment-1)];

false ->

[]

end,

[Agent_Id|MutantAgent_Ids];

false ->

io:format( “Deleting agent:~p~n ”,[Agent_Id]),

genotype:delete_Agent(Agent_Id),

[]

end,

gather_survivors(AlotmentsP,Normalizer,lists:append(SurvivingAgent_Ids,Acc));

gather_survivors([],_Normalizer,Acc)->

io:format( “New Population:~p PopSize:~p~n ”,[Acc,length(Acc)]),

Acc.

%The gather_survivors/3 function accepts the list composed of the allotment tuples and the population normalizer value calculated by the competition/3 function. Using these values it cal- culates the actual number of offspring that each agent should produce, creates the mutant off- spring, and accumulates the new generation agent ids. For each Agent_Id the function first cal- culates the normalized offspring allotment value, to ensure that the final number of agents in the specie is within the population limit of that specie. If the offspring allotment value is less


than 0, the agent is killed. If the offspring allotment is 1, the agent is allowed to survive to the next generation, but is not allowed to create any new offspring. If the offspring allotment is greater than one, then the function creates Normalized_MutantAlotment-1 number of offspring from this fit agent, by calling upon the create_MutantAgentCopy/1 function which returns the id of the new mutant offspring. Once all the offspring have been created, the function returns to the caller a list of ids, composed of the surviving parent agent ids, and their offspring.

create_MutantAgentCopy(Agent_Id)->

AgentClone_Id = genotype:clone_Agent(Agent_Id),

io:format( “AgentClone_Id:~p~n ”,[AgentClone_Id]),

genome_mutator:mutate(AgentClone_Id),

AgentClone_Id.

%The create_MutantAgentCopy/1 first creates a clone of the Agent_Id, and then uses the ge- nome_mutator:mutate/1 function to mutate that clone, returning the id of the cloned agent to the caller.

create_MutantAgentCopy(Agent_Id,safe)->

A = genotype:dirty_read({agent,Agent_Id}),

S = genotype:dirty_read({specie,A#agent.specie_id}),

AgentClone_Id = genotype:clone_Agent(Agent_Id),

Agent_Ids = S#specie.agent_ids,

genotype:write(S#specie{agent_ids = [AgentClone_Id|Agent_Ids]}),

io:format( “AgentClone_Id:~p~n ”,[AgentClone_Id]),

genome_mutator:mutate(AgentClone_Id),

AgentClone_Id.

%The create_MutantAgentCopy/2 function is similar to arity 1 function of the same name, but it also adds the id of the cloned mutant agent to the specie record to which the parent genotype belonged. The specie with its updated agent_ids is then written to database, and the id of the mutant clone is returned to the caller.

top3(_Valid_AgentIds,0,Acc)->

Acc;

top3(Valid_AgentIds,OffspringIndex,Acc)->

Parent_AgentId = lists:nth(random:uniform(length(Valid_AgentIds)),Valid_AgentIds),

MutantAgent_Id = create_MutantAgentCopy(Parent_AgentId),

top3(Valid_AgentIds,OffspringIndex-1,[MutantAgent_Id|Acc]).

%The top3/3 function is a very simple selection algorithm, which just selects the top 3 most fit agents, and then uses the create_MutantAgentCopy/1 function to create their offspring. Each parent agent is allowed to create the same number of offspring.

delete_population(Population_Id)->

P = genotype:dirty_read({population,Population_Id}),

Specie_Ids = P#population.specie_ids,

[delete_specie(Specie_Id) || Specie_Id <- Specie_Ids], mnesia:delete({population,Population_Id}).


%The delete_population/1 function deletes the entire population, by deleting the specie records belonging to the Population_Id, by deleting the agent records belonging to those species, and then by deleting the population record itself.

delete_specie(Specie_Id)->

S = genotype:dirty_read({specie,Specie_Id}),

Agent_Ids = S#specie.agent_ids,

[genotype:delete_Agent(Agent_Id) || Agent_Id <- Agent_Ids], mnesia:delete({specie,Specie_Id}).

%The delete_specie/1 function deletes the agents associated with the Specie_Id, and then de- letes the specie record itself.

calculate_EnergyCost(Population_Id)->

Agent_Ids = extract_AgentIds(Population_Id,all),

TotEnergy = lists:sum([extract_AgentFitness(Agent_Id) || Agent_Id<-Agent_Ids]), TotNeurons = lists:sum([extract_AgentTotNeurons(Agent_Id) || Agent_Id <- Agent_Ids]), EnergyCost = TotEnergy/TotNeurons,

EnergyCost.

%The calculate_EnergyCost/1 function calculates the average cost of each neuron, based on the fitness of each agent in the population, and the total number of neurons in the population. The value is calculated by first adding up all the fitness scores of the agents belonging to the popula- tion, then adding up the total number of neurons composing each agent in the population, and then finally by producing the EnergyCost, where EnergyCost = TotEnergy/TotNeurons. After- wards, the function returns this value to the caller.

extract_AgentTotNeurons(Agent_Id)->

A = genotype:dirty_read({agent,Agent_Id}),

Cx = genotype:dirty_read({cortex,A#agent.cx_id}),

Neuron_Ids = Cx#cortex.neuron_ids,

length(Neuron_Ids).

extract_AgentFitness(Agent_Id)->

A = genotype:dirty_read({agent,Agent_Id}),

A#agent.fitness.

%The function extract_AgentTotNeurons simply extracts the neuron_ids list, and returns the length of that list to the caller. The length of the list is the total number of neurons belonging to the NN based system.

calculate_SpecieFitness({specie,S})->

Agent_Ids = S#specie.agent_ids,

FitnessAcc = calculate_fitness(Agent_Ids),

Sorted_FitnessAcc=lists:sort(FitnessAcc),

[MinFitness|_] = Sorted_FitnessAcc,

[MaxFitness|_] = lists:reverse(Sorted_FitnessAcc),

AvgFitness = functions:avg(FitnessAcc),


Std = functions:std(FitnessAcc),

{AvgFitness,Std,MaxFitness,MinFitness};

calculate_SpecieFitness(Specie_Id)->

S = genotype:dirty_read({specie,Specie_Id}),

calculate_SpecieFitness({specie,S}).

%The calculate_SpecieFitness/1 function calculates the general fitness statistic of the specie:

the average, max, min, and standard deviation of the specie's fitness. The function first com- poses a fitness list by accessing the fitness scores of each agent belonging to it, and then calcu- lates the above noted statistics from that list, returning the tuple with these three values, to the caller.

calculate_fitness(Agent_Ids)->

calculate_fitness(Agent_Ids,[]).

calculate_fitness([Agent_Id|Agent_Ids],FitnessAcc)->

A = genotype:dirty_read({agent,Agent_Id}),

case A#agent.fitness of

undefined ->

calculate_fitness(Agent_Ids,FitnessAcc);

Fitness ->

calculate_fitness(Agent_Ids,[Fitness|FitnessAcc])

end;

calculate_fitness([],FitnessAcc)->

FitnessAcc.

%The calculate_fitness/1 function composes a fitness list using the fitness values belonging to the agents in the Agent_Ids list. If the agent does not yet have a fitness score, if for example it

has just been created/mutated but not yet evaluated, it is skipped. The composed fitness list is

then returned to the caller.

Having now completed the population_monitor module, we move onwards and update the exoself module in the next section.

8.5.16 Updating the exoself Module

The exoself is a process that has a global view of the NN, and can be used to monitor the NN processes, to restore damaged neurons, to recover the NN based system from crashes, and to offer the NN system other services. It is also the pro- gram that in a sense is a phenotypical representation of the agent record. The exoself is the process that is spawned first, and which then in turn converts its NN's genotype to phenotype. It is the exoself process that tunes the NN system's neural weights, and summons the private scapes with which the NN system inter- faces.


Unlike in the previous chapter, there is no trainer in the neuroevolutionary system. The population monitor will spawn exoselfs, which will then spawn the NN based sys- tems. Previously, the exoself used an augmented stochastic hill climbing algorithm to optimize the weights. In this chapter we are creating a memetic and genetic algorithm based TWEANN system. If we evolve the agent's topology during one phase, and let the exoself optimize synaptic weights during another, the system will be a memetic al- gorithm based TWEANN. If we make ?MAX_ATTEMPTS equal to 0, the exoself does not optimize the weights outside the selection/mutation phase ran by the popula- tion_monitor process, weight perturbation is done during the mutation phase using the mutate_weights MO only, and thus this neuroevolutionary system begins to behave as a standard genetic algorithm based TWEANN.

Beside the switch to the mnesia database read and write functions, and the exoself's connection to the population_monitor instead of the trainer process, the main addition to the algorithm has to do with the fact that the new NN has recur- sive connections. Note that when we have spawned the phenotype of a NN with recurrent connections, the neurons which have these recursive output connections, cannot output any signals because they await the input signals from their presyn- aptic connections. Some of these presynaptic neurons cannot produce an output signal either, not until they receive their signals from the recursively connected neurons in the later layers... Thus there is a deadlock, as shown in Fig-8.14 .

Fig. 8.14 Deadlock occurring in recurrent NN based systems.

Let us go through the steps of the above figure:

1. The sensor acquires the sensory signal, from an interaction with the environ- ment, or by generating it internally, for example.

2. The sensor forwards the sensory signal to the two neurons it is connected to, A1 and A2.

3. Neuron A1 is only waiting for a single input signal, from S. As soon as it re- ceives this signal, it processes it and forwards an output to B1. Neuron A2 on the other hand waits for two input signals, a signal from S, and a signal from B2. It receives the signal from S, but not from B2, because B2 is waiting for the signal from A2 before it can send a signal to A2... thus there is a deadlock.


It is for this reason that we have created the ro_ids list for each neuron. Since each neuron that has recursive connections knows about them through ro_ids. It then can, as soon as it is spawned, send a default signal: [0], to the elements in its ro_ids list. This effectively breaks the deadlock, if there was any, since any ele- ment dependent on this input signal, can now continue with processing the input signals and output a signal of its own.

But this approach also leads to a problem when our system functions as a memetic algorithm based TWEANN, and when it performs multiple evaluations of the NN sys- tem. If you look at Fig-8.15, you will note that the default recursive signals, [0], sent out by the neurons when they have just been created, ensures that when the NN has finished its evaluation and should be restarted (when for example we wish to reset or revert the neural weights and perform weight perturbation), will result in some of the neurons (those connected to from the recurrent neurons) have a populated inbox. The neurons which are connected from the recurrent neurons, will have recursive signals in their inbox. To deal with this situation and reset the neurons back to their initial, pris- tine conditions after the NN's evaluation has ended, we need to flush each neuron's inbox. To do this, we first ensure that all neurons are put on pause, then have their in- boxes flushed, then reset all neurons (which might mean that some of the neurons will send out fresh recursive signals [0]) in the NN, and then finally reactive the cortex pro- cess, so that it can start its synchronization of sensors and actuators anew. This new flush buffering function added to the neurons, and the ability to reset/clear neurons to their initial state, is the main new functional addition to the exoself.

Fig. 8.15 To reset the NN based agent after its evaluation, first pause the neurons, then flush the neurons, and then reactivate all neurons. At this point each neuron which has a recurrent connection will send out a new [0] signal to all elements in its ro_ids list.


Let us quickly go through the steps of the above figure before we update the exoself module with this functionality:

1.

2.

3.

4.

5.

6.

7.

8.

9.

Sensor S gathers sensory signals. While at the same time, neuron B2 sends out a default [0] signal to elements in its ro_ids list, which in this case is the neu- ron A2 .

The sensor forwards the sensory signals to A1 and A2 .

A1 and A2 process signals, and because A2 has received in step 1 the signal from B2, it is able to process the two input signals.

A1 and A2 forward their output signals to B1 and B2 respectively.

B1 and B2 process their input signals, and produce output signals.

B1 forwards its signal to the actuator, while B2 forwards its signal to the actua- tor, and to neuron A2.

The actuator element processes its input signals.

At this point the sensors could again gather the sensory signals and forward them to the neurons, but we assume here that the evaluation just finished. At this point, because in step 6 a signal was sent to A2 from B2, the neuron A2 currently has a signal from B2 in its inbox, even though the evaluation is now over . If we at this point restart the NN system, B2 will again send the default recurrent signal to A2, but A2 still has in its inbox the signal from the previous evaluation... For this reason the exoself first sends a pause signal to all the neu- rons, after which point they flush their buffers/inboxes to go back to their ini- tial, clean states.

When a neuron receives a pause signal from the exoself, it first pauses, and then flushes its buffer. Afterwards it awaits the reset signal, at which point it al- so resets its input pids list to its initial state, to again await input signals from all the elements it is connected from. If the neuron has any ids in its ro_ids list, it outputs to those elements the default signal: [0].

Having now discussed in detail the importance of buffer flushing, and the man- ner in which the exoself pauses and resets neurons after each evaluation, we now update the exoself module, as shown in Listing-8.19.

Listing-8.19: The updated implementation of the exoself module.

-module(exoself).

-compile(export_all).

-include( “records.hrl ”).

-record(state,{file_name,genotype,idsNpids,cx_pid,spids,npids,apids,highest_fitness,

tot_evaluations,tot_cycles}).

-define(MAX_ATTEMPTS,50).

start(Agent_Id,PM_PId)->

spawn(exoself,prep,[Agent_Id,PM_PId]).

%The start/2 function spawns a new Agent_Id exoself process belonging to the popula- tion_monitor process with the pid: PM_PId.


prep(Agent_Id,PM_PId)->

random:seed(now()),

IdsNPIds = ets:new(idsNpids,[set,private]),

A = genotype:dirty_read({agent,Agent_Id}),

Cx = genotype:dirty_read({cortex,A#agent.cx_id}),

SIds = Cx#cortex.sensor_ids,

AIds = Cx#cortex.actuator_ids,

NIds = Cx#cortex.neuron_ids,

ScapePIds = spawn_Scapes(IdsNPIds,SIds,AIds),

spawn_CerebralUnits(IdsNPIds,cortex,[Cx#cortex.id]),

spawn_CerebralUnits(IdsNPIds,sensor,SIds),

spawn_CerebralUnits(IdsNPIds,actuator,AIds),

spawn_CerebralUnits(IdsNPIds,neuron,NIds),

link_Sensors(SIds,IdsNPIds),

link_Actuators(AIds,IdsNPIds),

link_Neurons(NIds,IdsNPIds),

{SPIds,NPIds,APIds}=link_Cortex(Cx,IdsNPIds),

Cx_PId = ets:lookup_element(IdsNPIds,Cx#cortex.id,2),

loop(Agent_Id,PM_PId,IdsNPIds,Cx_PId,SPIds,NPIds,APIds,ScapePIds,0,0,0,0,1).

%The prep/2 function prepares and sets up the exoself's state before dropping into the main

loop. The function first reads the agent and cortex records belonging to the Agent_Id NN based system. The function then reads the sensor, actuator, and neuron ids, then spawns the private scapes using the spawn_Scapes/3 function, then spawns the cortex, sensor, actuator, and neuron processes, and then finally links up all these processes together using the link_.../2 functions. Once the phenotype has been generated from the genotype, the exoself drops into its main loop.

loop(Agent_Id,PM_PId,IdsNPIds,Cx_PId,SPIds,NPIds,APIds,ScapePIds,HighestFitness,

EvalAcc,CycleAcc,TimeAcc,Attempt)->

receive

{Cx_PId,evaluation_completed,Fitness,Cycles,Time}->

{U_HighestFitness,U_Attempt}=case Fitness > HighestFitness of

true ->

[NPId ! {self(),weight_backup} || NPId <- NPIds],

{Fitness,0};

false ->

Perturbed_NPIds=get(perturbed),

[NPId ! {self(),weight_restore} || NPId <- Perturbed_NPIds], {HighestFitness,Attempt+1}

end,

[PId ! {self(), reset_prep} || PId <- NPIds],

gather_acks(length(NPIds)),

[PId ! {self(), reset} || PId <- NPIds],

case U_Attempt >= ?MAX_ATTEMPTS of

true -> %End training


U_CycleAcc = CycleAcc+Cycles,

U_TimeAcc = TimeAcc+Time,

A=genotype:dirty_read({agent,Agent_Id}),

genotype:write(A#agent{fitness=U_HighestFitness}),

backup_genotype(IdsNPIds,NPIds),

terminate_phenotype(Cx_PId,SPIds,NPIds,APIds,ScapePIds), io:format( “Agent:~p terminating. Genotype has been backed

up.~n Fitness:~p~n TotEvaluations:~p~n TotCycles:~p~n TimeAcc:~p~n ”, [self(), U_HighestFitness, EvalAcc,U_CycleAcc, U_TimeAcc]),

gen_server:cast(PM_PId,{Agent_Id,terminated,

U_HighestFitness,EvalAcc,U_CycleAcc,U_TimeAcc});

false -> %Continue training

Tot_Neurons = length(NPIds),

MP = 1/math:sqrt(Tot_Neurons),

Perturb_NPIds=[NPId || NPId <- NPIds,random:uniform()<MP], put(perturbed,Perturb_NPIds),

[NPId ! {self(),weight_perturb} || NPId <- Perturb_NPIds], Cx_PId ! {self(),reactivate},

loop(Agent_Id,PM_PId,IdsNPIds,Cx_PId,SPIds, NPIds, APIds,

ScapePIds,U_HighestFitness, EvalAcc+1,CycleAcc+Cycles,TimeAcc+Time,U_Attempt)

end

end.

%The exoself process' main loop awaits from its cortex process the evoluation_completed mes- sage. Once the message is received, based on the fitness achieved, exoself decides whether to continue tuning the weights or terminate the system. Exoself tries to improve the fitness by per- turbing/tuning the weights of its neurons. After each tuning (synaptic weight perturbation) ses- sion, the Neural Network based system performs another evaluation by interacting with the scape until completion (the NN solves a problem, or dies within the scape or...). The order of events is important: When evaluation_completed message is received, the function first checks whether the newly achieved fitness is higher than the thus far highest achieved fitness. If it is not, the exoself sends the neurons a message to restore their weights to previous state, during which they achieved the highest fitness, instead of their current state which yielded the current lower fitness score. If on the other hand the new fitness is higher than the previously highest achieved fitness, then the function tells the neurons to backup their current weights, as these weights represent the NN's best, most fit form yet. Exoself then tells all the neurons to prepare for a reset by sending each neuron the {self(),reset_prep} message. Since the NN can have re- cursive connections, it is important for each neuron to flush its buffer/inbox to be reset into an initial fresh state, which is achieved after the neurons receive the reset_prep message. The exoself then sends the reset message to the neurons, which returns them to their main loop. Finally, the exoself checks whethe r it has already tried to improve the NN's fitness a maximum (?MAX_ATTEMPTS) number of times. If that is the case, the exoself process backs up the up- dated NN (the updated, tuned weights) to database using the backup_genotype/2 function, prints to screen that it is terminating, and sends to the population_monitor the accumulated sta- tistics (highest fitness, evaluation count, cycle count...). On the other hand, if the exoself is not yet done tuning the neural weights, if it has not yet reached its ending condition, it instead ran-


domly selects a set of neurons from its NPIds list, and requests that they perturb their synaptic

weights, then reactivates the cortex, and then finally drops back into its main loop. Each neuron

in the NPId list has a probability: 1/math(sqrt(Tot_Neurons)) of being selected for weight per- turbation, a value that is proportional to the total number of neurons in the NN, and grows with the NN size.

spawn_CerebralUnits(IdsNPIds,CerebralUnitType,[Id|Ids])->

PId = CerebralUnitType:gen(self(),node()),

ets:insert(IdsNPIds,{Id,PId}),

ets:insert(IdsNPIds,{PId,Id}),

spawn_CerebralUnits(IdsNPIds,CerebralUnitType,Ids);

spawn_CerebralUnits(_IdsNPIds,_CerebralUnitType,[])->

true.

%We spawn the process for each element based on its type: CerebralUnitType, using the gen function that belongs to the CerebralUnitType module. Then we enter the {Id,PId} tuple into

our ETS table for later use, thus establishing a mapping between Ids and their PIds. spawn_Scapes(IdsNPIds,Sensor_Ids,Actuator_Ids)->

Sensor_Scapes = [(genotype:dirty_read({sensor,Id}))#sensor.scape || Id<-Sensor_Ids], Actuator_Scapes = [(genotype:dirty_read({actuator,Id}))#actuator.scape || Id<-

Actuator_Ids],

Unique_Scapes = Sensor_Scapes++(Actuator_Scapes--Sensor_Scapes),

SN_Tuples=[{scape:gen(self(),node()),ScapeName} || {private,ScapeName}<-

Unique_Scapes],

[ets:insert(IdsNPIds,{ScapeName,PId}) || {PId,ScapeName} <- SN_Tuples],

[ets:insert(IdsNPIds,{PId,ScapeName}) || {PId,ScapeName} <-SN_Tuples],

[PId ! {self(),ScapeName} || {PId,ScapeName} <- SN_Tuples],

[PId || {PId,_ScapeName} <-SN_Tuples].

%The spawn_Scapes/3 function first extracts all the scapes that the sensors and actuators inter- face with. Then it creates a filtered scape list which only holds unique scape records. Finally, from this list it selects the private scapes, and then spawns them.

link_Sensors([SId|Sensor_Ids],IdsNPIds) ->

S=genotype:dirty_read({sensor,SId}),

SPId = ets:lookup_element(IdsNPIds,SId,2),

Cx_PId = ets:lookup_element(IdsNPIds,S#sensor.cx_id,2),

SName = S#sensor.name,

Fanout_Ids = S#sensor.fanout_ids,

Fanout_PIds = [ets:lookup_element(IdsNPIds,Id,2) || Id <- Fanout_Ids],

Scape=case S#sensor.scape of

{private,ScapeName}->

ets:lookup_element(IdsNPIds,ScapeName,2)

end,

SPId ! {self(),{SId,Cx_PId,Scape,SName,S#sensor.vl,Fanout_PIds}},

link_Sensors(Sensor_Ids,IdsNPIds);

Chapter 8 Developing a Simple Neuroevolutionary Platform link_Sensors([],_IdsNPIds)->

ok.

%The link_Sensors/2 function sends to the already spawned and waiting sensors their states, composed of the PId lists and other information which are needed by the sensors to link up and interface with other elements in the distributed phenotype.

link_Actuators([AId|Actuator_Ids],IdsNPIds) ->

A=genotype:dirty_read({actuator,AId}),

APId = ets:lookup_element(IdsNPIds,AId,2),

Cx_PId = ets:lookup_element(IdsNPIds,A#actuator.cx_id,2),

AName = A#actuator.name,

Fanin_Ids = A#actuator.fanin_ids,

Fanin_PIds = [ets:lookup_element(IdsNPIds,Id,2) || Id <- Fanin_Ids],

Scape=case A#actuator.scape of

{private,ScapeName}->

ets:lookup_element(IdsNPIds,ScapeName,2)

end,

APId ! {self(),{AId,Cx_PId,Scape,AName,Fanin_PIds}},

link_Actuators(Actuator_Ids,IdsNPIds);

link_Actuators([],_IdsNPIds)->

ok.

%The link_Actuators/2 function sends to the already spawned and waiting actuators their states, composed of the PId lists and other information which are needed by the actuators to link up and interface with other elements in the distributed phenotype.

link_Neurons([NId|Neuron_Ids],IdsNPIds) ->

N=genotype:dirty_read({neuron,NId}),

NPId = ets:lookup_element(IdsNPIds,NId,2),

Cx_PId = ets:lookup_element(IdsNPIds,N#neuron.cx_id,2),

AFName = N#neuron.af,

Input_IdPs = N#neuron.input_idps,

Output_Ids = N#neuron.output_ids,

RO_Ids = N#neuron.ro_ids,

Input_PIdPs = convert_IdPs2PIdPs(IdsNPIds,Input_IdPs,[]),

Output_PIds = [ets:lookup_element(IdsNPIds,Id,2) || Id <- Output_Ids],

RO_PIds = [ets:lookup_element(IdsNPIds,Id,2) || Id <- RO_Ids],

NPId ! {self(),{NId,Cx_PId,AFName,Input_PIdPs,Output_PIds,RO_PIds}},

link_Neurons(Neuron_Ids,IdsNPIds);

link_Neurons([],_IdsNPIds)->

ok.

%The link_Neurons/2 function sends to the already spawned and waiting neurons their states, composed of the PId lists and other information needed by the neurons to link up and interface with other elements in the distributed phenotype.

convert_IdPs2PIdPs(_IdsNPIds,[{bias,[Bias]}],Acc)->


lists:reverse([Bias|Acc]);

convert_IdPs2PIdPs(IdsNPIds,[{Id,Weights}|Fanin_IdPs],Acc)->

convert_IdPs2PIdPs(IdsNPIds,Fanin_IdPs, [{ets:lookup_element(IdsNPIds, Id,

2),Weights}|Acc]);

convert_IdPs2PIdPs(_IdsNPIds,[],Acc)->

lists:reverse(Acc).

%The convert_IdPs2PIdPs/3 converts the IdP tuples: {Id, Weights}, into tuples that use PIds instead of Ids: {PId, Weights}, such that the Neuron will know which weights are to be associ- ated with which incoming vector signals. The last element is the bias, which is added to the list in a none-tuple form. Afterwards, the list is reversed to its proper order, and returned to the caller.

link_Cortex(Cx,IdsNPIds) ->

Cx_Id = Cx#cortex.id,

Cx_PId = ets:lookup_element(IdsNPIds,Cx_Id,2),

SIds = Cx#cortex.sensor_ids,

AIds = Cx#cortex.actuator_ids,

NIds = Cx#cortex.neuron_ids,

SPIds = [ets:lookup_element(IdsNPIds,SId,2) || SId <- SIds],

NPIds = [ets:lookup_element(IdsNPIds,NId,2) || NId <- NIds],

APIds = [ets:lookup_element(IdsNPIds,AId,2) || AId <- AIds],

Cx_PId ! {self(),Cx_Id,SPIds,NPIds,APIds},

{SPIds,NPIds,APIds}.

%The link_Cortex/2 function sends to the already spawned and waiting cortex its state, com- posed of the PId lists and other information which is needed by the cortex to link up and inter- face with other elements in the distributed phenotype.

backup_genotype(IdsNPIds,NPIds)->

Neuron_IdsNWeights = get_backup(NPIds,[]),

update_genotype(IdsNPIds,Neuron_IdsNWeights),

io:format( “Finished updating genotype~n ”).

get_backup([NPId|NPIds],Acc)->

NPId ! {self(),get_backup},

receive

{NPId,NId,WeightTuples}->

get_backup(NPIds,[{NId,WeightTuples}|Acc])

end;

get_backup([],Acc)->

Acc.

%The backup_genotype/2 uses get_backup/2 to contact all the neurons in the NN and request the neuron's Ids and their Input_IdPs. Once the updated Input_IdPs from all the neurons have been accumulated, they are passed through the update_genotype/2 function to produce updated neuron tuples, and are then written to database. This effectively updates the NN genotype with the now tuned neurons.

Chapter 8 Developing a Simple Neuroevolutionary Platform update_genotype(IdsNPIds,[{N_Id,PIdPs}|WeightPs])->

N = genotype:dirty_read({neuron,N_Id}),

Updated_InputIdPs = convert_PIdPs2IdPs(IdsNPIds,PIdPs,[]), U_N = N#neuron{input_idps = Updated_InputIdPs}, genotype:write(U_N),

update_genotype(IdsNPIds,WeightPs);

update_genotype(_IdsNPIds,[])->

ok.

%For every {N_Id,PIdPs} tuple, the update_genotype/3 function extracts the neuron with the id: N_Id, updates the neuron's input_IdPs, and writes the updated neuron to database.

convert_PIdPs2IdPs(IdsNPIds,[{PId,Weights}|Input_PIdPs],Acc)->

convert_PIdPs2IdPs(IdsNPIds,Input_PIdPs, [{ets:lookup_element(IdsNPIds,

PId,2),Weights}|Acc]);

convert_PIdPs2IdPs(_IdsNPIds,[Bias],Acc)->

lists:reverse([{bias,[Bias]}|Acc]);

convert_PIdPs2IdPs(_IdsNPIds,[],Acc)->

lists:reverse(Acc).

%The convert_PIdPs2IdPs/3 function performs the conversion from PIds to Ids for every {PId,Weights} tuple in the Input_PIdPs list. The updated Input_IdPs list is then returned to the caller.

terminate_phenotype(Cx_PId,SPIds,NPIds,APIds,ScapePIds)->

[PId ! {self(),terminate} || PId <- SPIds],

[PId ! {self(),terminate} || PId <- APIds],

[PId ! {self(),terminate} || PId <- NPIds],

[PId ! {self(),terminate} || PId <- ScapePIds],

Cx_PId ! {self(),terminate}.

%The terminate_phenotype/5 function terminates sensors, actuators, neurons, all private scapes, and the cortex, making up the phenotype of the NN based system.

gather_acks(0)->

done;

gather_acks(PId_Index)->

receive

{_From,ready}->

gather_acks(PId_Index-1)

after 100000 ->

io:format( “******** Not all acks received:~p~n ”,[PId_Index])

end.

%gather_acks/1 function ensures that it receives all X number of {From, ready} messages from the neurons, before it returns the atom: done, to the caller. X is set by the caller of the function.


8.5.17 Updating the neuron Module

Though the next element in the hierarchy is the cortex element, it does not re- quire any updates. The only remaining module that needs to be updated is the neu- ron module. Unlike in the previous chapter, our current neurons need to be able to support recursive connections, and all the synchronization detail that comes with it. As discussed in the previous section, the neurons which have recursive connec- tions need to produce and forward the signal: [0], to the ids in the ro_ids list, when just being initiated. Furthermore, when the NN based system has completed inter- acting with a scape, and the cortex has deactivated (but not shut down), if the exoself wants to reactivate the cortex, the neurons need to flush their buff- ers/inboxes so that they can return to their initial, pristine form (but with updated synaptic weights). When flushing its buffer, the neuron gets rid of any recursive signals remaining in its inbox, as was shown in the previous section.

Another addition in this chapter is that the NN based systems have access to different kinds of neural activation functions. The activation function used by the neuron is randomly selected during the neuron's creation during evolution. Since in the future we will continue adding new activation functions, and in general new mathematical and geometric functions, we should create a new small module called functions.erl, which will contain all these activation and other types of func- tions. Thus, since the neuron record uses the element af which contains the name of the actual activation function, these activation function names will be the names of the actual implemented functions located in the functions.erl module. The neu- ron will need only call [functions:ActivationFunction(Dot_Product)], to produce its output value.

With these new additions, the neuron should be able to perform the following tasks:

1. Start up in a new process, wait for the exoself to set its state, check if it has any recursive output connections, and if so, send to those elements a default signal: {self(),forward,[0]} , and then drop into its main loop.

2. The neuron should be able to gather all the incoming signals from the In- put_PIds specified in its Input_PIdPs list, and calculate a dot product from the input signals and the weights associated with those input signals.

3. Once all the input signals correlated with the Input_PIds in its Input_PIdPs list have been received, the neuron adds the bias, if any, to the calculated dot prod- uct, and then, based on the AF (Activation Function) tag, calculates the output by executing: [functions:ActivationFunction(Dot_Product)] . The calculated output signal is then propagated to all the PIds in its Output_PIds list.

4. The neuron should be able to receive the weight_backup signal. When receiv- ing this signal, the neuron should store its current synaptic weights list to memory.

5. The neuron should be able to receive the weight_restore signal. When receiv- ing this signal, the neuron should restore the synaptic weights it stored in


memory, and use them instead of the weights it currently uses in association with the Input_PIds.

6. The neuron should be able to accept the weight_perturb message. When receiv- ing this message, the neuron should go through all its weights, and select each synaptic weight for perturbation with probability of 1/sqrt(TotWeights) . The perturbation intensity applied to each synaptic weight is chosen with uniform distribution to be between -Pi and Pi .

7. The neuron should be able to accept the reset_prep message. When the neuron receives this message, it should flush its inbox, and then wait for the reset mes- sage, after which it should send out the default recursive signal (if any), and drop back into its main loop. Since all the other neurons should have also flushed their inboxes (the reset_prep and reset signaling is synchronized by the exoself, to ensure that all neurons go back to initial state when needed), the neurons are ready to receive these new recursive signals. At this point, their in- boxes are empty, and not containing the recursive signals from their previous cycles.

The modified neuron module updated with these features is shown in the fol- lowing listing.

Listing 8.20: The updated neuron module.

-module(neuron).

-compile(export_all).

-include( “records.hrl ”).

-define(DELTA_MULTIPLIER,math:pi()*2).

-define(SAT_LIMIT,math:pi()*2).

-define(RO_SIGNAL,0).

gen(ExoSelf_PId,Node)->

spawn(Node,?MODULE,prep,[ExoSelf_PId]).

prep(ExoSelf_PId) ->

{V1,V2,V3} = now(),

random:seed(V1,V2,V3),

receive

{ExoSelf_PId,{Id,Cx_PId,AF,Input_PIdPs,Output_PIds,RO_PIds}} ->

fanout(RO_PIds,{self(),forward,[?RO_SIGNAL]}),

loop(Id,ExoSelf_PId,Cx_PId,AF,{Input_PIdPs,Input_PIdPs},Output_PIds,

RO_PIds,0)

end.

%When gen/2 is executed, it spawns the neuron element and immediately begins to wait for its initial state message from the exoself. Once the state message arrives, the neuron sends out the default forward signals to all elements in its ro_ids list. Afterwards, prep drops into the neu- ron's main receive loop.


loop(Id,ExoSelf_PId,Cx_PId,AF,{[{Input_PId,Weights}|Input_PIdPs],MInput_PIdPs},

Output_PIds,RO_PIds,Acc)->

receive

{Input_PId,forward,Input}->

Result = dot(Input,Weights,0),

loop(Id,ExoSelf_PId,Cx_PId,AF,{Input_PIdPs,MInput_PIdPs},Output_PIds,

RO_PIds,Result+Acc);

{ExoSelf_PId,weight_backup}->

put(weights,MInput_PIdPs),

loop(Id,ExoSelf_PId,Cx_PId,AF,{[{Input_PId,Weights}|Input_PIdPs],

MInput_PIdPs}, Output_PIds,RO_PIds,Acc);

{ExoSelf_PId,weight_restore}->

RInput_PIdPs = get(weights),

loop(Id,ExoSelf_PId,Cx_PId,AF,{RInput_PIdPs,RInput_PIdPs},Output_PIds,

RO_PIds,Acc);

{ExoSelf_PId,weight_perturb}->

PInput_PIdPs=perturb_IPIdPs(MInput_PIdPs),

loop(Id,ExoSelf_PId,Cx_PId,AF,{[{Input_PId,Weights}|Input_PIdPs],

PInput_PIdPs},Output_PIds,RO_PIds,Acc);

{ExoSelf,reset_prep}->

neuron:flush_buffer(),

ExoSelf ! {self(),ready},

receive

{ExoSelf, reset}->

fanout(RO_PIds,{self(),forward,[?RO_SIGNAL]})

end,

loop(Id,ExoSelf_PId,Cx_PId,AF,{MInput_PIdPs,MInput_PIdPs},Output_PIds,

RO_PIds,0);

{ExoSelf_PId,get_backup}->

ExoSelf_PId ! {self(),Id,MInput_PIdPs},

loop(Id,ExoSelf_PId,Cx_PId,AF,{[{Input_PId,Weights}|Input_PIdPs],

MInput_PIdPs},Output_PIds,RO_PIds,Acc);

{ExoSelf_PId,terminate}->

ok

end;

loop(Id,ExoSelf_PId,Cx_PId,AF,{[Bias],MInput_PIdPs},Output_PIds,RO_PIds,Acc)->

Output = functions:AF(Acc+Bias),

[Output_PId ! {self(),forward,[Output]} || Output_PId <- Output_PIds], loop(Id,ExoSelf_PId,Cx_PId,AF,{MInput_PIdPs,MInput_PIdPs},Output_PIds,RO_PIds,0);

loop(Id,ExoSelf_PId,Cx_PId,AF,{[],MInput_PIdPs},Output_PIds,RO_PIds,Acc)->

Output = functions:AF(Acc),

[Output_PId ! {self(),forward,[Output]} || Output_PId <- Output_PIds], loop(Id,ExoSelf_PId,Cx_PId,AF,{MInput_PIdPs,MInput_PIdPs},Output_PIds,RO_PIds,0).


%The neuron process waits for vector signals from all the processes that it's connected from, taking the dot product of the input and weight vectors, and then adding it to the accumulator. Once all the signals from Input_PIds are received, the accumulator contains the dot product to which the neuron then adds the bias and executes the activation function. After fanning out the output signal, the neuron again returns to waiting for incoming signals. When the neuron re- ceives the {ExoSelf_PId,get_backup} message, it forwards to the exoself its full MInput_PIdPs list, and its Id. The MInput_PIdPs contains the modified, tuned and most effective version of the input_idps. The neuron process also accepts the weight_backup signal, when receiving it, the neuron saves to process dictionary the current MInput_PIdPs. When the neuron receives the weight_restore signal, it reads back from the process dictionary the stored Input_PIdPs, and switches over to using it as its active Input_PIdPs list. When the neuron receives the weight_perturb signal from the exoself, it perturbs the weights by executing the per- turb_IPIdPs/1 function, which returns the updated/perturbed weight list. Finally, the neuron can also accept a reset_prep signal, which makes the neuron flush its buffer in the off chance that it has a recursively sent to it signal in its inbox. After flushing its buffer, the neuron waits for the exoself to send it the reset signal, at which point the neuron, now fully refreshed after the flush_buffer/0, outputs a default forward signal to its recursively connected elements (ro_ids), if any, and then drops back into its main receive loop.

dot([I|Input],[W|Weights],Acc) ->

dot(Input,Weights,I*W+Acc);

dot([],[],Acc)->

Acc;

dot([],[Bias],Acc)->

Acc+Bias.

%The dot/3 function accepts an input vector and a weight list, and computes the dot product of the two vectors.

fanout([Pid|Pids],Msg)->

Pid ! Msg,

fanout(Pids,Msg);

fanout([],_Msg)->

true.

%The fanout/2 function fans out the Msg to all the PIds in its list.

flush_buffer()->

receive

_ ->

flush_buffer()

after 0 ->

done

end.

%The flush_buffer/0 empties out the element's inbox.

perturb_IPIdPs(Input_PIdPs)->


Tot_Weights=lists:sum([length(Weights) || {_Input_PId,Weights}<-Input_PIdPs]),

MP = 1/math:sqrt(Tot_Weights),

perturb_IPIdPs(MP,Input_PIdPs,[]).

perturb_IPIdPs(MP,[{Input_PId,Weights}|Input_PIdPs],Acc)->

U_Weights = perturb_weights(MP,Weights,[]),

perturb_IPIdPs(MP,Input_PIdPs,[{Input_PId,U_Weights}|Acc]);

perturb_IPIdPs(MP,[Bias],Acc)->

U_Bias = case random:uniform() < MP of

true->

sat((random:uniform()-0.5)*?DELTA_MULTIPLIER+Bias,-?SAT_LIMIT,

?SAT_LIMIT);

false ->

Bias

end,

lists:reverse([U_Bias|Acc]);

perturb_IPIdPs(_MP,[],Acc)->

lists:reverse(Acc).

%The perturb_IPIdPs/1 function calculates the probability with which each neuron in the In- put_PIdPs is chosen to be perturbed. The probability is based on the total number of weights in the Input_PIdPs list, with the actual mutation probability equating to the inverse of square root of the total number of weights. The perturb_IPIdPs/3 function goes through each weights block and calls the perturb_weights/3 to perturb the weights.

perturb_weights(MP,[W|Weights],Acc)->

U_W = case random:uniform() < MP of

true->

sat((random:uniform()-0.5)*?DELTA_MULTIPLIER+W,-

?SAT_LIMIT, ?SAT_LIMIT);

false ->

W

end,

perturb_weights(MP,Weights,[U_W|Acc]);

perturb_weights(_MP,[],Acc)->

lists:reverse(Acc).

%The perturb_weights/3 function is one that actually goes through each weight block, and per- turbs each weight with a probability of MP. If the weight is chosen to be perturbed, the pertur- bation intensity is chosen uniformly between -Pi and Pi.

sat(Val,Min,Max)->

if

Val < Min -> Min;

Val > Max -> Max;

true -> Val

end.


%The sat/3 function simply ensures that the Val is neither less than min or greater than max. When used with synaptic weights (or other parameters), this function makes sure that the syn- aptic weights get saturated at the Min and Max values, rather than growing in magnitude with- out bound.

As you have noticed, the neuron executes functions:AF(DotProduct) , since all activation functions, and other useful mathematical functions, are specified in the new functions module. For clarity and completeness, the functions module is shown in the following listing.

Listing-8.21: The implementation of the functions module.

-module(functions).

-compile(export_all).

saturation(Val)->

case Val > 1000 of

true -> 1000;

false ->

case Val < -1000 of

true -> -1000;

false -> Val

end

end.

%The function saturation/1 accepts a value Val, and returns the same if its magnitude is below 1000. Otherwise it returns -1000 or 1000, if it's less than or greater than -1000 or 1000 respec- tively. Thus Val saturates at -1000 and 1000.

saturation(Val,Spread)->

case Val > Spread of

true -> Spread;

false ->

case Val < -Spread of

true -> -Spread;

false -> Val

end

end.

%The saturation/2 function is similar to saturation/1, but here the spread (symmetric Max and Min values) is specified by the caller.

scale([H|T],Max,Min)->

[scale(Val,Max,Min)||Val<-[H|T]];

scale(Val,Max,Min)-> %Nm = (Y*2 - (Max + Min))/(Max-Min)

case Max == Min of

true -> 0;


false -> (Val*2 - (Max+Min))/(Max-Min)

end.

%The scale/3 function accepts a list of values, and scales them to be between the specified Min and Max values.

sat(Val,Max,Min)->

case Val > Max of

true -> Max;

false ->

case Val < Min of

true -> Min;

false -> Val

end

end.

%The sat/3 function is similar to saturation/2 function, but here the Max and Min can be differ- ent, and are specified by the caller.

sat_dzone(Val,Max,Min,DZMax,DZMin)->

case (Val < DZMax) and (Val > DZMin) of

true -> 0;

false -> sat(Val,Max,Min)

end.

%The sat_DZone/5 function is similar to the sat/3 function, but here, if Val is between DZMin and DZMax, it is zeroed.

%%%%%%%%%%%%%%%% Activation Functions %%%%%%%%%%%%%%%%%%%

tanh(Val)->

math:tanh(Val).

cos(Val)->

math:cos(Val).

sin(Val)->

math:sin(Val).

sgn(0)->

0;

sgn(Val)->

case Val > 0 of

true -> 1;

false -> -1

end.

bin(Val)->

case Val > 0 of

Chapter 8 Developing a Simple Neuroevolutionary Platform true -> 1;

false -> 0

end.

% The bin/1 function converts Val into a binary value, 1 if Val > 0, and 0 if Val = < 0. trinary(Val)->

if

(Val < 0.33) and (Val > -0.33) -> 0;

Val >= 0.33 -> 1;

Val =< -0.33 -> -1

end.

%The trinary/1 function converts Val into a trinary value.

multiquadric(Val)->

math:pow(Val*Val + 0.01,0.5).

absolute(Val)->

abs(Val).

linear(Val)->

Val.

quadratic(Val)->

sgn(Val)*Val*Val.

gaussian(Val)->

gaussian(2.71828183,Val).

gaussian(Const,Val)->

V = case Val > 10 of

true -> 10;

false ->

case Val < -10 of

true -> -10;

false -> Val

end

end,

math:pow(Const,-V*V).

sqrt(Val)->

sgn(Val)*math:sqrt(abs(Val)).

log(Val)->

case Val == 0 of

true -> 0;

8.6 Summary

false -> sgn(Val)*math:log(abs(Val))

end.

sigmoid(Val)->

V = case Val > 10 of

true -> 10;

false ->

case Val < -10 of

true -> -10;

false -> Val

end

end,

2/(1+math:pow(2.71828183,-V)) - 1.

sigmoid1(Val)->

Val/(1+abs(Val)).

avg(List)->

lists:sum(List)/length(List).

%The avg/1 function accepts a List for a parameter, and then returns the average of the list to

the caller.

std(List)->

Avg = avg(List),

std(List,Avg,[]).

std([Val|List],Avg,Acc)->

std(List,Avg,[math:pow(Avg-Val,2)|Acc]);

std([],_Avg,Acc)->

Variance = lists:sum(Acc)/length(Acc),

math:sqrt(Variance).

%The std/1 function accepts a List for a parameter, and then returns to the caller the standard deviation of the list.


In this chapter we converted our single NN optimization system, into a fully fledged topology and weight evolving artificial neural network system. We have implemented an approach to present various problems to our NN systems, an ap- proach utilizing the scapes and the morphology concepts. We created a popula- tion_monitor, a system that can spawn and monitor a population of NN based agents, select the fit, delete the unfit, and in general apply the evolutionary method to the population of agents. We also implemented the constraint record, through which we can specify all the various parameters which the evolving specie will be constrained by. Finally, we developed the necessary complexifying topological mutation operators, which can add/remove bias values to/from neurons, mutate


their activation functions, add new neurons to the NNs, add new synaptic connec- tions to the NNs, and add sensors and actuators to the NNs.

We have also created a new functions module, which will from now on contain the activation functions used by the neuron, and other mathematical functions used by our system. Through the functions module, we can fully decouple the activa- tion functions from the neurons using them. A neuron can now use any activation function, no matter its form, as long as it returns a properly formatted value. This also means that our neuron can now function as anything, as an AND gate, as an OR gate… depending on what activation functions we give it access to. This gives our system a good starting point with regards to its flexibility, and areas in which we can apply it to.

We have now covered and created all the necessary modules of our basic neuroevolutionary system. The remaining logger and benchmark modules are non- essentials, and we will build them when we begin expanding our neuroevolutionary platform in later chapters. At this point, we are ready to move forward and perform a detailed test of the mutation operators, and then the entire neuroevolutionary sys- tem. We test our newly constructed system in the next chapter.

8.7 Reference

[1] Khepera robot: http://www.k-team.com/

Chapter 9 Testing the Neuroevolutionary System

Abstract In this chapter we test the newly created basic neuroevolutionary sys- tem, by first testing each of its mutation operators, and then by applying the whole system to the XOR mimicking problem. Though the XOR problem test will run to completion and without errors, a more detailed, manual analysis of the evolved to- pologies and genotypes of the fit agents will show a number of bugs to be present. The origins of the bugs is then analyzed, and the errors are fixed. Afterwards, the updated neuroevolutionary system is then successfully re-tested.

9.1 Testing the Mutation Operators

Having created the basic neuroevolutionary system, we need to test whether the mutation operators work as we intended them to. We have set up all the complexifying mutation operators to leave the system in a connected state. This means that when we apply these mutation operators, the resulting NN topology is such, that the signal can get from the sensors, all the way through the NN, and to the actuators. The pruning mutation operators: remove_inlink, remove_outlink, remove_neuron, remove_sensor, remove_actuator, may leave the NN in such a state that it is no longer able to process information, by creating a break in the connected graph, as shown in the example of Fig-9.1 . We could start using the prun- ing mutation operators later on, after we have first created a program inside the genome_mutator module that ensures that all the resulting mutant NN systems are not disconnected after such pruning mutation operators have been applied.

DOI 10.1007/978-1-4614- 4463 - 3_9 , © Springer Science+Business Media New York 2013


Fig. 9.1 Pruning mutation operators that leave a NN disconnected.

Let us now run a few mutation operator tests, to see if the resulting topologies after we have applied some mutation operators to the NN, are as expected. When you perform the same tests, the results may slightly differ from mine, since the el- ements in your NN will have different Ids, and because the mutation operators are applied randomly. The test of each mutation operator will have the following steps:

1. Generate a test NN, which is composed of a single neuron, connected from the sensor xor_GetInput , and connected to the actuator xor_SendOutput . This is done by simply executing genotype:create_test() , which creates a xor_mimic morphology based seed agent.

2. Apply an available mutation operator by executing: genome_mutator:test(test, Mutator).

3. Execute genotype:print(test) to print the resulting genotype to console, and then compare it to the original genotype to ensure that the resulting mutated geno- type is as expected based on the mutation operator used.

4. Test the resulting NN on the simple XOR problem for which it has the sensor and actuator, by executing exoself:start(test,void). There will not exist a popu- lation_monitor process at this time, but that should not affect the results. The goal here is to ensure that the NN does not stall, that the signals can go all the way through it, from sensors to actuators, and that the NN system is functional. In this case we do not expect the NN to solve the problem, because the topolo- gy is not evolving towards any particular goal.


Let us now go through these steps for each mutation operator. For the sake of being brief, I will show the entire console printout for the first mutation operator test, but for all the other mutation operators I will only display the most significant console printout parts.

mutate_weights: This mutation operator selects a random neuron in the NN and perturbs/mutates its synaptic weights.

2> genotype:create_test().

{agent,test,0,undefined,test,

{{origin,7.572689688224582e-10},cortex},

{[{0,1}],

[],

[{sensor,undefined,xor_GetInput,undefined,

{private,xor_sim},

2

[{{0,7.572689688218573e-10},neuron}],

undefined}],

[{actuator,undefined,xor_SendOutput,undefined,

{private,xor_sim},

1

[{{0,7.572689688218573e-10},neuron}],

undefined}]},

{constraint,xor_mimic,[tanh,cos,gauss,abs]},

[],undefined,0,

[{0,[{{0,7.572689688218573e-10},neuron}]}]}

{cortex,{{origin,7.572689688224582e-10},cortex},

test,

[{{0,7.572689688218573e-10},neuron}],

[{{-1,7.572689688218636e-10},sensor}],

[{{1,7.572689688218589e-10},actuator}]}

{sensor,{{-1,7.572689688218636e-10},sensor},

xor_GetInput,

{{origin,7.572689688224582e-10},cortex},

{private,xor_sim},

2

[{{0,7.572689688218573e-10},neuron}],

undefined}

{neuron,{{0,7.572689688218573e-10},neuron},

0

{{origin,7.572689688224582e-10},cortex},

tanh,

[{{{-1,7.572689688218636e-10},sensor},

[-0.08541081650616245,-0.028821611144310255] }],

[{{1,7.572689688218589e-10},actuator}],

Chapter 9 Testing the Neuroevolutionary System []}

{actuator,{{1,7.572689688218589e-10},actuator},

xor_SendOutput,

{{origin,7.572689688224582e-10},cortex},

{private,xor_sim},

1

[{{0,7.572689688218573e-10},neuron}],

undefined}

{atomic,{atomic,[ok]}}

3> genome_mutator:test(test,mutate_weights).

{atomic,{atomic,ok}}


{agent,test,0,undefined,test,

{{origin,7.572689688224582e-10},cortex},

{[{0,1}],

[],

[{sensor,undefined,xor_GetInput,undefined,

{private,xor_sim},

2

[{{0,7.572689688218573e-10},neuron}],

undefined}],

[{actuator,undefined,xor_SendOutput,undefined,

{private,xor_sim},

1

[{{0,7.572689688218573e-10},neuron}],

undefined}]},

{constraint,xor_mimic,[tanh,cos,gauss,abs]},

[{mutate_weights,{{0,7.572689688218573e-10},neuron}}],

undefined,0,

[{0,[{{0,7.572689688218573e-10},neuron}]}]}

{cortex,{{origin,7.572689688224582e-10},cortex},

test,

[{{0,7.572689688218573e-10},neuron}],

[{{-1,7.572689688218636e-10},sensor}],

[{{1,7.572689688218589e-10},actuator}]}

{sensor,{{-1,7.572689688218636e-10},sensor},

xor_GetInput,

{{origin,7.572689688224582e-10},cortex},

{private,xor_sim},

2

[{{0,7.572689688218573e-10},neuron}],

undefined}

{neuron,{{0,7.572689688218573e-10},neuron},

0

{{origin,7.572689688224582e-10},cortex},


tanh,

[{{{-1,7.572689688218636e-10},sensor},

[-1.81543903255671,0.28220989176010963] }],

[{{1,7.572689688218589e-10},actuator}],

[]}

{actuator,{{1,7.572689688218589e-10},actuator},

xor_SendOutput,

{{origin,7.572689688224582e-10},cortex},

{private,xor_sim},

1

[{{0,7.572689688218573e-10},neuron}],

undefined}

{atomic,[ok]}

As you can see from the printout, the mutate_weights operator chose a random neuron in the NN, which in this case is just the single existing neuron, and then mutated the synaptic weights associated with the sensor that it is connected from. The synaptic weights were mutated from their original values of:

[-0.08541081650616245, -0.028821611144310255]

to:

[-1.81543903255671, 0.28220989176010963].

We now test the mutated NN system on the problem that its morphology de- fines it for, the XOR mimicking problem.

5> exoself:start(test,void).

<0.128.0>

Finished updating genotype

Terminating the phenotype:

Cx_PId:<0.131.0>

SPIds:[<0.132.0>]

NPIds:[<0.134.0>]

APIds:[<0.133.0>]

ScapePids:[<0.130.0>]

Sensor:{{-1,7.572689688218636e-10},sensor} is terminating.

Agent:<0.128.0> terminating. Genotype has been backed up.

Fitness:0.505631430344058

TotEvaluations:52

TotCycles:208

TimeAcc:7226

Cortex:{{origin,7.572689688224582e-10},cortex} is terminating.


It works! The exoself ran, and after having finished tuning the weights with our augmented stochastic hill-climber algorithm, it updated the genotype, terminated the phenotype by terminating all the processes associated with it (SPIds, NPIds, APIds, and ScapePids), and then printed to screen the stats of the NN system's run: the total evaluations, total cycles, and the total time the NN system was run- ning.

To see that the genotype was indeed updated, we can print it out again, to see what the new synaptic weights are for the single neuron of this NN system:

7> genotype:print(test).

...

{neuron,{{0,7.572689688218573e-10},neuron},

0

{{origin,7.572689688224582e-10},cortex},

tanh,

[{{{-1,7.572689688218636e-10},sensor},

[-1.81543903255671,-2.4665070928720794] }],

[{{1,7.572689688218589e-10},actuator}],

[]}

The original synaptic weights associated with the sensor were: [- 1.81543903255671, 0.28220989176010963] which have been tuned to the values: [-1.81543903255671, -2.4665070928720794] . The synaptic weight vector is of length two, and we can see that in this case only the second weight in the vector was perturbed, where as when we applied the mutation operator, it mutated only the first weight in the vector. The mutation and perturbation process is stochastic.

The system passed the test, the mutate_weights operator works, we have manu- ally examined the resulting NN system, which has the right topology, which is the same but with a mutated synaptic weight vector. We have tested the phenotype, and have confirmed that it works. It ran for a total of 52 evaluations, so it made 52 attempts to tune the weights. We can guess that at least 50 did not work, because we know that it takes, due to the MAX_ATTEMPTS = 50 in the exoself module, 50 failing attempts before exoself gives up tuning the weights. We also know that 1 of the evaluations was the very first one, when the NN system ran with the orig- inal genotype. So we can even extrapolate that it was the second attempt, the se- cond evaluation, during which the perturbed synaptic weights were improved in this scenario. When you perform the test, your results will most likely be different.

add_bias: This mutation operator selects a random neuron in the NN and, if the neuron's input_idps list does not already have a bias, the mutation operator adds one.


2> genotype:create_test().

...

{neuron,{{0,7.572678978164637e-10},neuron},

0

{{origin,7.572678978164722e-10},cortex},

gaussian,

[{{{-1,7.572678978164681e-10},sensor},

[0.41211176719508646,0.06709671037415732]}],

[{{1,7.572678978164653e-10},actuator}],

[]}

...

3> genome_mutator:test(test,add_bias).

{atomic,{atomic,ok}}


...

{neuron,{{0,7.572678978164637e-10},neuron},

0

{{origin,7.572678978164722e-10},cortex},

gaussian,

[{{{-1,7.572678978164681e-10},sensor},

[0.41211176719508646,0.06709671037415732]},

{bias,[-0.1437300365267422]}],

[{{1,7.572678978164653e-10},actuator}],

[]}

...

5> exoself:start(test,void).

It works! The original genotype had a neuron connected from the sensor, using a gaussian activation function, with the synaptic weight vector associated with the sensor: [0.41211176719508646, 0.06709671037415732] . After the add_bias mu- tation operator was executed, the neuron acquired the bias weight: [- 0.1437300365267422] . Finally, we now test out the new NN system by converting the genotype to its phenotype by executing the exoself:start(test,void) function. As in the previous test, when I ran it with this mutated agent, there were no errors, and the system terminated normally.

mutate_af: This mutation operator selects a random neuron in the NN and changes its activation function to a new one, selected from the list available in the constraint's neural_afs list.

2> genotype:create_test().

...

{neuron,{{0,7.572652623199229e-10},neuron},

0


{{origin,7.57265262319932e-10},cortex},

absolute ,

[{{{-1,7.572652623199274e-10},sensor},

[-0.16727779071660276,0.12410379914428638]}],

[{{1,7.572652623199246e-10},actuator}],

[]}

...

3> genome_mutator:test(test,mutate_af).

{atomic,{atomic,ok}}


...

{neuron,{{0,7.572652623199229e-10},neuron},

0

{{origin,7.57265262319932e-10},cortex},

cos,

[{{{-1,7.572652623199274e-10},sensor},

[-0.16727779071660276,0.12410379914428638]}],

[{{1,7.572652623199246e-10},actuator}],

[]}

...

{atomic,[ok]}

25> exoself:start(test,void).

...

The original randomly selected activation function of the single neuron in the test agent was the absolute activation function. After we have applied the mu- tate_af operator to the NN system, the activation function was changed to cos . As before, here too converting the genotype to phenotype worked, as there were no errors when running exoself:start(test,void) .

add_outlink & add_inlink: The add_outlink operator chooses a random neu- ron and adds an output connection from it , to another randomly selected element in the NN system. The add_inlink operator chooses a random neuron and adds an input connection to it , from another randomly selected element in the NN. We will only test one of them, the add_outlink, as they both function very similarly.

2> genotype:create_test().

...

{sensor,{{-1,7.572648155161364e-10},sensor},

xor_GetInput,

{{origin,7.572648155161404e-10},cortex},

{private,xor_sim},

2

[{{0,7.572648155161313e-10},neuron}],

undefined}


{neuron,{{0,7.572648155161313e-10},neuron},

0

{{origin,7.572648155161404e-10},cortex},

absolute,

[{{{-1,7.572648155161364e-10},sensor},

[-0.02132967923622686,-0.38581737041377817]}],

[{{1,7.572648155161335e-10},actuator}],

[] }

{actuator,{{1,7.572648155161335e-10},actuator},

xor_SendOutput,

{{origin,7.572648155161404e-10},cortex},

{private,xor_sim},

1

[{{0,7.572648155161313e-10},neuron}],

undefined}

{atomic,{atomic,[ok]}}

3> genome_mutator:test(test,add_outlink).

{atomic,{atomic,ok}}


...

{sensor,{{-1,7.572648155161364e-10},sensor},

xor_GetInput,

{{origin,7.572648155161404e-10},cortex},

{private,xor_sim},

2

[{{0,7.572648155161313e-10},neuron}],

undefined}

{neuron,{{0,7.572648155161313e-10},neuron},

0

{{origin,7.572648155161404e-10},cortex},

absolute,

[{{{0,7.572648155161313e-10},neuron},[-0.13154644819577532]},

{{{-1,7.572648155161364e-10},sensor},

[-0.02132967923622686,-0.38581737041377817]}],

[{{0,7.572648155161313e-10},neuron},

{{1,7.572648155161335e-10},actuator}],

[{{0,7.572648155161313e-10},neuron}]}

{actuator,{{1,7.572648155161335e-10},actuator},

xor_SendOutput,

{{origin,7.572648155161404e-10},cortex},

{private,xor_sim},

1

[{{0,7.572648155161313e-10},neuron}],

undefined}

{atomic,[ok]}


It works! The original neuron had the form: {neuron,{{0,7.572648155161313e-10},neuron},

0

{{origin,7.572648155161404e-10},cortex},

absolute,

[{{{-1,7.572648155161364e-10},sensor},

[-0.02132967923622686,-0.38581737041377817]}],

[{{1,7.572648155161335e-10},actuator}],

[] }

It only had a single input connection which was from the sensor, and a single output connection to the actuator. After the add_outlink operator was executed, the new NN system's neuron had the following form:

{neuron,{{0,7.572648155161313e-10},neuron},

0

{{origin,7.572648155161404e-10},cortex},

absolute,

[{{{0,7.572648155161313e-10},neuron},[-0.13154644819577532]},

{{{-1,7.572648155161364e-10},sensor},

[-0.02132967923622686,-0.38581737041377817]}],

[{{0,7.572648155161313e-10},neuron},

{{1,7.572648155161335e-10},actuator}],

[{{0,7.572648155161313e-10},neuron}]}

In this case the neuron formed a new synaptic connection to another randomly chosen element in the NN system, in this case that other element was itself. We can see that this new connection is recursive, and we can tell this from the last el- ement of the neuron defining tuple, which specifies ro_ids , a list of recurrent link ids. There is also a new synaptic weight associated with this recurrent self connec- tion: {{{0,7.572648155161313e-10},neuron},[-0.13154644819577532]} . The dia- gram of this NN topology before and after the mutation operator was applied, is shown in Fig-9.2 .


Fig. 9.2 The NN system topology before and after add_outlink mutation operator was ap- plied .

We now map the genotype to phenotype, to see if the new NN system is func- tional:

5> exoself:start(test,void).

<0.101.0>

Finished updating genotype

Terminating the phenotype:

It works! Though I did not show the complete printout (which looked very sim- ilar to the first fully shown console printout), the NN system worked and terminat- ed successfully. With this test complete, we now move to a more complex muta- tion operator, the addition of a new random neuron to the existing NN system.

add_neuron: This mutation operator chooses a random neural layer in the NN, and then creates a new neuron and connects it from and to, two randomly selected elements in the NN system respectively.

2> genotype:create_test().

...

{cortex,{{origin,7.572275935869961e-10},cortex},

test,

[{{0,7.572275935869875e-10},neuron}],

[{{-1,7.57227593586992e-10},sensor}],

Chapter 9 Testing the Neuroevolutionary System [{{1,7.572275935869891e-10},actuator}]}

{sensor,{{-1,7.57227593586992e-10},sensor},

xor_GetInput,

{{origin,7.572275935869961e-10},cortex},

{private,xor_sim},

2

[{{0,7.572275935869875e-10},neuron}],

undefined}

{neuron,{{0,7.572275935869875e-10},neuron},

0

{{origin,7.572275935869961e-10},cortex},

cos,

[{{{-1,7.57227593586992e-10},sensor},

[0.43717109366382956,0.33904698258991184]}],

[{{1,7.572275935869891e-10},actuator}],

[]}

{actuator,{{1,7.572275935869891e-10},actuator},

xor_SendOutput,

{{origin,7.572275935869961e-10},cortex},

{private,xor_sim},

1

[{{0,7.572275935869875e-10},neuron}],

undefined}

{atomic,{atomic,[ok]}}

3> genome_mutator:test(test,add_neuron).

{aborted, ”******** ERROR:link_FromNeuronToActuator:: Actuator already fully con- nected ”}

4> genome_mutator:test(test,add_neuron).

{atomic,{atomic,ok}}

5> genotype:print(test).

...

{cortex,{{origin,7.572275935869961e-10},cortex},

test,

[{{0,7.572275884968449e-10},neuron},

{{0,7.572275935869875e-10},neuron}],

[{{-1,7.57227593586992e-10},sensor}],

[{{1,7.572275935869891e-10},actuator}]}

{sensor,{{-1,7.57227593586992e-10},sensor},

xor_GetInput,

{{origin,7.572275935869961e-10},cortex},

{private,xor_sim},

2

[{{0,7.572275935869875e-10},neuron}],

undefined}

{neuron,{{0,7.572275884968449e-10},neuron},


0

{{origin,7.572275935869961e-10},cortex},

gaussian,

[{{{0,7.572275935869875e-10},neuron},[-0.17936473163045719]}],

[{{0,7.572275935869875e-10},neuron}],

[{{0,7.572275935869875e-10},neuron}]}

{neuron,{{0,7.572275935869875e-10},neuron},

0

{{origin,7.572275935869961e-10},cortex},

cos,

[{{{0,7.572275884968449e-10},neuron},[0.2879930434277844]},

{{{-1,7.57227593586992e-10},sensor},

[0.43717109366382956,0.33904698258991184]}],

[{{0,7.572275884968449e-10},neuron},

{{1,7.572275935869891e-10},actuator}],

[{{0,7.572275884968449e-10},neuron}]}

{actuator,{{1,7.572275935869891e-10},actuator},

xor_SendOutput,

{{origin,7.572275935869961e-10},cortex},

{private,xor_sim},

1

[{{0,7.572275935869875e-10},neuron}],

undefined}

{atomic,[ok]}

Something very interesting happened in this test. In “ 2> “ we create a new test NN system. A new NN system is fully connected to its sensors and actuators. When we try to apply the add_neuron mutation operator in “ 3> “, the mutation op- erator must have randomly chosen to connect the new neuron to the existing ac- tuator. But the actuator already has all the connections it needs, the vector signal it uses to execute its functionality, already has all the elements and is already con- nected to all the neurons it requires to function, which in this case is just a single neuron. So the mutation is rejected, as seen by the line: {aborted, ”******** ERROR:link_FromNeuronToActuator:: Actuator already fully connected ”} . During the process of neuroevolution, at this point our topology and weight evolv- ing artificial neural network (TWEANN) system would simply try another muta- tion operator. Which is what I did manually in this test in “ 4> “.

The new mutation worked, it created a new neuron and connected it from and to, the already existing neuron in the NN system. We can see the newly formed connection in the genotype here:

{neuron,{{0,7.572275884968449e-10},neuron},

0

{{origin,7.572275935869961e-10},cortex},


gaussian,

[{{{0,7.572275935869875e-10},neuron},[-0.17936473163045719]}],

[{{0,7.572275935869875e-10},neuron}],

[{{0,7.572275935869875e-10},neuron}]}

{neuron,{{0,7.572275935869875e-10},neuron},

0

{{origin,7.572275935869961e-10},cortex},

cos,

[{{{0,7.572275884968449e-10},neuron},[0.2879930434277844]},

{{{-1,7.57227593586992e-10},sensor},

[0.43717109366382956,0.33904698258991184]}],

[{{0,7.572275884968449e-10},neuron},

{{1,7.572275935869891e-10},actuator}],

[{{0,7.572275884968449e-10},neuron}]}

The initial test NN system had a single neuron with the id: {{0,7.572275935869875e-10},neuron} , The newly added neuron has the id: {{0,7.572275884968449e-10},neuron} . We can see that after the mutation, both neurons have recurrent connections, which in our neuron record is represented by the last list in the tuple. The original neuron's recurrent connection list ro_ids is: [{{0,7.572275884968449e-10},neuron}] , containing the id of the new neuron. The newly added neuron's or_ids list is: [{{0,7.572275935869875e-10},neuron}] , con- taining in it the id of the original neuron.

Fig. 9.3 The NN system topology before and after the add_neuron mutation operator was applied.


We can also see that the new neuron is using the gaussian activation function, and that both of the neurons formed new weights for their new synaptic connec- tions. The above figure shows the NN system's topology before and after the add_neuron mutation operator is applied.

We now test the new topology live, by mapping the genotype to its phenotype: 6> exoself:start(test,void).

<0.866.0>

Finished updating genotype

Terminating the phenotype:

Cx_PId:<0.868.0>

SPIds:[<0.869.0>]

NPIds:[<0.871.0>,<0.872.0>]

APIds:[<0.870.0>]

ScapePids:[<0.867.0>]

Sensor:{{-1,7.57227593586992e-10},sensor} is terminating.

Agent:<0.866.0> terminating. Genotype has been backed up.

Fitness:1.3179457789331406

TotEvaluations:163

TotCycles:656

TimeAcc:23321

Cortex:{{origin,7.572275935869961e-10},cortex} is terminating.

It works! And from the highlighted NPIds, we can see the two spawned neuron PIds. The system terminated successfully, the topology we analyzed manually is correct given the mutation operator, and the phenotype works perfectly. Thus this mutation operator is functional, at least in this simple test, and we move on to the next one.

outsplice: This mutation operator selects a random neuron A in the NN, then selects the neuron's random output connection to some element B, disconnects A from B, creates a new neuron C in the layer between neuron A and element B (creating the new layer if it does not already exist, or using an existing one if A and B are one or more layers apart), and then reconnects A to B through C:

2> genotype:create_test().

...

{cortex,{{origin,7.57225527862836e-10},cortex},

test,

[{{0,7.572255278628331e-10},neuron}],

[{{-1,7.572255278628343e-10},sensor}],

[{{1,7.572255278628337e-10},actuator}]}

{sensor,{{-1,7.572255278628343e-10},sensor},

xor_GetInput,

{{origin,7.57225527862836e-10},cortex},

Chapter 9 Testing the Neuroevolutionary System {private,xor_sim},

2

[{{0,7.572255278628331e-10},neuron}],

undefined}

{neuron,{{0,7.572255278628331e-10},neuron},

0

{{origin,7.57225527862836e-10},cortex},

tanh,

[{{{-1,7.572255278628343e-10},sensor},

[0.4094174115111171,0.40477840576669655]}],

[{{1,7.572255278628337e-10},actuator}],

[]}

{actuator,{{1,7.572255278628337e-10},actuator},

xor_SendOutput,

{{origin,7.57225527862836e-10},cortex},

{private,xor_sim},

1

[{{0,7.572255278628331e-10},neuron}],

undefined}

{atomic,{atomic,[ok]}}

3> genome_mutator:test(test,outsplice).

{atomic,{atomic,ok}}


...

{cortex,{{origin,7.57225527862836e-10},cortex},

test,

[{{0.5,7.572255205521553e-10},neuron},

{{0,7.572255278628331e-10},neuron}],

[{{-1,7.572255278628343e-10},sensor}],

[{{1,7.572255278628337e-10},actuator}]}

{sensor,{{-1,7.572255278628343e-10},sensor},

xor_GetInput,

{{origin,7.57225527862836e-10},cortex},

{private,xor_sim},

2

[{{0,7.572255278628331e-10},neuron}],

undefined}

{neuron,{{0.5,7.572255205521553e-10},neuron},

0

{{origin,7.57225527862836e-10},cortex},

absolute,

[{{{0,7.572255278628331e-10},neuron},[0.08385901270641671]}],

[{{1,7.572255278628337e-10},actuator}],

[]}

{neuron,{{0,7.572255278628331e-10},neuron},


0

{{origin,7.57225527862836e-10},cortex},

tanh,

[{{{-1,7.572255278628343e-10},sensor},

[0.4094174115111171,0.40477840576669655]}],

[{{0.5,7.572255205521553e-10},neuron}],

[]}

{actuator,{{1,7.572255278628337e-10},actuator},

xor_SendOutput,

{{origin,7.57225527862836e-10},cortex},

{private,xor_sim},

1

[{{0.5,7.572255205521553e-10},neuron}],

0}

{atomic,[ok]}

It works! The genotype:create_test() function created the genotype of a simple test NN system, with a single neuron:

{neuron,{{0,7.572255278628331e-10},neuron},

0

{{origin,7.57225527862836e-10},cortex},

tanh,

[{{{-1,7.572255278628343e-10},sensor},

[0.4094174115111171,0.40477840576669655]}],

[{{1,7.572255278628337e-10},actuator}],

[]}

Which is connected from the sensor: {{-1,7.572255278628343e-10},sensor} and is connected to the actuator: {{1,7.572255278628337e-10},actuator} . From the neuron's Id, we can see that it is in layer 0. After we executed the outsplice mutation operator, our NN system acquired a new neuron, thus the NN now had two neurons:

{neuron,{{0.5,7.572255205521553e-10},neuron},

0

{{origin,7.57225527862836e-10},cortex},

absolute,

[{{{0,7.572255278628331e-10},neuron},[0.08385901270641671]}],

[{{1,7.572255278628337e-10},actuator}],

[]}

{neuron,{{0,7.572255278628331e-10},neuron},

0

{{origin,7.57225527862836e-10},cortex},

tanh,


[{{{-1,7.572255278628343e-10},sensor},

[0.4094174115111171,0.40477840576669655]}],

[{{0.5,7.572255205521553e-10},neuron}],

[]}

Note that where as in the initial genotype the NN was composed of a single neuron: {{0,7.572255278628331e-10}, neuron} , which was connected from the sensor: {{-1,7.572255278628343e-10}, sensor} , and connected to the actuator: {{1,7.572255278628337e-10}, actuator} , after the mutation operator was applied, the NN acquired a new neuron, which was inserted into a new layer 0.5 (we de- termine that fact from its Id, which contains the layer index specification). Also note that the original neuron is no longer connected to the actuator, but instead is connected to the new neuron: {{0.5,7.572255205521553e-10},neuron} , which is now the one connected to the actuator. The diagram of the before and after topol- ogy of this NN system is shown in Fig-9.4 .

Fig. 9.4 The NN System topology before and after the outsplice mutation operator is ap- plied to it .

Let's test this NN system by mapping its genotype to its phenotype, and apply- ing it to the problem that its morphology defines (mimicking the XOR operator):

5> exoself:start(test,void).

<0.919.0>

Finished updating genotype

Terminating the phenotype:

Cx_PId:<0.921.0>

SPIds:[<0.922.0>]

NPIds:[<0.924.0>,<0.925.0>]


APIds:[<0.923.0>]

ScapePids:[<0.920.0>]

Agent:<0.919.0> terminating. Genotype has been backed up. Fitness:0.5311848171954074

TotEvaluations:58

TotCycles:236

TimeAcc:7384

Cortex:{{origin,7.57225527862836e-10},cortex} is terminating.

Sensor:{{-1,7.572255278628343e-10},sensor} is terminating.

It works! And we can also see that there are two NPIds, since there are now two neurons. We have visually inspected the NN system genotype before and after the mutation operator was applied, and found the new genotype to be correct. We have also tested the phenotype, to ensure that it is functional, and confirmed that it is. We next test the two last remaining mutation operators: add_sensor and add_actuator.

add_sensor & add_actuator: The add_sensor mutation operator adds a new random sensor, still unused by the NN system. The sensor is chosen from the sen- sor list available to the morphology of the NN based agent. A random neuron in the NN is then chosen, and the sensor is connected to that neuron. The add_actuator mutation operator adds a new random actuator, still unused by the NN system. A random neuron in the NN is then chosen, and a link is established between this neuron and the new actuator.

2> genome_mutator:test(test,add_sensor).

{aborted, ”********ERROR:add_sensor(Agent_Id):: NN system is already using all available sensors ”}

3> genome_mutator:test(test,add_actuator).

{aborted, ”********ERROR:add_actuator(Agent_Id):: NN system is already using all available actuators ”}

This is as expected. The test NN system uses the xor_mimic morphology, and if we look in the morphology module, we see that it only has one sensor and one actuator. Thus, when we run the mutation operators for this particular test, our neuroevolutionary system does not add a new sensor, or a new actuator, because there are no new ones available. When we begin expanding the neuroevolutionary platform we're designing here, we will see the affects of a system that can incor- porate new sensors and actuators into itself as it evolves. We can similarly test the mutation operators: add_sensorlink & add_actuatorlink, but just as the above two mutation operators, they have no new elements to connect to and from, respective- ly, when it comes to the seed NN.

We have now successfully tested most of the complexifying mutation operators on the simple, seed NN based agent. But this does not necessarily mean that there are no bugs in our system. Perhaps there are scenarios when it does fail, we just


haven't come across them yet because we've only tested the operators on the most simple type of topology, the single neuron NN system topology.

Before we proceed, let's create a small program that applies X random muta- tion operators to the test NN system, and then converts the mutated genotype to its phenotype, to ensure that it still functions. The goal here is to ensure that the re- sulting NN is simply connected, and does not crash, or stall during operation. Fur- thermore, we can run this mutation operator test itself, a few thousand times. If at any point it gets stuck, or there is an unexpected error, we can then try to figure out what happened.

The following listing shows this simple, topological mutation testing function that we add to the genome_mutator module:

Listing-9.1 The long_test/1 function, which creates a seed agent, and applies TotMutateApplications number of mutation operators to it, and tests the resulting phenotype af- terwards.

long_test(TotMutateApplications) when (TotMutateApplications > 0) ->

genotype:create_test(),

short_test(TotMutateApplications).

short_test(0)->

exoself:start(test,void);

short_test(Index)->

test(),

short_test(Index-1).

%This is a simple function that executes the test() function the number of times with which the long_test/1 function was initially called. The test/0 function executes mutate(test), which ap-

plies a random number of mutation operators to the genotype, where that number ranges from 1

to sqrt(Tot_neurons). After all the mutation operators have been applied successfully, the func- tion executes exoself:start(test,void), mapping the genotype to phenotype, to test whether the resulting NN system is functional.

The long_test/1 function will perform the following steps:

1. Create a test genotype.

2. Execute the mutate(test) function TotMutateApplications number of times.

3. Convert the genotype to phenotype to ensure that the resulting NN system is functional.

Lets run the long_test function with TotMutateApplications = 300 . For the sake of being brief, I will only present the first and last few lines of the printout to con- sole in the following Listing-9.2.


Listing-9.2 Running the long_test function, which applies a random number of mutation opera- tors to the original seed agent, 300 times.

2>genome_mutator:long_test(300).

{agent,test,0,undefined,test,

{{origin,7.571534416338085e-10},cortex},

{[{0,1}],

[],

[{sensor,undefined,xor_GetInput,undefined,

{private,xor_sim},

2

[{{0,7.571534416338051e-10},neuron}],

undefined}],

[{actuator,undefined,xor_SendOutput,undefined,

{private,xor_sim},

1

[{{0,7.571534416338051e-10},neuron}],

undefined}]},

{constraint,xor_mimic,[tanh,cos,gaussian,absolute]},

[],undefined,0,

[{0,[{{0,7.571534416338051e-10},neuron}]}]}

Tot neurons:1 Performing Tot mutations:1 on:test

Mutation Operator:add_outlink

                • Mutation Succesful.

Tot neurons:1 Performing Tot mutations:1 on:test

Mutation Operator:add_actuator

                • Error:{aborted, ”********ERROR:add_actuator(Agent_Id):: NN system is already using all available actuators ”}

Retrying with new Mutation...

Mutation Operator:outsplice

                • Mutation Succesful.

Tot neurons:2 Performing Tot mutations:1 on:test

Mutation Operator:mutate_af

                • Mutation Succesful.

...

Tot neurons:95 Performing Tot mutations:5 on:test

Mutation Operator:outsplice

Mutation Operator:add_bias

Mutation Operator:mutate_weights

Mutation Operator:add_outlink

Mutation Operator:mutate_af

                • Mutation Succesful.

<0.2460.0>

Finished updating genotype


Terminating the phenotype:

Cx_PId:<0.2463.0>

SPIds:[<0.2464.0>]

NPIds:[<0.2467.0>,<0.2468.0>,<0.2469.0>,<0.2470.0>,<0.2471.0>,<0.2472.0>,<0.2473.0>,

<0.2474.0>,<0.2475.0>,<0.2476.0>,<0.2477.0>,<0.2478.0>,<0.2479.0>,<0.2480.0>,

<0.2481.0>,<0.2482.0>,<0.2483.0>,<0.2484.0>,<0.2485.0>,<0.2486.0>,<0.2487.0>,

<0.2488.0>,<0.2489.0>,<0.2490.0>,<0.2491.0>,<0.2492.0>,<0.2493.0>,<0.2494.0>,

<0.2495.0>,<0.2496.0>,<0.2497.0>,<0.2498.0>,<0.2499.0>,<0.2500.0>,<0.2501.0>,

<0.2502.0>,<0.2503.0>,<0.2504.0>,<0.2505.0>,<0.2506.0>,<0.2507.0>,<0.2508.0>,

<0.2509.0>,<0.2510.0>,<0.2511.0>,<0.2512.0>,<0.2513.0>,<0.2514.0>,<0.2515.0>,

<0.2516.0>,<0.2517.0>,<0.2518.0>,<0.2519.0>,<0.2520.0>,<0.2521.0>,<0.2522.0>,

<0.2523.0>,<0.2524.0>,<0.2525.0>,<0.2526.0>,<0.2527.0>,<0.2528.0>,<0.2529.0>,

<0.2530.0>,<0.2531.0>,<0.2532.0>,<0.2533.0>,<0.2534.0>,<0.2535.0>,<0.2536.0>,

<0.2537.0>,<0.2538.0>,<0.2539.0>,<0.2540.0>,<0.2541.0>,<0.2542.0>,<0.2543.0>,

<0.2544.0>,<0.2545.0>,<0.2546.0>,<0.2547.0>,<0.2548.0>,<0.2549.0>,<0.2550.0>,

<0.2551.0>,<0.2553.0>,<0.2554.0>,<0.2555.0>,<0.2556.0>,<0.2557.0>,<0.2558.0>,

<0.2559.0>,<0.2560.0>,<0.2561.0>,<0.2562.0>,<0.2563.0>]

APIds:[<0.2465.0>,<0.2466.0>]

ScapePids:[<0.2461.0>,<0.2462.0>]

Sensor:{{-1,7.57153413903982e-10},sensor} is terminating.

Agent:<0.2460.0> terminating. Genotype has been backed up.

Fitness:0.5162814284277237

TotEvaluations:65

TotCycles:132

TimeAcc:21664

Cortex:{{origin,7.571534139039844e-10},cortex} is terminating.

From the above console printout, you can see that the first mutation operator applied was the add_outlink, which was successful. The second was add_actuator, which was not. At this stage, every time the mutate(test) gets executed, the func- tion only applies a single mutation operator to the genotype, we know this from the line: Tot neurons:1 Performing Tot mutations:1 on:test . We then skip to the end, the last execution of the mutate(test). From the line: Tot neurons:95 Per- forming Tot mutations:5 on:test , we can see that at this point the NN system has 95 neurons, and the randomly chosen number of mutation operators to be applied is 5. This means that 5 mutation operators are applied in series to the NN system to produce the mutant agent, and only after the 5 mutation operators are applied, is the agent's fitness evaluated.

Once all the mutation operators have been applied, the exoself converts the genotype of the test NN system to its phenotype, applying it to the problem that its morphology designated it for. From the console printout, we see that the NN sys- tem successfully terminated, and so we can be assured that the NN topology does not have any discontinuities, and that it does produce a functional, albeit not very

9.2 Testing the Neuroevolutionary System on the Simple XOR Benchmark

fit, phenotype. Also, none of the mutation operators produced any type of errors that originate from actual crashes.

Having now tested the main mutation operators and the mapping from geno- type to phenotype, we can move on and see if the population_monitor is function- al, by running the small XOR based benchmark, as we did in Chapter-7.


Having now tested some of the important independent functions and elements of our topology and weight evolving artificial neural network (TWEANN) system, we can move on to testing the system as a whole. Our morphology module con- tains various morphologies at our disposal, where a morphology is a list of sensors and actuators that a NN system can incorporate through evolution if it is of that particular morphology. Furthermore, the sensors and actuators define what the NN system can interface with, what the NN system does, and thus, what the problems the NN system is applied to. For example, if the sensor available to our NN sys- tem is one that reads values from a database, and the actuator is one that simply outputs the NN's output vector signal, and furthermore the database from which the sensor reads its data is a XOR truth table, then we could train this NN system to mimic a XOR logic operator. We could compare the NN based agent's output to what that output should be if the agent was a XOR logic operator, rewarding it if it's output is similar to the expected XOR operator output, and punishing it if not.

If the sensors were to have been programs that interfaced with a simulated world through sensors embedded in some simulated organism inhabiting a simu- lated world, and if the actuators were to have been programs controlling the simu- lated organism (avatar), then our NN system would be the evolving brain of an or- ganism in an Artificial Life experiment. Thus, the sensors and actuators define what the NN system does, and its morphology is a set of sensors and actuators, as a package, available to the NN system during its evolution. Thus it is the mor- phology that defines the problem to what the NN system is applied. We choose a morphology to which the NN system belongs, and it evolves and learns how to use the sensors and actuators belonging to that morphology.

Thus far we have only created one morphology, the xor_mimic . The xor_mimic morphology contains a single sensor with the name xor_GetInput , and a single ac- tuator with the name xor_SendOutput . Thus if we evolve agents of this particular morphology, they will only be able to evolve into XOR logical operator mimics. Agents cannot switch morphologies mid-evolution, but new sensors and actuators can be added to the morphology by updating the morphology module, and after- wards these new interfaces can then be incorporated into the NN system over time.


We created the population_monitor process which creates a seed population of NN systems belonging to some specified morphologies, and then evolves those NN based agents. Since the morphologies define the scapes the NN system inter- faces with, and the scape computes the fitness score of the agent interfacing with it, the population_monitor process has the ability to evolve the population by hav- ing access to each agent's fitness in the population, applying a selection function to the population, and then mutating the selected agents, creating new and mutated offspring from them. We now test this process by getting the population_monitor process to spawn a seed population of agents with the xor_mimic morphology, and see how quickly our current version of the neuroevolutionary system can evolve a solution to this problem, how quickly it can evolve a XOR logic operator using neurons as the basic elements of the evolving network.

We will run the population_monitor:test() function with the following parame- ters:

%%%%%%%%%%%%%% Population Monitor Options & Parameters %%%%%%%%%%% -define(SELECTION_ALGORITHM,competition).

-define(EFF,0.2).

-define(INIT_CONSTRAINTS,[#constraint{morphology=Morphology, neu-

ral_afs=Neural_AFs}|| Morphology<-[xor_mimic],Neural_AFs<-tanh]).

-define(SURVIVAL_PERCENTAGE,0.5).

-define(SPECIE_SIZE_LIMIT,10).

-define(INIT_SPECIE_SIZE,10).

-define(INIT_POPULATION_ID,test).

-define(OP_MODE,gt).

-define(INIT_POLIS,mathema).

-define(GENERATION_LIMIT,100).

-define(EVALUATIONS_LIMIT,100000).

-define(DIVERSITY_COUNT_STEP,500).

-define(GEN_UID,genotype:generate_UniqueId()).

-define(CHAMPION_COUNT_STEP,500).

-define(FITNESS_GOAL,inf).

The population will thus be composed of NN systems using the xor_mimic morphology (and thus be applied to that particular problem), and whose neurons will use only the tanh activation function. The population will maintain a size close to 10. Finally, neuroevolution will continue for at most 100 generations, or at most 100000 evaluations. The fitness goal is set to inf, which means that it is not a stopping condition and the evolution will continue until one of the other ter- minating conditions is reached. The fitness score for each agent is calculated by the scape it is interfacing with. Having set up the parameters for our neuroeovlutionary system, we compile the population_monitor module, and exe- cute the population_monitor:test() function, as shown next:


2> population_monitor:test().

Specie_Id:7.570104741922324e-10 Morphology:xor_mimic

                • Population monitor started with parameters:{gt,test,competition}

Selection Algorirthm:competition

Valid_AgentSummaries:[{91822.42396111514,3,{7.570065786458927e-10,agent}},

{82128.75594984594,3,{7.570065785419657e-10,agent}},

{66717.38827549343,3,{7.570065785184491e-10,agent}},

{66865.26402662563,4,{7.570065786995862e-10,agent}},

{66859.35543290272,4,{7.570065785258691e-10,agent}},

{60974.864233884604,4,{7.570065785388116e-10,agent}}]

Invalid_AgentSummaries:[{56725.927279906005,4,{7.570065787547878e-10,agent}},

{46423.91939090131,4,{7.570065786090063e-10,agent}},

{34681.35604691528,3,{7.570065790439459e-10,agent}},

{67.37546054504678,4,{7.570065785110257e-10,agent}},

{13.178830126581289,5,{7.570065785335377e-10,agent}}]

NeuralEnergyCost:13982.434363128335

NewPopAcc:9.218546902348272

Population size normalizer:0.9218546902348272

Agent_Id:{7.570065785388116e-10,agent} Normalized_MutantAlotment:1

Agent_Id:{7.570065785258691e-10,agent} Normalized_MutantAlotment:1

Agent_Id:{7.570065786995862e-10,agent} Normalized_MutantAlotment:1

Agent_Id:{7.570065785184491e-10,agent} Normalized_MutantAlotment:2

...

                • Population_Monitor:test shut down with Reason:normal OpTag:continue, while in OpMode:gt
                • Tot Agents:9 Population Generation:100 Eval_Acc:63960 Cycle_Acc:217798 Time_Acc:12912953

It works! Highlighted in green (2nd and 3rd line in the black & white printed version) are the first two lines printed to screen after population monitor:test() is executed. It states that the population_monitor is started with selection algorithm competition , a population with the id test , and op_mode (operational mode) being gt, whose operational importance we will set in a later chapter.

Based on how we designed our population_monitor system, every generation it prints out the fitness score of the population. Highlighted in red and italicized is the 100 generation, and each agent with its fitness score. The most fit agent with its fitness score in the last generation is: {91822.42396111514, 3, {7.570065786458927e-10, agent}} . Based on how the xor_sim scape calculates fitness, this fitness score amounts to the agent having a mean squared sum error of 1/91822, and it took a total of 63960 evaluations for our neuroevolutionary system to reach it.

th


This is quite a bit of computational time for such a simple problem, but it is not usually the case to take the circuit to this level of accuracy. Let us change the fit- ness goal to 1000, make MAX_ATTEMPTS = 10 in the exoself module, and then try again.

In my experiment, I had the following results:

Valid_AgentSummaries:[{1000.4594763865106,2,{7.570051345044739e-10,agent}},

{272.7339484226029,2,{7.570051345273578e-10,agent}},

{249.64913390960575,2,{7.57005134500996e-10,agent}},

{227.82980202627456,4,{7.570051345098297e-10,agent}},

{193.32888692741093,2,{7.570051345440797e-10,agent}}]

Invalid_AgentSummaries:[{56.2580273824466,2,{7.570051346068126e-10,agent}},

{18.43287953405122,2,{7.570051345575052e-10,agent}},

{6.1532819188772505,2,{7.570051345123884e-10,agent}},

{0.49999782678670823,3,{7.570051345394602e-10,agent}}]

                • Population_Monitor:test shut down with Reason:normal OpTag:continue, while in OpMode:gt
                • Tot Agents:9 Population Generation:78 Eval_Acc:10701 Cycle_Acc:41178 Time_Acc:2259258

This time it took only 10701 evaluations. But there is something very interest- ing that happened here. Take a look at the most fit agent in the population, with the id: {1000.4594763865106,2,{7.570051345044739e-10,agent}} . It only has 2 neurons! That's not possible, since this particular circuit requires at least 3 neu- rons, if those neurons are using tanh activation function. We have the agent's Id, let's check out its topology, as shown in the following listing:

Listing-9.3 The console printout of the topology of the fittest agent in the population.

3> genotype:print({7.570051345044739e-10,agent}).

{agent,{7.570051345044739e-10,agent},

15,undefined,7.570051363681182e-10,

{{origin,7.570051345042693e-10},cortex},

{[{0,1},{0.5,1}],

[{add_bias,{0.5,neuron}},

{mutate_af,{0,neuron}},

{mutate_weights,{0.5,neuron}},

{add_actuator,{0,neuron},{1,actuator}},

{outsplice,{0,neuron},{0.5,neuron},{1,actuator}},

{mutate_weights,{0,neuron}},

{mutate_af,{0,neuron}},

9.2 Testing the Neuroevolutionary System on the Simple XOR Benchmark {mutate_af,{0,neuron}},

{mutate_weights,{0,neuron}},

{mutate_af,{0,neuron}},

{mutate_weights,{0,neuron}},

{mutate_af,{0,neuron}},

{mutate_weights,{0,neuron}},

{add_bias,{0,neuron}},

{add_inlink,{0,neuron},{0,neuron}}],

[{sensor,undefined,xor_GetInput,undefined,

{private,xor_sim},

2

[{{0,7.570051345042682e-10},neuron}],

undefined}],

[{actuator,undefined,xor_SendOutput,undefined,

{private,xor_sim},

1

[{{0.5,7.570051345042677e-10},neuron}],

11},

{actuator,undefined,xor_SendOutput,undefined,

{private,xor_sim},

1

[{{0,7.570051345042682e-10},neuron}] ,

undefined}]},

{constraint,xor_mimic,[tanh]},

[ {add_bias,{{0.5,7.570051345042677e-10},neuron}},

{mutate_af,{{0,7.570051345439552e-10},neuron}},

{mutate_weights,{{0.5,7.570051346065783e-10},neuron}},

{add_actuator,{{0,7.57005134638638e-10},neuron},

{{1,7.57005134636634e-10},actuator}},

{outsplice,{{0,7.57005134670089e-10},neuron},

{{0.5,7.570051346689715e-10},neuron},

{{1,7.570051346700879e-10},actuator}},

{mutate_weights,{{0,7.570051347808065e-10},neuron}},

{mutate_af,{{0,7.570051347949999e-10},neuron}},

{mutate_af,{{0,7.570051348731883e-10},neuron}},

{mutate_weights,{{0,7.57005134905699e-10},neuron}},

{mutate_af,{{0,7.570051352005185e-10},neuron}},

{mutate_weights,{{0,7.57005135384367e-10},neuron}},

{mutate_af,{{0,7.570051357421974e-10},neuron}},

{mutate_weights,{{0,7.570051357953169e-10},neuron}},

{add_bias,{{0,7.570051361212367e-10},neuron}},

{add_inlink,{{0,7.570051363350866e-10},neuron},

{{0,7.570051363350866e-10},neuron}}],

1000.4594763865106,0,

[{0,[{{0,7.570051363631578e-10},neuron}]},


{0.5,[{{0.5,7.570051346689715e-10},neuron}]}]}

{cortex,{{origin,7.570051345042693e-10},cortex},

{7.570051345044739e-10,agent},

[{{0.5,7.570051345042677e-10},neuron},

{{0,7.570051345042682e-10},neuron}],

[{{-1,7.570051345042671e-10},sensor}],

[{{1,7.570051345042659e-10},actuator},

{{1,7.570051345042664e-10},actuator}]}

{sensor,{{-1,7.570051345042671e-10},sensor},

xor_GetInput,

{{origin,7.570051345042693e-10},cortex},

{private,xor_sim},

2

[{{0,7.570051345042682e-10},neuron}],

undefined}

{neuron,{{0.5,7.570051345042677e-10},neuron},

15

{{origin,7.570051345042693e-10},cortex},

tanh,

[{{{0,7.570051345042682e-10},neuron},[-4.9581978771372395]},

{bias,[-2.444318048832683]}],

[{{1,7.570051345042659e-10},actuator}],

[]}

{neuron,{{0,7.570051345042682e-10},neuron},

14

{{origin,7.570051345042693e-10},cortex},

tanh,

[{{{0,7.570051345042682e-10},neuron},[6.283185307179586]},

{{{-1,7.570051345042671e-10},sensor},

[-4.3985975891263305,-2.3223009779757877]},

{bias,[1.3462974501315348]}],

[{{1,7.570051345042664e-10},actuator},

{{0.5,7.570051345042677e-10},neuron},

{{0,7.570051345042682e-10},neuron}],

[{{0,7.570051345042682e-10},neuron}]}

{actuator,{{1,7.570051345042659e-10},actuator},

xor_SendOutput,

{{origin,7.570051345042693e-10},cortex},

{private,xor_sim},

1

[{{0.5,7.570051345042677e-10},neuron}],

11}

{actuator,{{1,7.570051345042664e-10},actuator},

xor_SendOutput,

{{origin,7.570051345042693e-10},cortex},

9.2 Testing the Neuroevolutionary System on the Simple XOR Benchmark {private,xor_sim},

1

[{{0,7.570051345042682e-10},neuron}],

undefined}

{atomic,[ok,ok]}

Though we've decided to look at the NN system's genotype to see how it was possible for our neuroevolutionary system to evolve a solution with only two neu- rons, instead, if you look through the genotype, you will see that we just uncov- ered a large number of errors in the way our system functions. Let's take a look at each part in turn, before returning to the actual evolved topology of the NN sys- tem.

Boldfaced in the console printout above are the following errors, discussed and corrected in the following sections:

1.

2.

3.

4.

mutate_af operator is applied to the agent multiple times, but we have opted to only use the tanh activation function, which means this mutation operator does nothing to the network, and is a waste of a mutation attempt, and thus should not be present.

When looking at the mutate_af, we also see that it is applied to neurons with different Ids, 5 of them, even though there are only 2 neurons in the system. This NN system evolved a connection to two actuators, but this morphology supports only 1, what happened?

In the agent's fingerprint, the sensors and actuators contain N_Ids. This is an error, since the fingerprint must not contain any Id specific information, it must only contain the general information about the NN system, so that we can have an ability to roughly distinguish between different species of the NN systems (those with different topologies, morphologies, sensors and actuators, or those with significantly different sets of activation functions).

In the following sections, we deal with each of these errors one at a time.

9.2.1 The mutate_af Error

Looking at the agent's evo_hist list, shown in Listing-9.4, we can see that mul- tiple mutate_afs are applied. The goal of a mutation operator is to modify the NN system, and if a mutation operator cannot be applied, due to for example the state in which the NN system is, or because it leads to a non-functional NN, then we should revert the mutation operator and try applying another one. Each NN sys- tem, when being mutated, undergoes a specific number of mutations, ranging from 1 to sqrt(Tot_Neurons), chosen randomly. Thus, every time we apply a mutation operator to the NN system, and it does nothing, that is one mutation attempt wast- ed. This can result in a clone which was not mutated at all, or not mutated properly.


Afterwards, this clone is sent back into the environment to be evaluated. For example assume a fit agent creates an offspring by first creating a clone of itself, and then applying to it the mutate_af operator, if mutate_af is being applied to an agent that only has tanh for its available activation functions list, the resulting off- spring is exactly the same as its parent, since tanh was swapped for tanh. There is no reason to test out a clone, since we already know how such a NN system func- tions, because its parent has already been evaluated and tested for fitness. It is thus essential that whatever is causing this error, is fixed.

Listing-9.4 The agent's evo_hist list.

[{add_bias,{{0.5,7.570051345042677e-10},neuron}},

{mutate_af,{{0,7.570051345439552e-10},neuron}},

{mutate_weights,{{0.5,7.570051346065783e-10},neuron}},

{add_actuator,{{0,7.57005134638638e-10},neuron},

{{1,7.57005134636634e-10},actuator}},

{outsplice,{{0,7.57005134670089e-10},neuron},

{{0.5,7.570051346689715e-10},neuron},

{{1,7.570051346700879e-10},actuator}},

{mutate_weights,{{0,7.570051347808065e-10},neuron}},

{mutate_af,{{0,7.570051347949999e-10},neuron}},

{mutate_af,{{0,7.570051348731883e-10},neuron}},

{mutate_weights,{{0,7.57005134905699e-10},neuron}},

{mutate_af,{{0,7.570051352005185e-10},neuron}},

{mutate_weights,{{0,7.57005135384367e-10},neuron}},

{mutate_af,{{0,7.570051357421974e-10},neuron}},

{mutate_weights,{{0,7.570051357953169e-10},neuron}},

{add_bias,{{0,7.570051361212367e-10},neuron}},

{add_inlink,{{0,7.570051363350866e-10},neuron},

{{0,7.570051363350866e-10},neuron}}],

To solve this problem we need to check the genome_mutator:mutate_af/1 func- tion, as shown in listing-9.5.

Listing-9.5 The mutate_af/1 function.

mutate_af(Agent_Id)->

A = genotype:read({agent,Agent_Id}),

Cx_Id = A#agent.cx_id,

Cx = genotype:read({cortex,Cx_Id}),

N_Ids = Cx#cortex.neuron_ids,

N_Id = lists:nth(random:uniform(length(N_Ids)),N_Ids),

Generation = A#agent.generation,

N = genotype:read({neuron,N_Id}),

AF = N#neuron.af,


Activation_Functions = (A#agent.constraint)#constraint.neural_afs -- [AF],

NewAF = genotype:generate_NeuronAF(Activation_Functions),

U_N = N#neuron{af=NewAF,generation=Generation},

EvoHist = A#agent.evo_hist,

U_EvoHist = [{mutate_af,N_Id}|EvoHist],

U_A = A#agent{evo_hist=U_EvoHist},

genotype:write(U_N),

genotype:write(U_A).

Though: Activation_Functions = (A#agent.constraint)#constraint.neural_afs – [AF] , does result in an empty list (since #constraint.neural_afs list is: [tanh]), it does not matter because the genotype:generate_NeuronAF(Activation_Functions) function itself chooses the default tanh activation function when executed with an empty list parameter. This is the cause of this error. What we need to do is simply exit the mutation operator as soon as we find that there is only one activation func- tion, that it is already being used by the neuron, and that there is nothing to mu- tate. We thus modify mutate_af/1 function to be as follows:

Listing-9.6 The mutate_af function after the fix is applied.

mutate_af(Agent_Id)->

A = genotype:read({agent,Agent_Id}),

Cx_Id = A#agent.cx_id,

Cx = genotype:read({cortex,Cx_Id}),

N_Ids = Cx#cortex.neuron_ids,

N_Id = lists:nth(random:uniform(length(N_Ids)),N_Ids),

Generation = A#agent.generation,

N = genotype:read({neuron,N_Id}),

AF = N#neuron.af,

case (A#agent.constraint)#constraint.neural_afs -- [AF] of

[] ->

exit( “********ERROR:mutate_af:: There are no other activation func-

tions to use. ”);

Activation_Functions ->

NewAF = lists:nth(random:uniform(length(Activation_Functions)),

Activation_Functions),

U_N = N#neuron{af=NewAF,generation=Generation},

EvoHist = A#agent.evo_hist,

U_EvoHist = [{mutate_af,N_Id}|EvoHist],

U_A = A#agent{evo_hist=U_EvoHist},

genotype:write(U_N),

genotype:write(U_A)

end.


The fix is shown in boldface. In this fixed function, as soon as the mutation op- erator determines that there are no other activation functions that it can swap the currently used one for, it simply exits with an error. The genome mutator then tries out another mutation operator.

9.2.2 Same Neuron, But Different Ids in the evo_hist List

Looking at the Listing-9.3 again, we also see that even though the NN has only 2 neurons, as was shown in the original printout to console, there were 5 mu- tate_af operators applied and each one was applied to a different neuron_id. But how is that possible if there are only 2 neurons and thus only 2 different neuron ids?

This error occurs because when we clone the NN, all neurons get a new id, but we never update the evo_hist list, converting those old ids into new ones. This means that the Ids within the evo_hist are not of the elements belonging to the agent in its current state, but the element ids which belong to its ancestors. Though it does not matter what the particular ids are, it is essential that they are consistent, so that we can reconstruct the evolutionary path of the NN based system, which is not possible if we don't know which mutation operator was applied to which ele- ment in the NN system being analyzed. To be able to see when, and to what par- ticular elements of the topology the mutation operators were applied, we need a consistent set of element ids in the evo_hist, so that the evolutionary path can be reconstructed based on the actual ids used by the NN based agent.

To fix this, we need to modify the cloning process so that it does not only up- date all the element ids in the NN system, but also the element ids in the evo_hist, ensuring that the system is consistent. The cloning process is performed in the genotype module, through the clone_Agent/2 function. Therefore, it is this func- tion that we need to correct. The fix is simple, we need to create a new function called map_EvoHist/2, and call it from the clone_Agent/2 function with the old evo_hist list and an ETS table containing a map from old ids to new ones. The map_EvoHist/2 function can then map the old ids to new ids in the evo_hist list. The cloned agent will then use this updated evo_hist, with its updated new ids, in- stead of the old ids which belonged to its parent. The updated map_EvoHist/2 function is shown in Listing-9.7.

Listing-9.7 A new function, map_EvoHist/2, which updates the element ids of the evo_hist list, mapping the ids of the original agent to the ids of the elements used by its clone.

map_EvoHist(TableName,EvoHist)->

map_EvoHist(TableName,EvoHist,[]).

map_EvoHist(TableName,[{MO,E1Id,E2Id,E3Id}|EvoHist],Acc)->

9.2 Testing the Neuroevolutionary System on the Simple XOR Benchmark Clone_E1Id = ets:lookup_element(TableName,E1Id,2),

Clone_E2Id = ets:lookup_element(TableName,E2Id,2),

Clone_E3Id = ets:lookup_element(TableName,E3Id,2),

map_EvoHist(TableName,EvoHist,[{MO,Clone_E1Id,Clone_E2Id, Clone_E3Id}| Acc]);

map_EvoHist(TableName,[{MO,E1Id,E2Id}|EvoHist],Acc)->

Clone_E1Id = ets:lookup_element(TableName,E1Id,2),

Clone_E2Id = ets:lookup_element(TableName,E2Id,2),

map_EvoHist(TableName,EvoHist,[{MO,Clone_E1Id,Clone_E2Id}|Acc]);

map_EvoHist(TableName,[{MO,E1Id}|EvoHist],Acc)->

Clone_E1Id = ets:lookup_element(TableName,E1Id,2),

map_EvoHist(TableName,EvoHist,[{MO,Clone_E1Id}|Acc]);

map_EvoHist(_TableName,[],Acc)->

lists:reverse(Acc).

%map_EvoHist/2 is a wrapper for map_EvoHist/3, which in turn accepts the evo_hist list con- taining the mutation operator tuples that have been applied to the NN system. The function is

used when a clone of a NN system is created. The function updates the original Ids of the ele- ments the mutation operators have been applied to, to the ids used by the elements of the clone, so that the updated evo_hist can reflect the clone's topology, as if all the mutation operators have been applied to it instead, and that it is not a clone. Once all the tuples in the evo_hist have been updated with the clone's element ids, the list is reversed to its proper order, and the updat- ed list is returned to the caller.

Having fixed this bug, we move on to the next one.

9.2.3 Multiple Actuators of the Same Type

Looking again at the Listing-9.3, we see that one of the mutation operators was add_actuator . Since only successful mutation operators are allowed to be in the evo_hist list, it must be the case that only those mutation operators that actually mutated the genotype are present in the evo_hist list, which is what allows us to use it to trace back the evolutionary path of the evolved agent. But the presence of add_actuator in evo_hist must be an error, because the xor_mimic morphology on- ly gives the agent access to a single actuator, there are no variations of that actua- tor, and the agent starts with that single actuator. It should not be possible to add a new actuator to the NN system since there are no new ones available, and this tag should not exist in the evo_hist list. This mutation operator was applied in error, let's find out why.

Looking at the add_actuator/1 in the genome_mutator module, we can see that it does check whether all the actuators are already used. But if we look at the agent's fingerprint section of the console printout in Listing-9.3:


[{actuator,undefined,xor_SendOutput,undefined,

{private,xor_sim},

1

[{{0.5,7.570051345042677e-10},neuron}],

11 },

{actuator,undefined,xor_SendOutput,undefined,

{private,xor_sim},

1

[{{0,7.570051345042682e-10},neuron}],

undefined }]

We notice the problem. The last element in the record defining the actuator in the genotype is the generation element. One actuator has the generation set to 11 , the other has it set to undefined . In the add_actuator function, we do not reset the generation value, as we do with the id and the cx_id . This must be it. When we subtract the list of the actuators used by the agent from the morphology's list of available actuators, the resulting list is not empty. The reason why an actuator still remains in the list, is because we did not set the generation parameter of the agent's actuator to undefined. Since the two actuators are not exactly the same (with all their agent specific features been set to defaults), the actuator used by the agent is not removed from the list of available actuators of the morphology's actu- ator list.

This also raises the issue of what should we do, in a consistent manner, with the generation parameter of the actuator? Lets update the source code and treat the generation of the actuator element as we treat it in the neuron elements: Initially set it to the value of the generation when it was created, and update its value every time it has been affected by a mutation. We make the same modification to the sensor elements.

In the add_actuator/1 function we change the line:

...

case morphology:get_Actuators(Morphology)--[(genotype:read({actuator, A_Id}))#actuator{ cx_id=undefined, id=undefined, fanin_ids=[]} || A_Id<-A_Ids] of

...

To:

case morphology:get_Actuators(Morphology)--[(genotype:read({actuator, A_Id}))#actuator{ cx_id=undefined, id=undefined, fanin_ids=[], generation=undefined } || A_Id<-A_Ids] of


We do the same thing to the add_sensor/1 function. And then to ensure that the actuator's generation is updated every time a mutation operator affects it, we up- date the function linkFromNeuronToActuator/3 from using the line:

genotype:write(ToA#actuator{ fanin_ids=U_Fanin_Ids})

To one using:

genotype:write(ToA#actuator{ fanin_ids = U_Fanin_Ids, generation=Generation})

To make sure that the sensor's generation is also updated, we modify the func- tion link_FromSensorTo/2 from:

link_FromSensor(FromS,ToId)->

FromFanout_Ids = FromS#sensor.fanout_ids,

case lists:member(ToId, FromFanout_Ids) of

true ->

exit( “******** ERROR:link_FromSensor[cannot add ToId to Sensor]: ~p al-

ready a member of ~p~n ”,[ToId,FromS#sensor.id]);

false ->

FromS#sensor{

fanout_ids = [ToId|FromFanout_Ids]

}

end.

To the function link_FromSensorTo/3 :

link_FromSensor(FromS,ToId ,Generation )->

FromFanout_Ids = FromS#sensor.fanout_ids,

case lists:member(ToId, FromFanout_Ids) of

true ->

exit( “******** ERROR:link_FromSensor[can not add ToId to Sensor]: ~p al-

ready a member of ~p~n ”,[ToId,FromS#sensor.id]);

false ->

FromS#sensor{

fanout_ids = [ToId|FromFanout_Ids],

generation=Generation

}

end.

Finally, we also update the genotype module's function construct_Cortex/3 , from using:

Sensors = [S#sensor{id={{-1,generate_UniqueId()},sensor},cx_id=Cx_Id}|| S<-

morphology:get_InitSensors(Morphology)],


Actuators = [A#actuator{id={{1,generate_UniqueId()},actuator},cx_id=Cx_Idn}||A<-

morphology:get_InitActuators(Morphology)],

To one using:

Sensors = [S#sensor{id={{-1,generate_UniqueId()},sensor},cx_id=Cx_Id, generation=Generation } || S<- morphology:get_InitSensors(Morphology)],

Actuators = [A#actuator{id={{1,generate_UniqueId()},actuator},cx_id=Cx_Id,

generation=Generation } || A<-morphology:get_InitActuators(Morphology)],

Which ensures that we can keep track of the generation from the very start.

9.2.4 Making Fingerprint Store Generalized Sensors & Actuators

The fingerprint of the agent is used to vaguely represent the species that the agent belongs to. For example, if we have two NN systems which are exactly the same, except for the ids of their elements and the synaptic weights their neurons use, then these two agents belong to the same species. We cannot compare them directly to each other, because they will have those differences (the ids and the synaptic weights), but we can create a more generalized fingerprint for each agent which will be exactly the same for both. Some of the general features which we might use to classify a species is the NN topology and the sensors and actuators the NN system uses.

The 4 error we noticed was that we forgot to get rid of the N_Ids in the gener- alized sensor and actuator tuples within the fingerprint. We got rid of all the Id specific parts (the element's own id, and the cx_id) of those tuples before entering them into the fingerprint tuple, but forgot to do the same for the fanin_ids and the fanout_ids in the actuator and sensor tuples respectively. The fix is very simple, in the genotype module, we modify two lines in the update_fingerprint/1 function from:

GeneralizedSensors= [(read({sensor,S_Id}))#sensor{id=undefined,cx_id=undefined}

|| S_Id<-Cx#cortex.sensor_ids],

GeneralizedActuators= [(read({actuator,A_Id}))#actuator{id=undefined, cx_id=undefined}

|| A_Id<-Cx#cortex.actuator_ids],

To:

GeneralizedSensors= [(read({sensor,S_Id}))#sensor{id=undefined,cx_id=undefined,

fanout_ids =[]} || S_Id<-Cx#cortex.sensor_ids],

th


GeneralizedActuators= [(read({actuator,A_Id}))#actuator{id=undefined,cx_id =undefined,

fanin_ids=[]} || A_Id<-Cx#cortex.actuator_ids],

This change fixes the 4 and final error we've noticed. With this done, we now take our attention towards the remaining noticed anomaly, the 2 neuron NN solu- tion. How is it possible?

9.2.5 The Quizzical Topology of the Fittest NN System

The first thing we noticed, and the reason for a closer analysis of the evolved agent, was the NN's topology, the fact that it had 2 neurons instead of 3+ neurons. After our analysis though, and finding out that it also had 2 actuators, while inter- facing with only a single private scape, which means that both actuators were sending signals to it... there might be all kinds of different reasons for the 2 neuron solution. Nevertheless, let us still build it to see what exactly has evolved. Fig-9.5 shows the diagram of the final evolved NN system, based on the genotype in List- ing-9.3.

Fig. 9.5 The NN topology of the fittest agent in the population solving the XOR test, from Listing-9.3.

If we ignore the strange part about this NN system having two actuators, the reason behind which we have already solved in Section-9.2.3, we immediately spot another interesting feature. We have evolved a recurrent NN!

th


A recurrent NN can use memory, which means that the evolved solution, among other things, is most likely also sequence specific. This means that this so- lution takes into account the order in which the input data is presented. Since in the real world these types of signals would not be presented in any particular order to the XOR logic operator in question, our evolved system would not simulate the XOR operator properly anyway, even after having all the other errors fixed. A proper XOR mimicking neural network must not be sequence specific. Thus it is essential that for this problem we evolve a non recurrent NN system.

We need to be able to control and choose whether we want the evolving neural network systems to have recurrent connections or not. In the same way that we can choose what activation functions the NN system has access to (through the constraint record), we can also specify whether recurrent connections are allowed or not. To add this feature before we can retest our system, we need to: 1. Modify the records.hrl file to add the new element to the constraint tuple. And 2. Modify the genome_mutator module so that it checks whether recurrent or only feedforward connections are allowed, before choosing which elements to link to- gether.

Modifying the records.hrl file is easy, we simply change the constraint record from:

-record(constraint,{

morphology=xor_mimic, %xor_mimic

neural_afs=[tanh,cos,gaussian,absolute] %[tanh,cos,gaussian,absolute,sin,sqrt,sigmoid]

}).

To:

-record(constraint,{

morphology=xor_mimic, %xor_mimic

connection_architecture = recurrent, %recurrent|feedforward

neural_afs=[tanh,cos,gaussian,absolute] %[tanh,cos,gaussian,absolute,sin,sqrt,sigmoid]

}).

The new parameter: connection_architecture , can take on two values, either the atom: recurrent, or the atom: feedforward . Though we've added the new element, connection_architecture, to the constraint record, we still need to modify the ge- nome_mutator module so that it actually knows how to use this new parameter. In the genome_mutator module we need to modify all the mutation_operators that add new connections, and ensure that before a new connection is created, the func- tion takes the value of the connection_architecture parameter into consideration. The mutation operators that we need to modify for this are: add_outlink/1 , add_inlink/1 , and add_neuron/1 .


The updated add_outlink/1 function first builds an output id pool, which is a list of all available ids to which the selected neuron can choose to establish a link to. The general id pool is composed by combining together the list of actuator and neuron ids. We must remove from this id list the neuron's own Output_Ids list, which leaves a list of element Ids to which the neuron is not yet connected to. We then check whether the agent allows for recurrent connections, or only feedforward. If recurrent connections are allowed, then a random Id from this list is chosen, and the neuron and the chosen element are linked together. If on the other hand only the feedforward connections are allowed, the neuron's own layer index is checked, and then the composed id pool is filtered such that the remaining id list contains only the element ids whose layer index is greater than that of the neuron. This effectively creates a list of element ids which are 1 or more neural- layers ahead of the chosen neuron, and to whom if a connection is established, would be considered feedforward. To implement this new approach, we convert the original add_outlinke/1 function from:

add_outlink(Agent_Id)->

A = genotype:read({agent,Agent_Id}),

Cx_Id = A#agent.cx_id,

Cx = genotype:read({cortex,Cx_Id}),

N_Ids = Cx#cortex.neuron_ids,

A_Ids = Cx#cortex.actuator_ids,

N_Id = lists:nth(random:uniform(length(N_Ids)),N_Ids),

N = genotype:read({neuron,N_Id}),

Output_Ids = N#neuron.output_ids,

case lists:append(A_Ids, N_Ids ) -- Output_Ids of

[] ->

exit( “********ERROR:add_outlink:: Neuron already connected to all ids ”); Available_Ids ->

To_Id = lists:nth(random:uniform(length(Available_Ids)),Available_Ids),

link_FromElementToElement(Agent_Id,N_Id,To_Id),

EvoHist = A#agent.evo_hist,

U_EvoHist = [{add_outlink,N_Id,To_Id}|EvoHist],

U_A = A#agent{evo_hist=U_EvoHist},

genotype:write(U_A)

end.

To one that uses a filtered neuron id pool, Outlink_NIdPool, for the feedforward connections, and the entire id pool for when recurrent connections are allowed:

add_outlink(Agent_Id)->

A = genotype:read({agent,Agent_Id}),

Cx_Id = A#agent.cx_id,

Cx = genotype:read({cortex,Cx_Id}),


N_Ids = Cx#cortex.neuron_ids,

A_Ids = Cx#cortex.actuator_ids,

N_Id = lists:nth(random:uniform(length(N_Ids)),N_Ids),

N = genotype:read({neuron,N_Id}),

Output_Ids = N#neuron.output_ids,

Outlink_NIdPool = filter_OutlinkIdPool(A#agent.constraint,N_Id,N_Ids),

case lists:append(A_Ids, Outlink_NIdPool ) -- Output_Ids of

[] ->

exit( “********ERROR:add_outlink:: Neuron already connected to all ids ”); Available_Ids ->

To_Id = lists:nth(random:uniform(length(Available_Ids)),Available_Ids),

link_FromElementToElement(Agent_Id,N_Id,To_Id),

EvoHist = A#agent.evo_hist,

U_EvoHist = [{add_outlink,N_Id,To_Id}|EvoHist],

U_A = A#agent{evo_hist=U_EvoHist},

genotype:write(U_A)

end.

The filter_OutlinkIdPool(Constraint,N_Id,N_Ids) function has to filter the neu- ron ids (N_Ids) based on the specification in the constraint record. This new fil- ter_OutlinkIdPool/3 function, is shown in the following listing:

Listing-9.8 The implementation of filter_OutlinkIdPool/3, a constraint based neuron id filtering function.

filter_OutlinkIdPool(C,N_Id,N_Ids,Type)->

case C#constraint.connection_architecture of

recurrent ->

N_Ids;

feedforward ->

{{LI,_},neuron} = N_Id,

case Type of

outlink ->

[{{Outlink_LI,Outlink_UniqueId},neuron} || {{Outlink_LI,

Outlink_UniqueId}, neuron} <- N_Ids, Outlink_LI > LI];

inlink ->

[{{Inlink_LI,Inlink_UniqueId},neuron} || {{Inlink_LI,

Inlink_UniqueId},neuron} <- N_Ids, Inlink_LI < LI]

end

end.

%The function filter_OutlinkIdPool/3 uses the connection_architecture specification in the con- straint record of the agent to return a filtered neuron id pool. For the feedforward connec- tion_architecture, the function ensures that only the neurons in the forward facing layers are al- lowed in the id pool.


We can modify the add_inlink/1 mutation operator in the same way. In this function though, if we are to only have feedforward connections, then the filtered neuron id pool needs to have neurons whose layer is less than that of the chosen neuron which is trying to add an inlink. The add_inlink/1 function is modified in the same manner as the add_outlink/1, only we create and use the fil- ter_InlinkIdPool/3 function instead, which is shown in the following listing:

Listing-9.9 The implementation of filter_InlinkIdPool/3, a constraint based neuron ids filtering function.

filter_InlinkIdPool(C,N_Id,N_Ids)->

case C#constraint.connection_architecture of

recurrent ->

N_Ids;

feedforward ->

{{LI,_},neuron} = N_Id,

[{{Inlink_LI,Inlink_UniqueId},neuron} || {{Inlink_LI,

Inlink_UniqueId},neuron} <- N_Ids, Inlink_LI < LI]

end.

%The function filter_InlinkIdPool/3 uses the connection_architecture specification in the con- straint record of the agent to return a filtered neuron id pool. For the feedforward connec- tion_architecture, the function ensures that only the neurons in the previous layers are allowed in the filtered neuron id pool.

Finally, we modify the add_neuron/1 mutation operator. In this operator a new neuron B is created, and is then connected from a randomly chosen neuron A, and to a randomly chosen neuron C. As in the previous two mutation operators, we compose an Id pool specified by the architecture_constraint parameter, from which the Ids of A and C are then chosen. The modified version of the add_neuron/1 function is shown in Listing-9.10.

Listing-9.10 The modified add_neuron/1 mutation operator, which now uses id pools that satis-

fy the connection_architecture constraint specification. The bold parts of the code are the added and modified parts of the function.

add_neuron(Agent_Id)->

A = genotype:read({agent,Agent_Id}),

Generation = A#agent.generation,

Pattern = A#agent.pattern,

Cx_Id = A#agent.cx_id,

Cx = genotype:read({cortex,Cx_Id}),

N_Ids = Cx#cortex.neuron_ids,

S_Ids = Cx#cortex.sensor_ids,

A_Ids = Cx#cortex.actuator_ids,

{TargetLayer,TargetNeuron_Ids} = lists:nth(random:uniform(length(Pattern)),Pattern),


NewN_Id = {{TargetLayer,genotype:generate_UniqueId()},neuron},

U_N_Ids = [NewN_Id|N_Ids],

U_Pattern = lists:keyreplace(TargetLayer, 1, Pattern,

{TargetLayer,[NewN_Id|TargetNeuron_Ids]}),

SpecCon = A#agent.constraint,

genotype:construct_Neuron(Cx_Id,Generation,SpecCon,NewN_Id,[],[]),

Inlink_NIdPool = filter_InlinkIdPool(A#agent.constraint,NewN_Id,N_Ids),

Outlink_NIdPool = filter_OutlinkIdPool(A#agent.constraint,NewN_Id,N_Ids),

FromElementId_Pool = Inlink_NIdPool ++S_Ids,

ToElementId_Pool = Outlink_NIdPool ,

case (Inlink_NIdPool == []) or (Outlink_NIdPool == []) of

true ->

exit( “********ERROR::add_neuron(Agent_Id)::Can't add new neuron

here, Inlink_NIdPool or Outlink_NIdPool is empty. ”);

false ->

From_ElementId =

lists:nth(random:uniform(length(FromElementId_Pool)),FromElementId_Pool),

To_ElementId =

lists:nth(random:uniform(length(ToElementId_Pool)),ToElementId_Pool),

link_FromElementToElement(Agent_Id,From_ElementId,NewN_Id),

link_FromElementToElement(Agent_Id,NewN_Id,To_ElementId),

U_EvoHist = [{add_neuron,From_ElementId,NewN_Id, To_ElementId} |

A#agent.evo_hist],

genotype:write(Cx#cortex{neuron_ids = U_N_Ids}),

genotype:write(A#agent{pattern=U_Pattern,evo_hist=U_EvoHist})

end .

We do not need to modify outsplice/1 mutation operator, even though it does establish new connections. The reason for this is that if the connec- tion_architecture allows recurrent connections, then there is nothing to modify, and if it is feedforward, then all the connections are already made in the right di- rection, since if we add a new neuron, we either create a new layer for it, or put it in the layer located between the two spliced neurons, which allows the NN to re- tain the feedforward structure.

9.3 Retesting Our Neuroevolutionary System

Having now modified all the broken mutation operators, and fixed all the er- rors, we can compile all the modified modules, and retest our neuroevolutionary system. First, we will once again apply multiple mutation operators to our NN sys- tem, and then analyze the resulting NN architecture, manually checking if every- thing looks as it supposed to. We will then run multiple xor_mimic tests, each test


with a slightly different parameter set. This will give us a better understanding of how our system performs.

During this test, we still let the NN evolve recurrent connections. In the follow- ing listing we first compile and load the modules by executing polis:sync() .We then execute genome_mutator:long_test(10) . And then finally, we print the result- ing NN system's genotype to console, so that we can visually inspect it:

Listing-9.11 The long_test function applied to our now fixed neuroevolutionary system.

3> genome_mutator:long_test(10).

...


{agent,test,10,undefined,test, ...

[{mutate_weights,{{0.5,7.565644036503407e-10},neuron}},

{add_neuron,{{0.5,7.565644036503407e-10},neuron},

{{0.5,7.565644036354212e-10},neuron},

{{0.5,7.565644036503407e-10},neuron}},

{add_bias,{{0,7.565644036525425e-10},neuron}},

{add_outlink,{{0.5,7.565644036503407e-10},neuron},

{{0,7.565644036562396e-10},neuron}},

{add_outlink,{{0,7.565644036562396e-10},neuron},

{{0,7.565644036525425e-10},neuron}},

{mutate_af,{{0,7.565644036535494e-10},neuron}},

{mutate_af,{{0,7.565644036562396e-10},neuron}},

{add_bias,{{0,7.565644036535494e-10},neuron}},

{outsplice,{{0,7.565644036562396e-10},neuron},

{{0.5,7.565644036503407e-10},neuron},

{{1,7.565644036562401e-10},actuator}},

{mutate_af,{{0,7.565644036535494e-10},neuron}},

{mutate_weights,{{0,7.565644036562396e-10},neuron}},

{add_inlink,{{0,7.565644036525425e-10},neuron},

{{0,7.565644036525425e-10},neuron}},

{add_neuron,{{-1,7.565644036562414e-10},sensor},

{{0,7.565644036525425e-10},neuron},

{{0,7.565644036562396e-10},neuron}},

{add_neuron,{{0,7.565644036562396e-10},neuron},

{{0,7.565644036535494e-10},neuron},

{{0,7.565644036562396e-10},neuron}},

{add_outlink,{{0,7.565644036562396e-10},neuron},

{{0,7.565644036562396e-10},neuron}}],

0.13228659163157622,0,

[{0,

[{{0,7.565644036525425e-10},neuron},

{{0,7.565644036535494e-10},neuron},

Chapter 9 Testing the Neuroevolutionary System {{0,7.565644036562396e-10},neuron}]},

{0.5,

[{{0.5,7.565644036354212e-10},neuron},

{{0.5,7.565644036503407e-10},neuron}]}]}

{cortex,{{origin,7.56564403656243e-10},cortex},

test,

[{{0.5,7.565644036354212e-10},neuron},

{{0.5,7.565644036503407e-10},neuron},

{{0,7.565644036525425e-10},neuron},

{{0,7.565644036535494e-10},neuron},

{{0,7.565644036562396e-10},neuron}],

[{{-1,7.565644036562414e-10},sensor}],

[{{1,7.565644036562401e-10},actuator}]}

{sensor,{{-1,7.565644036562414e-10},sensor},

xor_GetInput,

{{origin,7.56564403656243e-10},cortex},

{private,xor_sim},

2

[{{0,7.565644036525425e-10},neuron},

{{0,7.565644036562396e-10},neuron}],

3}

{neuron,{{0.5,7.565644036354212e-10},neuron},

10

{{origin,7.56564403656243e-10},cortex},

absolute,

[{{{0.5,7.565644036503407e-10},neuron},[-0.07865790723708455]}],

[{{0.5,7.565644036503407e-10},neuron}],

[{{0.5,7.565644036503407e-10},neuron}]}

{neuron,{{0.5,7.565644036503407e-10},neuron},

10

{{origin,7.56564403656243e-10},cortex},

gaussian,

[{{{0.5,7.565644036354212e-10},neuron},[0.028673644861684]},

{{{0,7.565644036562396e-10},neuron},[0.344474633962796]}],

[{{0.5,7.565644036354212e-10},neuron},

{{0,7.565644036562396e-10},neuron},

{{1,7.565644036562401e-10},actuator}],

[{{0.5,7.565644036354212e-10},neuron},

{{0,7.565644036562396e-10},neuron}]}

{neuron,{{0,7.565644036525425e-10},neuron},

9

{{origin,7.56564403656243e-10},cortex},

cos,

[{{{0,7.565644036562396e-10},neuron},[0.22630117969617192]},

{{{0,7.565644036525425e-10},neuron},[0.06839553053285097]},


{{{-1,7.565644036562414e-10},sensor},

[0.4907662278024556,-0.3163769342514735]},

{bias,[-0.4041650818621978]}],

[{{0,7.565644036525425e-10},neuron},

{{0,7.565644036562396e-10},neuron}],

[{{0,7.565644036525425e-10},neuron},

{{0,7.565644036562396e-10},neuron}]}

{neuron,{{0,7.565644036535494e-10},neuron},

7

{{origin,7.56564403656243e-10},cortex},

cos,

[{{{0,7.565644036562396e-10},neuron},[0.30082326020002736]},

{bias,[0.00990196169812485]}],

[{{0,7.565644036562396e-10},neuron}],

[{{0,7.565644036562396e-10},neuron}]}

{neuron,{{0,7.565644036562396e-10},neuron},

9

{{origin,7.56564403656243e-10},cortex},

tanh,

[{{{0.5,7.565644036503407e-10},neuron},[0.29044390963714084]},

{{{0,7.565644036525425e-10},neuron},[-0.11820697604732322]},

{{{0,7.565644036535494e-10},neuron},[2.203261827127093]},

{{{0,7.565644036562396e-10},neuron},[0.13355748834368064]},

{{{-1,7.565644036562414e-10},sensor},

[-2.786539611443157,3.0562965644493305]}],

[{{0,7.565644036525425e-10},neuron},

{{0.5,7.565644036503407e-10},neuron},

{{0,7.565644036535494e-10},neuron},

{{0,7.565644036562396e-10},neuron}],

[{{0,7.565644036525425e-10},neuron},

{{0,7.565644036535494e-10},neuron},

{{0,7.565644036562396e-10},neuron}]}

{actuator,{{1,7.565644036562401e-10},actuator},

xor_SendOutput,

{{origin,7.56564403656243e-10},cortex},

{private,xor_sim},

1

[{{0.5,7.565644036503407e-10},neuron}],

5}

It works! Figure-9.6 shows the visual representation of this NN system's topol- ogy. If we inspect the mutation operators, and the actual connections, everything is in perfect order.


Fig. 9.6 The randomly evolved topology through the genome_mutator:long_test(10) execu- tion .

We will now test our system on the xor_mimic problem with the following set of parameters:

1. Constraint's activation functions set to [tanh], and MAX_ATTEMPTS to 50, 10, and 1:

This is done by changing the MAX_ATTEMPTS in the exoself module, for each separate test.

2. Activation functions are not constrained, connection_architecture is set to feedforward , and MAX_ATTEMPTS is set to 50, 10, and 1:

This is done by changing the INIT_CONSTRAINTS in the population_monitor module from one which previously constrained the activation functions, to one that no longer does so:

-define(INIT_CONSTRAINTS,[#constraint{morphology=Morphology,neural_afs=

Neural_AFs, connection_architecture=CA} || Morphology<-[xor_mimic], Neural_AFs<- tanh, CA<-[feedforward] ]).

To:


-define(INIT_CONSTRAINTS, [#constraint{morphology=Morphology,

connection_architecture=CA} || Morphology<-[xor_mimic], CA<-[feedforward] ]).

We have developed different kinds of activation functions, and created our neuroevolutionary system to give NN systems the ability to incorporate these var- ious functions based on their need. Also, the MAX_ATTEMPTS variable speci- fies the duration of the tuning phases, how well each topology is tested before it is given its final fitness score. A neuroevolutionary setup using MAX_ATTEMPTS = 1 is equivalent to it using a standard genetic algorithm rather than a memetic al- gorithm based approach, since the tuning phase then only acts as a way to assess the NN system's fitness, and all the mutation operators (including the weight per- turbation) are applied in the topological mutation phase. When the MAX_ATTEMPTS variable is set to 50, then each topology is tuned for a consid- erable amount of time.

To acquire the test-results of the above specified setup, we first set the parame- ters: INIT_CONSTRAINTS and the MAX_ATTEMPTS, to their new values, then run polis:sync() to update and load the modified modules, and then run the popula- tion_monitor:test() function to perform the actual test, the results of which are shown next:

Activation function: tanh, MAX_ATTEMPTS=50:

                • Population_Monitor:test shut down with Reason:normal OpTag:continue, while in OpMode:gt
                • Tot Agents:10 Population Generation:25 Eval_Acc:14806 Cycle_Acc:59224 Time_Acc:8038997

With the last generation's NN systems having the number of neurons ranging from: 6-9.

Activation function: tanh, MAX_ATTEMPTS=10:

                • Population_Monitor:test shut down with Reason:normal OpTag:continue, while in OpMode:gt
                • Tot Agents:10 Population Generation:33 Eval_Acc:5396 Cycle_Acc:21584 Time_Acc:2456883

With the last generation's NN systems having the number of neurons ranging from: 7-9.

Activation function: tanh, MAX_ATTEMPTS=1:

                • Population_Monitor:test shut down with Reason:normal OpTag:continue, while in OpMode:gt
                • Tot Agents:11 Population Generation:100 Eval_Acc:2281 Cycle_Acc:9124 Time_Acc:2630457


In this setup, the system failed to produce a solution, with the maximum fitness reached being ~7. This is understandable, since in the standard genetic algorithm's 97% of the mutations are weight perturbation based mutations, with the remainder being topological mutation operators. In our setup though, because our system does weight tuning in a different phase, the topological mutation phase uses the weight_perturbation operator with the same probability as any other. We will change this in the future.

Activation function: tanh, cos, gaussian, absolute MAX_ATTEMPTS=50:

                • Population_Monitor:test shut down with Reason:normal OpTag:continue, while in OpMode:gt
                • Tot Agents:9 Population Generation:1 Eval_Acc:910 Cycle_Acc:3640 Time_Acc:234083

Activation function: tanh, cos, gaussian, absolute MAX_ATTEMPTS=10:

                • Population_Monitor:test shut down with Reason:normal OpTag:continue, while in OpMode:gt
                • Tot Agents:10 Population Generation:4 Eval_Acc:694 Cycle_Acc:2776 Time_Acc:209243

Activation function: tanh, cos, gaussian, absolute MAX_ATTEMPTS=1:

                • Population_Monitor:test shut down with Reason:normal OpTag:continue, while in OpMode:gt
                • Tot Agents:9 Population Generation:22 Eval_Acc:565 Cycle_Acc:2260 Time_Acc:266885

Fig. 9.7 The discovered solution for the XOR problem, using only a single neuron .

9.4 Summary

The benchmark results when we allow for all activation functions to be used, are remarkably different. We've developed our neuroevolutionary system to allow the evolving NN systems to efficiently incorporate any available activation func- tions. In these last 3 scenarios, the evolved solutions all contained a single neuron, as shown in Fig-9.7 . In all 3 tests the solutions were reached within 1000 evalua- tions, very rapidly. The discovered solution? It was a single neuron without a bias, using a cos activation function.

We have now tested our neuroevolutionary system on the basic benchmark problem. We have confirmed that it can evolve solutions, that it can evolve topol- ogies and synaptic weights, that those solutions are correct, and that the evolved topologies are as expected. Though we've only developed a basic neuroevolutionary system thus far, it is decoupled and general enough that we can augment it, and easily improve it further, which is exactly what we will do in later chapters.


In this chapter we have thoroughly tested every mutation operator that we've added in the previous chapter. Though initially the mutation operator tests seemed successful, when testing our system on the XOR problem, and applying numerous mutation operators and then analyzing the evolved topology manually, we noticed errors to be present. We explored the origin of these detected errors, and then cor- rected them, re-testing our system on the XOR problem, successfully so.

The evolutionary algorithms built to evolve around problems, will also result in being able to evolve around small errors present in the algorithm itself. Thus, though it may seem that a test ran to completion, and did so successfully, as we've found out in this chapter, sometimes it is worthwhile to analyze the results, and the evolved agents, manually. It is during the thorough manual analysis that the more difficult to find errors are discovered. We have done just that in this chapter, and gained experience in the process of performing manual analysis of evolved NNs. This will give us an advantage in the future, as we continue adding more advanced features to our system, which will require debugging sooner or later.

Part III

A Case Study

In this part I will provide a case study of an already existing general topology and weight evolving artificial neural network (TWEANN) system created in Er- lang. Though there are a number of neuroevolutionary systems out there, I am most familiar with the following three which have shown to be the top performers within the field: DXNN [1,2], NEAT/HyperNEAT [3,4], and EANT1/2 [5,6]. One of these TWEANNs was written in Erlang, it is the system which I created and which I called: Deus Ex Neural Network (DXNN). The case study presented in the next chapter will be of this particular TWEANN platform.

[1] Sher GI (2010) DXNN Platform: The Shedding of Biological Inefficiencies. Neuron, 1-36. Available at: http://arxiv.org/abs/1011.6022.

[2] Sher GI (2012) Evolving Chart Pattern Sensitive Neural Network Based Forex TradingAgents. Available at: http://arxiv.org/abs/1111.5892.

[3] Stanley KO, and Miikkulainen R (2002) Evolving neural Networks Through Augmenting Topologies. Evolutionary Computation 10, 99-127.

[4] Gauci J, Stanley K (2007) Generating Large-Scale Neural Networks Through Discovering Geometric Regularities. Proceedings of the 9th annual conference on Genetic and evolution- ary computation GECCO 07, 997.

[5] Kassahun Y, Sommer G (2005) Efficient Reinforcement Learning Through Evolutionary Ac- quisition of Neural Topologies. In Proceedings of the 13th European Symposium on Artifi- cial Neural Networks ESANN 2005 (ACM Press), pp. 259-266.

[6] Siebel NT, Sommer G (2007) Evolutionary Reinforcement Learning of Artificial Neural Networks. International Journal of Hybrid Intelligent Systems 4, 171-183.

Chapter 10 DXNN: A Case Study

Abstract This chapter presents a case study of a memetic algorithm based TWEANN system that I developed in Erlang, called DXNN. Here we will discuss how DXNN functions, how it is implemented, and the various details and imple- mentation choices I made while building it, and why. We also discuss the various features that it has, the features which we will eventually need to add to the system we're building together. Our system has a much cleaner and decoupled implemen- tation, and which by the time we've reached the last chapter will supersede DXNN in every way.

Deus Ex Neural Network (DXNN) platform is the original topology and weight evolving artificial neural network system that I developed in Erlang. What you and I are creating here in this book is the next generation of it. We're developing a more decoupled version, a simpler to generalize and more refined version, and one with cleaner architecture and implementation. In this chapter we'll discuss the al- ready existing system, how it differs from what we've created so far, and what features it has that we will in later chapters need to add to the system we've devel- oped thus far. By the time this book ends, we'll have created not just a TWEANN system, but a Topology and Parameter Evolving Universal Learning Network framework, capable of evolving neural networks, circuits, be used as a parallel distributed genetic programming framework, posses some of the most advanced features currently known, and designed in such a way that new features can easily be added to it by simply incorporating new modules (hence the importance of developing a system where almost everything is decoupled from everything else).

DXNN is a memetic algorithm based TWEANN platform. As we discussed, the most advanced approach to neuroevolution and universal learning networks in general, is through a system that uses evolutionary algorithms to optimize both, the topology and the synaptic weights/node-parameters of the graph system. The weights and topology of a NN are evolved so as to increase the NN system's fit- ness, based on some fitness criteria/function.

In the following sections we will cover the algorithm and the various features that make up the DXNN system.

10.1 The Reason for the Memetic Approach to Synaptic Weight Optimization

As we have discussed in the first chapters, the standard genetic algorithm per- forms global and local search in a single phase, while the memetic algorithm sepa-

DOI 10.1007/978-1-4614- 4463 - 3_10, © Springer Science+Business Media New York 2013


rates these two searches into separate stages. When it comes to neural networks, the global search is done through the exploration of NN topologies, and the local search is done through the optimization of synaptic weights.

Based on the benchmarks, and ALife performance of DXNN, the memetic ap- proach has shown to be highly efficient and agile. The primary benefit of separat- ing the two search phases is due to the importance of finding the right synaptic weights for a particular topology before deciding on the final fitness score of that topology. Standard TWEANNs typically operate using the standard genetic algo- rithm based mutation operator probabilities. In such systems, when creating an offspring the parent is chosen and then a single mutation operator is applied to it, with a probability of more than 97% that the mutation operator will be a synaptic weight perturbation operator. This type of operator simply selects some number of neurons and perturbs some random number of synaptic weights belonging to them The other mutation operators are the standard topology augmenting operators.

In standard TWEANNs, a system might generate an optimal topology for the problem, but because during that one innovation of the new topology the at-that- point existing synaptic weights make that topology ineffective, the new NN topol- ogy might be disregarded and removed. Also, in most TWEANNs, the synaptic weight perturbations are applied indiscriminately to all neurons of the NN, and thus if for example a NN is composed of 1 million neurons, and a new neuron is added, the synaptic weight mutations might be applied to any of the 1000001 neu- rons... making the probability of optimizing the new and the right neuron and its synaptic weights, very low.

As in the system we've built so far, the DXNN platform evolves new NN to- pologies during each generation, and then through the application of an augmented stochastic hill climbing optimizes the synaptic weights for those topologies. Thus, when the “tuning phase ”, which is what the local search phase is called in DXNN, has completed, the tuned NN has roughly the best set of synaptic weights for its particular topology, and thus the fitness that is given to the NN is a more accurate representation of its true performance fitness and potential.

Furthermore, the synaptic weight optimization through perturbation is not ap- plied to all the neurons indiscriminately throughout the NN, but instead is concen- trated on primarily the newly created neurons, or those neurons which have been recently affected by a mutation applied to the NN. Thus, the tuning phase optimiz- es the newly added neural elements so that they work and contribute positively to the NN they have been added to.

With this approach, the DXNN system is able to slowly grow and optimize the NN systems. Adding new features/elements and optimizing them to work with the already existing structures. This I believe gives DXNN a much greater ability to scale, for there is zero chance of being able to create vast neural networks when


after adding a single new neuron to a 1000000 neuron NN system, we try to then perturb random synaptic weights in hopes of somehow making the whole system cohesive and functional. Building the NN slowly, complexifying it, adding new features and ensuring that they work with the existing system in a positive way, al- lows us to concentrate and optimize those few newly added elements, no matter how large the already existing NN system is.

Thus, during the local search phase, during the tuning phase , we optimize the synaptic weights of the newly added and modified elements. And during the glob- al search, during the topological mutation phase , we apply enough topological mutation operators when creating an offspring, such that we are able to create in- novation in the newly resulting NN system, but few enough of them such that the newly added elements to the NN can still be optimized to work with the existing much larger, already proven system.

Having discussed the why behind the memetic algorithm approach taken by DXNN, we now cover the two approaches this system uses when creating off- spring, clarified to a much greater detail in the next two sections. These two ap- proaches are the generational evolution, and the steady_state evolution.

The most common approach to offspring creation, and timing of selection and mutation operator application, is generational . Generational evolution simply means that we create a population of some size X of seed agents, apply them to some problem, wait until all agents in the population have been evaluated and giv- en a fitness score, then select the best of the population, allow them to create off- spring, and then create the next generation composed of the best agents of the previous generation plus their offspring, or some other appropriate combination of fit parents and newly created offspring. That is the essence of the standard genera- tional evolution.

The steady state evolution tries to emulate the biological world to a slightly greater degree. In this approach there is no wait for the entire population to be evaluated before a new agent is created. Instead, as soon as one agent has finished working on a problem (has been evaluated), or has perished or gathered enough resources (in the case of an ALife simulation), a new offspring is created. The new offspring is either created by some already existing agent through the execution of a create_offspring actuator, or is created by the neuroevolutionary system itself, after it has calculated what genotype/s to use as the base for the offspring creation process, and whether the environment can support another agent. In this manner, the population size working on a problem, or existing in a simulated environment, is kept relatively constant. There are always organisms in the environment, when some die, new ones are created. There is a constant turnover of new agents and new genotypes and phenotypes in the population.


Before we begin discussing the general algorithm of the generational and steady_state evolution, before we begin discussing the DXNN system and its vari- ous features, it would be helpful for me to first explain the architectures of the NN systems that are evolved by it. The DXNN's genotype encoding, and the TWEANN's architecture, differs slightly from what we've been developing in the past few chapters.

10.2 The DXNN Encoding and Architecture

The genotype encoding used by DXNN is almost exactly the same as the one used by the system we are building together. It is tuple encoded, with the tuples stored in the mnesia database. The list of records composing the genotype of each NN system in the DXNN platform is as follows:

-record(dx,{id,cx_id,n_ids,specie_id,constraint,morphology,generation,fitness,

profile,summary, evo_hist,mode, evo_strat}).

-record(cortex,{id,sensors,actuators,cf,ct,type,plasticity,pattern,cids,su_id,

link_form,dimensions,densities, generation}).

-record(neuron,{id,ivl,i,ovl,o,lt,ro,type,dwp,su_id,generation}).

The dx record plays the role that the agent record does in our TWEANN. The other thing that immediately stands out is that there are no sensor or actuator ele- ments. If you look in the DXNN's records.hrl [1] though, you will see those rec- ords, but they are not independent elements, the sensors and actuators are part of the cortex element. Indeed in the original DXNN system, the cortex element is not a synchronization element, but a gatekeeper element. The cortex element talks di- rectly to the neurons. The connection from the cortex to the neurons is accom- plished through the ct list (connected to), and the signals it gathers from the neu- rons is done through the cf list (connected from). The cortex also has a sensor and actuator list, which contain the names of the sensor and actuator functions, and the lists of the neurons that they are connected to and from respectively, based on the ct/cf lists. This DXNN's NN based agent architecture is shown in Fig-10.1 .


Fig. 10.1 The original DXNN based NN agent architecture.

The way a NN based system shown in Fig-10.1 functions, is as follows:

1. The genotype is first converted to the phenotype, composed of the cortex and the neurons, with the above shown architecture.

2. The cortex goes through all the sensor function names in its sensors list which has the following format : [{Sensor1,[{N_Id1,FilterTag1}, {N_Id2, FilterTag2}...]}...] . The cortex executes the sensor function names, and aggre- gates and packages the sensory signals generated through execution of the sensor functions. Because in the sensor list each sensor function comes with a list of neuron ids to which the resulting sensory signals are destined for, it is able to fanout those sensory signals to the specified neurons. Furthermore, in the above shown sensor list, the FilterTag has the following format: {single,Index}. {block,VL}, and {all,VL}. These tuples specify whether the sensor with a sen- sory signal of size vl, sends the entire sensory signal to the neuron, or just a single value from that vector list, a value located in the vector list at some par- ticular Index, respectively. The third FilterTag: {all, VL}, specifies that the cortex will append the sensory signals of all the sensors, and forward that list to the neuron in question.

3. The cortex then gets the neuron ids stored in the ct list, and forwards sensory signals to them, by mapping from the ct neuron ids to the sensor list neuron ids and their corresponding sensory vector signals (this design made sense when I was originally building the system, primarily because it originally also support- ed supervised learning, which required this design).

4. The neurons in the NN then process the sensory signals until the signals are generated by the neurons in the output layer.

5. The output layer neurons send their results to the cortex.


6. The cortex, as soon as it sends all the sensory signals to the neurons, waits until it receives the signals from the neurons whose PIds are the same as the PIds in its cf list, which are the signals destined for the actuators. It gathers these sig- nals into its accumulator, which is a list of lists, since the incoming signals are vectors.

7. After having gathered all the signals from the neurons, the cortex uses the actuators list and maps the composed output signal vectors to their respective actuators, and then executes the actuator functions using the output vectors as parameters.

8. GOTO 2

The original DXNN uses this particular convoluted architecture because I have developed it over a number of years, adding on new features, and modifying old fea- tures. Rather than redesigning the system once I've found a better way to represent or implement something, I simply modified it. DXNN has a modular version as well [2], where the evolved NN system is composed of modules called cores, where each core is a neural circuit, as shown in Fig-10.2 . In the modular DXNN, the cores can be hop- field networks, standard evolved neural networks, and even substrate encoded NNs. At one point, long ago, DXNN even had a back-propagation learning mode, which I eventually removed as I never used it, and it was inferior to the non supervised learn- ing algorithms I created. It is this long history of development, trial and error, testing and benchmarking, that left a lot of baggage in its architecture and implementation. Yet it is functional, and performs excellently.

Fig. 10.2 Modular DXNN.

10.3 Generational Evolution

In some sense, the neural modules within the modular DXNN system, were meant to be used in emulation of the various brain regions. In this manner I hoped to evolve different regions independently, and then put them together into a com- plete system, or evolve the different modules at the same time as a single NN sys- tem, or even let the NN start of as monolithic, and then modularize through evolu- tion. The performance though could not be established to be superior to standard homogeneous NN version at the time of experimentation, due to not yet having found a project benefiting from such an architecture. Nevertheless, the lessons learned were invaluable. The architecture of the TWEANN system we are devel- oping in this book, is made with future use of modules in mind. Indeed, the system we are developing here will not only have more features, and will be more decou- pled, but also its architecture will be cleaner, its implementation easier to under- stand, read, and expand, than that of DXNN. In the following sections I will ex- plain the functionality, algorithms, and features that DXNN possesses.


I will first provide a simple list based overview of the steps taken by DXNN's general neuroevolutionary algorithm, and then elaborate on each of the more com- plicated sub-algorithms the DXNN system uses. When using the generational ap- proach, DXNN uses the following set of steps:

1. Initialization Phase:

Create a seed population of size K of topologically minimalistic NN genotypes. 2. DO (Generational Neuroevolutionary loop):

3. Convert genotypes to phenotypes.

4. DO (Tuning Phase):

5. Test fitness of the NN system.

6. Perturb recently added or mutation operator affected synaptic weights.

UNTIL: NN's fitness has not increased for M times in a row.

7. Convert the NN systems back to their genotypes, with the now updat- ed and tuned synaptic weights.

8. Selection Phase:

9. Calculate the average energy cost of each neuron using the following method:

TotFitnessPoints = Agent_Fitness(1) + Agent_Fitness(2) + ...Agent_Fitness(K),

TotPopNeurons = Agent_TotNs(1) + Agent_TotNs(2) + …Agent_TotNs(K),

AvgNeuronCost = TotFitnessPoints/TotPopNeurons.


10.With all the NNs having now been given their fitness score, sort the genotypes based on their scores.

11.Mark the top 50% of the population as valid (fit), and the bot- tom 50% of the population as invalid (unfit).

12.Remove the bottom 50% of the population.

13.Calculate # of offspring for each agent:

14. For every agent(i) in K, calculate: Agent(i)_NeuronsAllotted=Agent_Fitness(i)/AvgNeuronCost,

Agent(i)_OffspringAlloted=

Agent(i)_NeuronsAlloted/Agent(i)_TotNs

15. To keep the population size of the new generation the same as the previous, calculate the population normalizer, and then normalize each agent's allotted offspring value: TotNewOffspring = Agent(1)_OffspringAlloted +

...Agent(i)_OffspringAlloted

Normalizer = TotNewOffspring/(K/2)

16. Now calculate the normalized number of offspring alloted for each agent:

Agent(i)_OffspringAllotedNorm =

round(Agent(i)_OffspringAlloted/Normalizer)

17.Create Agent(i)_OffspringAllotedNorm number of clones for every Agent(i) that belongs to the fit subset of the agents in the population. And then send each clone through the topological mutation phase, which converts that clone into an offspring.

18.Topological mutation phase:

19.Create the offspring by first cloning the parent, and then ap- plying to the clone, T number of mutation operators. The val- ue T is randomly chosen with uniform distribution to be be- tween 1 and sqrt(Agent(i)_TotNeurons), where TotNeurons is the number of neurons in the parent NN. Thus, larger NNs will produce offspring which have a chance of being pro- duced through a larger number of applied mutation operators.

20.Compose the population of the next generation by combining the gen- otypes of the fit parents with their newly created offspring.

UNTIL: Termination condition is reached (max # of evaluations, time, or fitness goal)

A diagram of this algorithm is shown in Fig-10.3 . The steps 1 (Initialization phase), 4 (Parametric Tuning Phase), 8 & 13 (The Selection Phase & Offspring Allocation), and 18 (Topological Mutation Phase), are further elaborated on in the subsections that follow.


Fig. 10.3 The different stages in the DXNN's learning algorithm: Initialization Stage, Tun- ing Phase, Selection Stage, and Topological Mutation Phase.

10.3.1 Step-1: Initialization Phase

During the initialization, every element created has its Generation set to 0. Ini- tially a seed population of size X is created. Each agent in the population starts with a minimal network, where the minimal starting topology depends on the total number of Sensors and Actuators the researcher decides to start the system with. If the NN is set to start with only 1 Sensor and 1 Actuator with a vl = 1 , then the DXNN starts with a single Cortex containing a single Neuron. For example, if the output is a vector of length 1 like in the Double Pole Balancing (DPB) control problem, the NN is composed of a single Neuron. If on the other hand the agent is initiated with N number of Sensors and K number of actuators, the seed NNs will contain 2 layers of fully interconnected Neurons. The first layer contains S Neu- rons, and the second contains A 1 +...A k Neurons. In this topology, S is the total number of Sensors, and A i is the size of the vector that is destined for Actuator i . It is customary for the NNs to be initialized with a single Sensor and a single Actua- tor, letting the agents discover any other auxiliary Sensors and Actuators through topological evolution.

Furthermore, the link from a Cortex to a Neuron can be of 3 types listed below:


1. Single-type link, in which the Cortex sends the Neuron a single value from one of its Sensors.

2. Block-type link, in which the Cortex sends the Neuron an entire vector that is output by one of the Sensors.

3. All-type link, in which the Cortex sends the Neuron a concatenated list of vec- tors from all the Sensors in its SensorList.

All this information is kept in the Cortex, the Neuron neither knows what type nor originally from which sensor the signal is coming. Each neuron only keeps track of the list of nodes it is connected from and the vector lengths coming from those nodes. Thus, to the Neuron all 3 of the previous link-types look exactly the same in its InputList, represented by a simple tuple {From_Id, Vector_Length}. The Vector_Length variable might of course be different for each of those connec- tions.

The different link-types add to the flexibility of the system and allow the Neu- rons to evolve a connection where they can concentrate on processing a single value or an entire vector coming from a Sensor, depending on the problem's need. I think this improves the general diversity of the population, allows for greater compactness to be evolved, and also improves the NN's ability to move through the fitness landscape. Since it is never known ahead of time what sensory values are needed and how they need to be processed to produce a proper output, differ- ent types of links should be allowed.

For example, a Cortex is routing to the Neurons a vector of length 100 from one of its Sensors. Assume that a solution requires that a Neuron needs to concen- trate on the 53rd value in the vector and pass it through a cosine activation func- tion. To do this, the Neuron would need to evolve weights equaling to 0 for all other 99 values in the vector. This is a difficult task since zeroing each weight will take multiple attempts, and during random weight perturbations zeroing one weight might un-zero another. On the other hand evolving a single link-type to that Sensor has a 1/100 chance of being connected to the 53rd value, a much better chance. Now assume that a solution requires for a neuron to have a connection to all of the 100 values in the vector. That is almost impossible to achieve, and would require at least 100 topological mutations if only a single link-type is used, but has a 1/3 chance of occurrence if we have block , all , and single type links at our dis- posal. Thus the use of Link-Types allows the system to more readily deal with the different and wide ranging lengths of signal vectors coming from the Sensors, and having a better chance of establishing a proper connection needed by the problem in question.

In a population, the agents themselves can also be of different types: Type = “neural ”, and Type = “substrate ”. The “neural ” type agent is one that is a standard recursive Neural Network system. The “substrate ” type agents use an architecture where the NNs drive a neural substrate, an encoding that was popularized by HyperNEAT [3]. In such agents the sensory vector is routed to the substrate and the output vector that comes from the substrate is parsed and routed to the actua-


tors. The supervised NN itself is polled to produce the weights for the embedded neurodes in the substrate. The type of substrates can further differ in density, and dimensionality. A diagram of the agent architecture that utilizes a substrate encod- ing is shown in Fig-10.4 . We will discuss the substrate encoded NN systems in greater detail in section 10.5.

Fig. 10.4 A DXNN evolved agent that uses a substrate encoded based architecture. In this figure the cortex goes through its sensors to produce the sensory signals, which it then packages and passes to the Substrate, which produces output signals and passes those to the Cortex which then postprocesses them and executes its actuators using these output vectors as parameters. The Substrate uses the NN to set the weights of its embedded neurodes.

10.3.2 Step-4: Parametric Tuning Phase

Since the offspring is created by taking the fit parent, creating its clone, and then applying topological mutation operators to it, we can tag any neuron in the NN that has been affected by the mutation operator. What counts as been affected by the mutation operator is as follows:

1. Having been just created, for example when a new neuron has just been added to the NN.

2. Having just acquired new input or output connection, for example when a neu- ron has just created a new link to another element, or when another element has just created a link to the neuron in question, the neuron is counted as having been affected by the mutation operator.

3. When during the topological mutation phase, the neuron's activation function, plasticity, or another parameter (other than weights) has been mutated.


Instead of just giving to such neurons the “mutationally affected ” tag, their generation parameter is reset, the same as is the case in the system we've built thus far. Thus, every element in the NN is given a generation during the initial seed population creation, and then every time the element is affected by a muta- tion, its generation is reset to the current generation, where the “current ” genera- tion is N where N increments every topological mutation phase, and is kept track of by the agent element. In this manner we can track which parts of the NN have been mutating, and which topological structures have stabilized and for a number of generations have not been affected by mutation. This stabilization usually oc- curs when the mutation of such structures produces a less fit offspring than its par- ent. So we can then, using this approach pick out the stabilized structures and crystallize them, making those structures a single unit (and be potentially repre- sented by a single process) that in the future will no longer be disturbed by muta- tion.

To choose whose synaptic weights to perturb during the tuning event, first the exoself chooses a random generation limit value as follows: GenLimit = 1/random:uniform() where the random:uniform() function generates a random value between 0 and 1 with a uniform distribution. Thus GenLimit will always be greater than 1, and have 50% of being 2, 25% of being 4... DXNN then uses the randomly generated GenLimit to compose a pool of neurons which have been af- fected by mutations within the last GenLimit of generations. In this neuron pool each neuron is chosen with a probability of 1/sqrt(NeuronPoolSize) to have its synaptic weights perturbed. The list of these chosen neurons is called the New Generation Neurons (NGN). The chosen neurons are then each sent a message by the exoself to have their synaptic weights perturbed. When a neuron receives such a message, it goes through its synaptic weight list and chooses each weight for perturbation with a probability of 1/sqrt(TotSynapticWeights) . The neuron then perturbs the chosen synaptic weights with a value randomly generated with uni- form distribution between -Pi and Pi.

This particular approach has the following benefits: 1. It concentrates on tuning and optimizing neurons that have only recently been added to the NN, thus ensur- ing that newly added neurons can contribute in a positive way to the NN. 2. There is a high variability in the number of neurons and weights that are chosen at any given time, thus there are times when a large number of neurons are all perturbed at the same time, and there are times when, by chance alone, only a few neurons and a few of their synaptic weights are chosen. Thus this approach allows the sys- tem to have a chance of doing both, tune into local optima on the fitness land- scape, and also at times choose a large number of neurons and weights to perturb, and thus search far and wide in the parametric space.

After NGN is composed, a variable MaxMistakes is created and set to abs ( BaseMaxMistakes + sqrt(TotWeights from NGNs)) rounded to the nearest in- teger. The BaseMaxMistakes variable is set by the researcher. Finally, a variable by the name AttemptCounter is created and set to 1.


The reason for the creation of the NGN list is due to the weight perturbations being applied only to the these new or recently modified Neurons, a method I refer to as “ Targeted Tuning ”. The reason to only apply perturbations to the NGNs is because evolution in the natural world works primarily through complexification and elaboration, there is no time to re-perturb all the neurons in the network after some minor topological or other type of addition to the system. As NNs grow in size it becomes harder and harder to set all the weights and parameters of all the Neurons at the same time to such values that produces a fit individual. A system composed of thousands of neurons might have millions of parameters in it. The odds of finding proper values for them all at the same time by randomly perturb- ing synaptic weights throughout the entire system after some minor topological mutation, is slim to none. The problem only becomes more intractable as the number of Neurons continues to grow. By concentrating on tuning only the newly created or newly topologically/structurally augmented Neurons and making them work with an already existing, tuned, and functional Neural Network, makes the problem much more tractable. Indeed in many respects it is how complexification and elaboration works in the biological NNs. In our organic brains the relatively recent evolutionary addition of the Neocortex was not done through some refur- bishing of an older NN structure, but through a completely new addition of neural tissue covering and working with the more primordial parts. The Neocortex works concurrently with the older regions, contributing and when possible overwriting the signals coming from our more ancient neural structures evolved earlier in our evolutionary history.

During the Tuning Phase each NN based agent tries to solve the problem based on its morphology. Afterwards, the agents receive fitness scores based on their performance in that problem. After being scored, each NN temporarily backs up its parameters. Every neuron in the NGN list has a probability of 1/sqrt(Tot_NGNs) of being chosen for weight perturbation. The Exoself sends the- se randomly chosen neurons a request to perturb some of their weights. Each cho- sen Neuron, after receiving such a message, chooses a set of its own synaptic weights, and perturbs them. The total number of weights to be perturbed is chosen randomly by every Neuron itself. The number of weights chosen for perturbation by each neuron is a random value between 1 and square root of total number of weights in that Neuron. The perturbation value is chosen with uniform distribution to be between -(WeightLimit/2) and (WeightLimit/2), where the WeightLimit is set to 2*Pi. By randomly selecting the total number of Neurons, the total number of weights to perturb, and using such a wide range for the perturbation intensity, we can achieve a very wide range of parametric perturbation. Sometimes the NN might have only a single weight in a single Neuron perturbed slightly, while at other times it might have multiple Neurons with multiple weights perturbed to a great degree. This allows the DXNN platform to make small intensity perturba- tions to fine tune the parameters, but also sometimes very large intensity (number of Neurons and weights) perturbations to allow NN based agents to jump over or out of local optima, an impossibility when using only small perturbations applied


to a small number of Neurons. This high mutation variability method is referred to in the DXNN platform as the Random Intensity Mutation (RIM). The range of mu- tation intensities grows as the square root of the total number of NGNs, as it logi- cally should since the greater the number of new or recently augmented Neurons in the NN, the greater the number of perturbations that needs to be applied to make a significant effect on the information processing capabilities of the system. At the same time, the number of neurons and weights affected during perturbation is limited only to the newly/recently added or topologically augmented elements, so that the system can try to adjust the newly added structures and those elements that are directly affected by them through new connections, to work and positively contribute to an already existing neural system.

After all the weight perturbations have been applied to the NN based agent, it attempts to solve the problem again. If the new fitness achieved by the agent is greater than the previous fitness it achieved, then the new weights overwrite the old backed up weights, the AttemptCounter is reset to 1, and a new set of weight perturbations is applied to the NN based agent. Alternatively, if the new fitness is not greater than the previous fitness, then the old weights are restored, the AttemptCounter is incremented, and another set of weight perturbations is applied to the individual.

When the agent's AttemptCounter == MaxMistakes , implying that a MaxMistakes number of unsuccessful RIMs have been applied in sequence with- out a single one producing an increase in fitness, the agent with its final best fit- ness and the correlated weights is backed up to the database through its conversion back to a list of tuples, its genotype, followed by the termination of the agent it- self. Utilizing the AttemptCounter and MaxMistakes strategy allows us, to some degree at least, test each topology with varying weights and thus let each NN after the tuning phase to represent roughly the best fitness that its topology can achieve. In this way there is no need to forcefully and artificially speciate and protect the various topologies since each NN represents roughly the highest potential that its topology can reach in a reasonable amount of time after the tuning phase com- pletes. This allows us to judge each NN based purely on its fitness. If one increas- es the BaseMaxMistakes parameter, then on average each NN will have more test- ing done on it with regards to weight perturbations, thus testing the particular topology more thoroughly before giving it its final fitness score. On the other hand the MaxMistakes parameter itself grows in proportion to the square root of the to- tal sum of NGN weights that should be tunned, since the greater the number of new weights that need to be tuned, the more attempts it would take to properly test the various permutations of neurons and their synaptic weights.


10.3.3 Step-8 & 13: The Selection & Offspring Allocation Phase

There are many TWEANNs that implement speciation during selection. Spe- ciation is used to promote diversity and protect unfit individuals who in the cur- rent generation do not possess enough fitness to get a chance of producing off- spring or mutating and achieving better results in the future. Promoters of speciation algorithms state that new ideas need time to develop and speciation pro- tects such innovations. Though I agree with the sentiment of giving ideas time to develop, I must point to [4] in which it was shown that such artificial and forced speciation and protection of unfit organisms can easily lead to neural bloating. DXNN platform does not implement forced speciation, instead it tests its individ- uals during the Tuning Phase and utilizes natural selection that also takes into ac- count the complexity of each NN during the Selection Stage. In my system, as in the natural world, smaller organisms require less energy and material to reproduce than their larger counterparts. As an example, for the same amount of material and energy that is required for a human to produce and raise an offspring, millions of ants can produce and raise offspring. When calculating who survives and how many offspring to allocate to each survivor, the DXNN platform takes complexity into account instead of blindly and artificially defending the unfit and insufficient- ly tested Neural Networks. In a way, it can also be thought that every NN topolo- gy represents a specie in its own right, and the tuning phase concisely tests out the different parametric permutations of that particular specie, same topologies with different weights. I believe that speciation and niching should be done not force- fully from the outside by the researcher, but by the artificial organisms themselves within the artificial environments they inhabit, if their environments/problems al- low for such a feat. When the organisms find their niches, they will automatically acquire higher fitness and secure their survival that way.

Due to the Tuning Phase, by the time Selection Stage starts, each individual presents its topology in roughly the best light it can reach within reasonable time. This is due to the consistent application of Parametric RIM to each NN during tar- geted tuning, and that only after a substantial number of continues failures to im- prove is the agent considered to be somewhere at the limits of its potential. Thus each NN can be judged purely by its fitness rather than have a need for artificial protection. When individuals are artificially protected within the population, more and more Neurons are added to the NN unnecessarily, thus producing the dreaded neural/topological bloating. This is especially the case when new neurons are add- ed, yet the synaptic weight perturbation and mutation is applied indiscriminately to all the synaptic weights in the NN. Topological bloating dramatically and cata- strophically hinders any further improvements due to a greater number of Neurons unnecessarily being in the NN and needing to have their parameters set concur- rently to just the right values to get the whole system functional. An example of such topological bloating was demonstrated in the robot arm control experiment using NEAT and EANT2 [4]. In that experiment, NEAT continued to fail due to significant neural bloating, whereas EANT2 was successful, which like DXNN is


a memetic algorithm based TWEANN. Once a NN passes some topological bloat- ing point, it simply cannot generate enough of concurrent perturbations to fix the faulty parameters of all the new neurons it acquired. At the same time, most TWEANN algorithms allow for only a small number of perturbations to be ap- plied at any one instance. In DXNN, through the use of Targeted Tuning and RIMs applied during the Tuning and Topological Mutation phases, we can successfully avoid bloating.

Finally, when all NNs have been given their fitness rating, we must use some method to choose those NNs that will be used for offspring creation. DXNN plat- form uses a selection algorithm I call “Competition ”, which tries to take into ac- count not just the fitness of each NN, but also the NN's size. The c ompetition se- lection algorithm is composed of the following steps:

1.

2.

3.

4.

5.

6.

7.

8.

9.

Calculate the average energy cost of the Neuron using the following steps: TotEnergy = Agent(1)_Fitness + Agent(2)_Fitness...

TotNeurons = Agent(1)_TotNeurons + Agent(2)_TotNeurons... AverageEnergyCost = TotEnergy/TotNeurons

Sort the NNs in the population based on their fitness. If 2 or more NNs have the same fitness, they are then sorted further based on size, more compact solutions are considered of higher fitness than less compact solutions.

Remove the bottom 50% of the population.

Calculate the number of alloted offspring for each Agent(i):

AllotedNeurons = (Fitness/AverageEnergyCost),

AllotedOffsprings(i) = round(AllotedNeurons(i)/Agent(i)_TotNeurons) Calculate total number of offspring being produced for the next generation: TotalNewOffsprings = AllotedOffsprings(1)+...AllotedOffsprings(n). Calculate PopulationNormalizer, to keep the population within a certain limit: PopulationNormalizer = TotalNewOffsprings/PopulationLimit Calculate the normalized number of offspring alloted to each Agent: NormalizedAllotedOffsprings(i) =

round(AllotedOffsprings(i)/PopulationNormalizer(i)).

If NormalizedAllotedOffsprings (NAO) == 1, then the Agent is allowed to sur- vive to the next generation without offspring, if NAO > 1, then the Agent is al- lowed to produce (NAO -1) number of mutated copies of itself, if NAO = 0 the Agent is removed from the population and deleted.

The Topological Mutation Phase is initiated, and the mutator program then passes through the database creating the appropriate NAO number of mutated clones of the surviving agents.

From this algorithm it can be noted that it becomes very difficult for bloated NNs to survive when smaller systems produce better or similar results. Yet when a large NN produces significantly better results justifying its complexity, it can begin to compete and push out the smaller NNs. This selection algorithm takes in- to account that a NN composed of 2 Neurons is doubling the size of a 1 Neuron NN, and thus should bring with it sizable fitness gains if it wants to produce just


as many offspring. On the other hand, a NN of size 101 is only slightly larger than a NN of size 100, and thus should pay only slightly more per offspring. This is exactly the principle behind the “competition ” selection algorithm we implemented in the system we are developing together in this book.

10.3.4 Step-18: The Topological Mutation Phase

An offspring of an agent is produced by first creating a clone of the parent agent, then giving it a new unique Id, and then finally applying Mutation Opera- tors to it. The Mutation Operators (MOs) that operate on the individual's topology are randomly chosen with uniform distribution from the following list:

1.

2.

3.

4.

5.

6.

7.

8.

9.

“Add Neuron ” to the NN and link it randomly to and from randomly chosen Neurons within the NN, or one of the Sensors/Actuators.

“Add Link ” (can be recurrent) to or from a Neuron, Sensor, or Actuator. “Splice Neuron ” such that that two random Neurons which are connected to each other are disconnected and reconnected through a newly created Neuron. “Change Activation Function ” of a random Neuron.

“Change Learning Method ” of a random Neuron.

“Add Bias ”, all neurons are initially created without bias.

“Remove Bias ”, removes a bias value in the neurons which have one.

“Add Sensor Tag ” which connects a currently unused Sensor present in the SensorList to a random Neuron in the NN. This mutation operator is selected with a researcher defined probability of X. In this manner new connections can be made to the newly added or previously unused sensors, thus expanding the sensory system of the NN.

“Add Actuator Tag ” which connects a currently unused Actuator present in the ActuatorList to a random Neuron in the NN. This mutation operator is selected with a researcher defined probability of Y. In this manner new connections can be made to the newly added or previously unused actuators, thus expanding the types of tools or morphological properties that are available for control by the NN.

The “Add Sensor Tag ” and “Add Actuator Tag ” can both allow for new links from/to the Sensor and Actuator programs not previously used by the NN to be- come available to it. In this manner the NN can expand its senses and control over new actuators and body parts. This feature becomes especially important when the DXNN platform is applied to the Artificial Life and Robotics experiments where new tools, sensors, and actuators might become available over time. The different sensors can also simply represent various features of a problem, and in this man- ner the DXNN platform naturally incorporates feature selection capabilities.

The total number of Mutation Operators (MOs) applied to each offspring of the DXNN is a value randomly chosen between 1 and square root of the total number of Neurons in the parent NN. In this way, once again a type of random intensity


mutation (RIM) approach is utilized. Some mutant clones will only slightly differ from their NN parent, while others might have a very large number of MOs ap- plied to them, and thus differ drastically. This gives the offspring a chance to jump out of large local optima that would otherwise prove impassible if a constant number of mutational operators were to have been applied every time, independ- ent of the parent NN's complexity and size. As the complexity and size of each NN increases, each new topological mutation plays a smaller and smaller part in changing the network's behavior, thus a larger and larger number of mutations needs to be applied to produce significant differences to the processing capabili- ties of that individual. For example, when the size of the NN is a single neuron, adding another one has a large impact on the processing capabilities of that NN. On the other hand, when the original size is a million neurons, adding the same single neuron to the network might not produce the same amount of change in the computational capabilities of that system. Increasing the number of MOs applied based on the size of the parent NN's size, allows us to make the mutation intensity significant enough to allow the mutant offspring to continue producing innova- tions in its behavior when compared to its parent, and thus exploring the topologi- cal fitness landscape far and wide. At the same time, due to RIM, some offspring will only acquire a few mutations and differ topologically only slightly and thus have a chance to tune and explore the local topological areas on the topological fitness landscape.

Because the sensors and actuators are represented by simple lists of existing sensor and actuator programs, just like in the system we're developing together in this book, the DXNN platform allows for the individuals within the population to expand their affecting and sensing capabilities. Such abilities integrated naturally into the NN lets individuals gather new abilities and control over functions as they evolve. For example, originally a population of very simple individuals with only distance sensors is created. At some point a fit NN will create a mutant offspring to whom the “Add Sensor Tag ” or “Add Actuator Tag ” mutational operator is ap- plied. When either of these mutational operators is randomly applied to one of the offspring of the NN, that offspring then has a chance of randomly linking from or to a new Sensor or Actuator respectively. In this manner the offspring can acquire color, sonar or other types of sensors present in the sensor list, or acquire control of a new body part/actuator, and thus further expand its own morphology. These types of expansions and experiments can be undertaken in the artificial life/robotics simulation environments like the Player/Stage/Gazebo Project [5]. Player/Stage/Gazebo in particular has a list of existing sensor and actuator types, making such experiments accessible at a very low cost.

Once all the offspring are generated, they and their parents once more enter the tuning phase to continue the cycle as was diagrammed in Fig-10.3.

10.4 Steady-State Evolution

Though the generational evolution algorithm is the most common approach, when applying neuroevolutionary systems to ALife, or even non ALife simulations and problems, steady-state evolution offered by DXNN can provide an advantage due to its content drift tracking ability, and a sub population called “Dead Pool ” which can immediately be used to develop committee machines. In a steady-state evolu- tion, the population solving the problem or existing within the simulated world (in the case of ALife for example) always maintains a constant operational popula- tion. When an organism/agent dies, or when there is more room in the environ- ment (either due to the expansion of the food source in ALife environment, or be- cause more computational power is added, or more exploration is wanted...) more concurrently existing agents are added to the operational phenotypes. The system does not wait for every agent in the population to finish being evaluated before generating a new agent and entering it into the population. Instead, the system computes the fitness of the just having perished agent, and then immediately gen- erates a new genotype from a pool of previously evaluated fit genotypes. Thus the system maintains a relatively constant population size by consistently generating new offspring at the same pace that agents complete their evaluations and are re- moved from the live population.

In DXNN, the steady-state evolutionary algorithm uses an “Augmented Com- petition ” (AC) selection algorithm. The AC selection algorithm keeps a list of size “PopulationSize ” of dead NN genotypes, this list is called the “dead pool ”. The variable PopulationSize is specified by the researcher. When an Agent dies, its genotype and fitness is entered into this list. If after entering the new genotype in- to the dead pool the list's size becomes greater than PopulationSize, then the low- est scoring DXNN genotype in the dead pool is removed. In this manner the dead pool is always composed of the top performing PopulationSize number of ancestor genotypes.

In this augmented version of the selection algorithm, the AllottedOffspring var- iables are converted into normalized probabilities used to select a parent from the dead pool to produce a mutated offspring. Finally, there is a 10% chance that in- stead of creating an offspring, the parent itself will enter the environment/scape or be re-applied to the problem. Using this “re-entry ” system, if the environment or the manner in which the fitness is allotted changes, the old strategies and their high fitness scores can be re-evaluated in the changed environment to see if they still deserve to stay in the dead pool, and if so, what their new fitness should be. This selection algorithm also has the side effect of having the dead pool implicitly track content drift of the problem to which the TWEANN is applied.

For example assume that the steady-state evolution with the dead_pool list is applied to an ALife simulation. Every time an agent in the simulated environment dies (has been evaluated), it is entered into the dead pool, and a new offspring is generated from the best in this dead pool. Once the dead pool size reaches that of


PopulationSize specified by the researcher, the DXNN system also begins to get rid of the poorly performing genotypes in the dead_pool. But what is important is that when an organism in the environment dies, there is a chance that a genotype in the dead pool has a chance of re-entering the simulated environment, instead of a new mutant offspring being generated. If it were not for this, then as the envi- ronment changes with the dynamics and fitness scoring and life expectancy all changing with it... and some organism dies, the old organisms, the genotypes from the “old world ” would be used to create the offspring. If the environment is highly dynamic and malleable, after a while the whole thing might change, the useful survival instincts and capabilities that were present in the environment to which the dead_pool organisms belonged, might no longer be present in the current, evolved environment. Suddenly we would be faced with a dead_pool of agents all with high scores, which though achievable in the previously simple environment, are no longer possible in the now much more complex and unforgiving environ- ment. Thus it is essential to re-evaluate the organisms in the dead_pool, are they still fit in the new environment, in the environment that itself has evolved and be- come more complex? Can the old agents compete in the new world?

The re-entry system allows us to change and update the dead_pool with the or- ganisms that are not simply more fit, but are more fit in the current state of the en- vironment. The environment can be either the simulated environment of the ALife system, or the new signal block in the time series of currency-pairs or stock prices for example. The patterns of the market that existed last year, might have changed completely this year, and it is essential that the new agents are judged by how they perform on this year's patterns and styles of the time series. This is the benefit of the content drift tracking dead pool. The dead pool represents the best of the popu- lation, a composition of agent genotypes that perform well in the relatively new environment, that perform well in the world of today, rather than the one of last year.

Furthermore, because the genotypes belonging to the dead_pool represent the best of the population, we can directly use the genotypes in it to compose a com- mittee machine. The current state of the dead pool is the voting population of the committee machine, the type of system we discussed in Section-1.2.2. This type of setup is shown in Fig-10.5 .


Fig. 10.5 A DXNN system using steady-state evolution used to evolve currency trading agents, and whose dead pool is used as a committee machine applied to real Forex trading.

The steps of the steady-state evolution algorithm in the DXNN platform are as follows:

1. Initialization Phase :

2. Create a seed population of size K of topologically minimalistic NN genotypes.


4. DO (Steady-State Neuroevolutionary loop) :

5. For Each Agent, DO (Tuning Phase) :

6. Test fitness of the NN system

7. Perturb the synaptic weights of recently added or mutation operator affected neurons

UNTIL: NN's fitness has not increased for X times in a series

8. Convert the NN system back to its genotype, with the now updated and tuned synaptic weights.

9. Add the agent's genotype to the dead_pool list of size K.

10. Steady-State Selection Phase (For genotypes in the dead_pool list) :

11.Calculate the average energy cost of each neuron using the following method:

TotFitnessPoints = Agent_Fitness(1) + Agent_Fitness(2) + ...Agent_Fitness(K),

TotPopNeurons = Agent_TotNs(1) + Agent_TotNs(2) + ...Agent_TotNs(K),

AvgNeuronCost = TotFitnessPoints/TotPopNeurons.


12.With all the NNs having now been given their fitness score, sort the genotypes based on their scores.

13.Extract the top K agents in this sorted dead_pool list, delete the others. This is done for the case when the addition of the new agent to the dead_pool, makes the size of the dead_pool larger than K. We only want to keep K agents in the dead_pool.

14. Select a dead_pool champion agent :

15.Agent(i)_NeuronsAllotted =

Agent_Fitness(i)/AvgNeuronCost,

Agent(i)_OffspringAllotted =

Agent(i)_NeuronsAllotted/Agent(i)_TotNs 16.Convert Agent(i)_OffspringAllotted for each agent

into a normalized percentage, such that a random agent from this list can be chosen with the uniform distribution probability proportional to its Agent(i)_OffspringAllotted value.

17.Choose the agent through step-16, and designate that agent as dead_pool champion.

18.Randomly choose whether to use the dead_pool champion as the parent of a new offspring agent, or whether to extract the champion from the dead_pool, convert it to its phenotype, and re-apply it to the problem. The split is 90/10, with 90% chance of us- ing the champion's genotype to create a new off- spring, and 10% chance of removing the agent from the dead_pool and re-applying (aka re-entry, re- evaluation...) the agent to the problem.

19. IF champion selected to create offspring :

20. Topological mutation phase :

21.Create the offspring by first cloning the parent, and then applying to the clone T number of mutation operators, T is random- ly chosen to be between 1 and sqrt(Agent(i)_TotNeurons). Where the TotNeurons is the number of neurons in the parent NN, and T is chosen with uniform distribution. Thus larger NNs will produce offspring which have a chance of being produced through a larger number of ap- plied mutation operators to them.

22.Designate the offspring agent as New_Agent . ELSE champion is chosen for re-entry :

23.Extract agent from the dead_pool.

24.Designate the agent as New_Agent .

10.5 Direct (Neural) and Indirect (Substrate) Encoding

25.Convert the agent designated as New_Agent to its phenotype.

UNTIL: Termination condition is reached (max # of evaluations, time, or fit- ness goal)

As can be noted from these steps, the algorithm is similar to the generational evolutionary approach, but in this case as soon as an agent dies (if in ALife exper- iment), or finishes its training or being applied to the problem, its fitness is imme- diately evaluated against the dead_pool agents, and a new agent is created (either through offspring creation or re-entry) and applied to the problem, or released into the simulated environment.

The tuning phase and the topological mutation phase are the same as in the generational evolutionary loop, discussed in the previous section. The steady-state selection algorithm only differs in that the allotted_offspring value is converted to a percentage of being selected for each agent in the dead_pool. The selected agent has a 90% chance of creating an offspring and 10% chance of being sent back to the problem, and being re-evaluated with regards to its fitness.

The following sections will cover a few finer points and features of DXNN. In the next section we will discuss its two types of encoding, neural and substrate. In section 10.6 we will briefly discuss the flatland simulator, a 2d ALife environ- ment. In section 10.7 we will discuss the modular version of DXNN. Finally, in section 10.8 and 10.9 we will discuss the ongoing projects and features being inte- grated into the DXNN system, and the neural network research repository being worked on by the DXNN Research Group.


The DXNN platform evolves both direct and indirect encoded NN agents. The direct encoded NN systems are as discussed in the above sections, these are stand- ard neural networks where every neuron is encoded as a tuple, and the mapping from the genotype to phenotype is direct. We simply translate the tuple containing the synaptic weights and link specifications into a process, linked to other pro- cesses and possessing the properties and synaptic weights dictated by the tuple.

The indirect encoding that the DXNN can also use is a form of substrate encod- ing, popularized by the HyperNEAT [3]. There are many variations of substrate encoding, and new ones are turning up every year. In a substrate encoded NN, the actual NN is not directly used to process input sensory signals and produce output signals to control the actuators. Instead, in a substrate encoded NN system the NN “paints ” the synaptic weights and connectivity patterns on a multidimensional substrate of interconnected neurodes. This substrate, based on the synaptic weights determined by the NN, is then used to process the input sensory signals and pro-


duce output signals used by the actuators. The architecture of such a system is shown in the following figure.

Fig. 10.6 Substrate encoded neural network system. This diagram is of a substrate encoded agent. The substrate, sensors, and actuators, are all part of the same process called Cortex. The NN is used to generate the synaptic weights between neurodes in the substrate, based on the coordinates of the presynaptic and postsynaptic neurodes. The sample agent shown is one that controls a simulated robot in an ALife experiment, a simulated robot that has a Range Sensor, a Distance Sensor, and a Differential Drive Actuator.

The neurodes in the substrate all use the sigmoid or tanh activation function, though this of course can be changed. Furthermore, the NN's output can be used for anything, and not only used as the synaptic weights for the coordinate speci- fied neurodes. For example, the output of the NN can be used and considered as the Delta Weight , the change in the synaptic weight between the pre- and post- synaptic neurodes, based on the coordinates of the said neurodes fed to the NN, in addition with the pre-synaptic neurode's output, the post-synaptic neurode's out- put, and the current synaptic weight between the two. We will further discuss the details of substrates and their functionality in the following section, followed by a discussion of the genotype encoding DXNN uses for substrates, the phenotype representation that it uses for such substrate encoded agents, and finally the differ- ent types of “substrate_sensors ” and “substrate_actuators ”, which further modify


the substrate encoded NN systems, allowing the NN to not only use the coordi- nates of the two connected neurodes when computing the synaptic weight between them, but various other geometrically significant features, like distance, spherical coordinates, planner coordinates, centripetal distance...

10.5.1 Neural Substrates

A neural substrate is simply a hypercube structure whose axis run from -1 to 1 on every dimension. The substrate has neurodes embedded in it, where each neurode has a coordinate based on its location within the hypercube. The neurodes are connected to each other, either in the feed forward fashion, a fully connected fashion, or random connection based fashion. An example of a 2d substrate is shown in Fig-10.7a , and a 3d substrate in Fig-10.7b .

Fig. 10.7 An example of different substrates in which the neurodes are connected to each other in the feed forward fashion.


The density of the substrate refers to the number of neurons on a particular ax- is. For example, if the substrate is a 2d one, and the density of the substrate is 5 by 3, then this plane substrate has 5 neurons, uniformly distributed on the x axis, with 3 total of such layers, which too are uniformly distributed on the y axis, as shown in Fig-10.7a . The Fig-10.7b shows a 3d substrate with the density distribution of 3x3x3. In this substrate, there are 3 planes on the Z axis, where each plane is com- posed out of 3x3 neurode patterns. Each plane is connected to the plane ahead of it, hence it is a feed forward based substrate, since the signals travel from the -Z direction, towards the +Z direction. We could of course have a fully connected substrate, where every neurode is connected to every other neurode. Also, the sub- strate does not necessarily need to be symmetric, it can have any type of pattern, any number of neurons per layer or hyperlayer, and positioned in any pattern with- in that layer or hyperlayer.

From these examples you can see that the processing, input, and output hyperlayers, are one dimension lower than the entire substrate. The sensory sig- nals travel from the negative side of the axis of the most external dimension (Y in the case of 2d, and Z in the case of 3d in the above examples), from the input hyperlayer, through the processing hyperlayers, and finally to the output hyperlayer, whose neurodes' output counts as the output of the substrate (but again, we could designate any neurode in the substrate as an output neurode, and wait until all such output neurodes produce a signal, and count that as the sub- strate's output). The manner in which we package the output signals of the neurodes within the output hyperlayer, and the manner in which we feed those packaged vectors to the actuators, determines what the substrate encoded NN based agent does. Finally, because the very first hyperlayer is the input to the sub- strate, and the very last hyperlayer is the output of the substrate, there must be at least 2 hyperlayers making up the substrate structure.

For example, assume we'd like to feed an image coming from a camera into the substrate encoded NN system. The image itself is a bitmap, let's say of resolution 10x10. This is perfect, for this type of input signal we can create a 3d substrate with a 10x10 input hyperlayer, 3x3 hidden processing hyperlayer, and a 1x5 out- put hyperlayer. Each hyperlayer is a 2d plane, all positioned on the 3 dimension, thus making the substrate 3d, as shown in Fig-10.8 . As can be seen, the input be- ing the very first layer located at Z = -1, has its signals sent to the second layer, located at Z = 0, which processes it, and whose neurode outputs are sent to the 3 layer at Z = 1, processed by the last 5 neurodes whose output is considered the final output of the substrate.

rd

rd


Fig. 10.8 A [{10,10},{3,3},{5,1}] substrate being fed a 2d plane image with a 10x10 ({10,10}) resolution.

You might be asking at this point “What is the advantage of using substrate en- coding? ” The answer is in the way we produce their weights. The weights are de- termined by the NN which calculates the synaptic weight between two connected neurodes based on their coordinates. The coordinates of the connected neurodes act as the input to the NN. Since the NN has the coordinates as input, it can do the following:

1. Extract geometrical patterns in the input hyperlayer, and thus it can be applied to highly complex problems where such geometrical information can be ex- ploited.

2. Be used to generate weights for very large and very dense substrates, with the connectivity and synaptic weight patterns based on the coordinates, and thus being of almost any complexity and form.

3. Due to never seeing the actual input signals, it cannot evolve a single synaptic weight for some particular element in the input vector during training, it cannot evolve some specific set of synaptic weights to pick out a particular single small pattern. In other words, a substrate encoded NN has a much lower chance of overtraining. It paints the synaptic weights broadly on the substrate, and thus it should be able to generalize that much better.

4. Because the NN produces a smooth function, and because each neurode in the substrate has presynaptic connections from a smooth spread of neurodes, with regards to their coordinates in the previous hyperlayer, the synaptic weights produced by the NN for any particular neurode, varies smoothly. This is the reason why it is much more difficult for such synaptic weights to overtrain on


some single particular points in the input stream of signals. Hence the superior generalization. The NN paints the synaptic weights and connectivity patterns on the substrate in “broad strokes ”, so to speak.

Let us discuss some of the things mentioned in more detail.

Geometrical Feature Sensitivity:

As discussed, the input to the NN is a list of coordinates for the connected pre- synaptic and post-synaptic neurodes. Not only are the coordinates used as input to the NN, but also the coordinates can be first converted to spherical coordinates, polar coordinates, distance between the connected neurodes, distance to the center of the substrate... before they are fed to the NN. Because a NN is a universal func- tion approximator, and the inputs are various geometrical elements, and because the input hyperlayer itself has coordinates, the NN gains the ability to pick out and deal with the geometrical features of the substrate, and the sensory signals.

Large Neural Network Structures:

Since the substrate neurode density is independent of the actual NN which we evolve, through substrate encoding it is possible to create very large/dense sub- strates, with thousands or millions of neurodes. Thinking again about the substrate analyzing the data/images coming from a camera, we can also see that the denser the substrate, the higher the resolution of images it can analyze. Also the resolu- tion of the sensory inputs and the output of the substrate, are independent of the NN painting the connectivity and synaptic weights on it. The “curse of dimension- ality ” does not plague this type of system as much, since we can concentrate on a smaller number of evolving parameters and topologies (of the actual evolving NN), while controlling a vast substrate embedded NN. Finally, it is also possible to implement synaptic plasticity using iterative , abc, and other types of substrate learning rules [6], which we will discuss in detail and implement in later chapters.

The “Broad Stroke ” property:

Because the neural network that calculates the synaptic weights for the neurodes in the substrate does not see the actual input vectors, and instead only deals with the coordinates. And because the output of the NN is a smooth func- tion, and the input coordinates to the NN are based on the connected neurodes, and each neurode is connected from a whole spectrum of neurodes in the previous hy- per-layer, with their coordinates changing smoothly from -1 to 1. The synaptic weights are painted in “Broad Strokes ”. Meaning, due to the inability of the NN to pick out any particular points in the incoming data, the synaptic weights it gener- ates are smooth over the whole substrate. A change in the NN system changes the weights, the output function of the substrate, in general and smoothly , bringing values smoothly up or down... This means that over-training is more difficult be- cause the weights of the neurodes do not lock up on some single particular data point in the input signals. Thus the generalization of the substrate encoded agent is superior, as was shown in papers: “Evolving a Single Scalable Controller for


an Octopus Arm with a Variable Number of Segments ” [7] and “Evolving Chart Pattern Sensitive Neural Network Based Forex Trading Agents ” [12].

10.5.2 Genotype Representation

As we saw in Fig-10.7 , the substrate is part of the cortex process. The genotypical specification for the cortex element in DXNN is:

{id,sensors,actuators,cf,ct, type,plasticity, pattern,cids,su_id, link_form,dimensions, densities, generation}

This tuple specifies the substrate dimensionsionality and its general properties through the dimensions and densities elements. Because the sensors and actuators of the substrate are independent of the actual substrate itself, the neurode densities of the substrate, the specification for the “processing hyperlayers ”, the “input hyperlayers ”, and the “output hyperlayers ”, are independent. Though this may at first sound somewhat convoluted, after the explanation you will notice the ad- vantages of this setup, especially for a neural network based system that is meant to evolve and grow.

When I say “processing hyperlayer ” I mean the substrate hyperlayer (2d, 3d … substrate layer of neurodes) that actually has neurodes that process signals. As was noted in the discussion on the substrate, the sensory inputs, which are sometimes multidimensional like in the case of the signals coming from a camera, are part of the substrate, located at the -1 side of the axis defining the depth of the substrate. The output hyperlayers of the substrate are of the processing type. Because the in- put hyperlayers and output hyperlayers need to be tailored for the particular set of sensors and actuators used by the agent, the input hyperlayers, processing hyperlayers, and the output hyperlayers of the substrate, are all specified separate- ly from one another.

So, to create the initial substrate for the agent, the substrate's topology is speci- fied in 3 parts. First DXNN figures out how many dimensions the substrate will be composed of. This is done by analyzing all the sensors and actuators available to the agent. In DXNN, the sensors and actuators not only specify the vector lengths of the signals, but also the geometrical properties (if any) that the signals will ex- hibit. This means that they specify whether the input signals are best viewed or analyzed as a plane with a resolution of X by Y, or a cube of a resolution X by Y by Z, or if there is no geometrical data and that the vector length L of the input signal can be viewed as just a list. If the NN based agent is substrate based, then the DXNN platform will use this extra geometry specification information to cre- ate the substrate topology most appropriate for it. Thus, if the morphology of the seed population being created is composed of 2 sensors and 3 actuators as follows:


sensors:

[#sensor{name=distance_scanner,id=cell_id,format={symmetric,Dim}, tot_vl=pow(Res,Dim), parameters=[Spread,Res,ROffset]} || Spread<-[Pi/2],Res<-[5], Roffset<-[Pi*0/2]] ++ [#sensor{name=color_scanner,id=cell_id,format={symmetric,Dim}, tot_vl=pow(Res,Dim), parameters=[Spread,Res,ROffset]} || Spread <-[Pi/2], Res <-[4], Roffset<-[Pi*0/2], Dim=2],

actuators:

[#actuator{name=two_wheels,id=cell_id,format=no_geo,tot_vl=2,parameters=[]},

  1. actuator{name=create_offspring,id=cell_id,format=no_geo,tot_vl=1,parameters=[]},
  1. actuator{name=spear,id=cell_id,format=no_geo,tot_vl=1,parameters=[]}]

Where the parameters element specifies the extra information necessary for the proper use of the sensor or actuator, and the format element specifies the geomet- rical formatting of the signal. We can see that the actuators all have their formats set to no_geo meaning, no geometric information, so the actuators expect from the substrate single dimensional vector outputs. On the other hand, the sensors both use format= {symmetric,2}, which specifies a two dimensional sensory signal with a symmetric resolution in both dimensions: X by Y where X = Y. The pa- rameters also specify, since these sensors are part of the simulated robot with dis- tance and color sensors, the simulated sensor's coverage area (Spread), camera resolution (Res), and sensor's radial offset from the robot's central line (based on the actual simulation of the robot which is specified during the ALife simulation). Based on the format, the DXNN knows that the sensors will produce two symmet- ric 2d input signals, with a resolution of 5 and 4 respectively. Thus the first senso- ry input will be a 5x5 plane, and the second a 4x4 plane. The DXNN also knows that the actuators expect single dimensional output vectors, the one called two_wheels expects the signal sent to it be a vector of length 2, with the other two actuators expecting the signals sent to them to also be single dimensional lists, vectors, and in this case of length 1 (the length is specified by the tot_vl parame- ter).

Having this information, DXNN knows to expect input signals that will be at least 2d (new sensors might be added in the future, which might of course have higher, or lower dimension), and that the output signals will be 1d. The DXNN thus calculates that the input hyperlayer composed of multiple 2d inputs will be at least 3d (2d planes stacked on a 3 dimension), and the output hyperlayer will be at least 2d (1d outputs stacked on the 2 dimension), which means that the sub- strate must be at least 4d. But why 4d?

Though certainly it is possible to devise substrates whose dimension is the same as the highest dimension of the sensor or actuator used by it, I usually implement a layer to layer feedforward substrate topology which requires the substrate's dimension to be the maximum sensor or actuator dimension, +2 . The reasoning for this is best explained through an example.

rd

nd


Let's say the substrate encoded NN based agent uses 2 sensors, each of which is 2d, and 2 actuators, each of which is 1d. The 2d input planes do not perform any type of processing because the processing is done in the hidden processing hyperlayers, and the output hyperlayer. So both of the 2d input planes must for- ward their signals to the processing hyperlayers. So we must first put these 2d planes on another dimension. Thus, to form an input hyperlayer we first put the 2d planes on the third dimension, forming a 3d input hyperlayer. But for the 3d input hyperlayer to forward its signals to another 3d processing hyperlayer, we need to put both on a 4 dimension. Thus, the final substrate is 4d. The input hyperlayer is 3d. The output hyperlayer, though really only needing to be 2d (due to the output signals being both 1d layers stacked on a 2 dimension to form an output hyperlayer), is also 3d because all neurodes haves to have the same dimensionali- ty.

Fig. 10.9 Input and output hyperlayers composed by stacking the sensor input planes into a single multidimensional input hyperlayer, and stacking the output processing planes into a single multidimensional output hyperlayer with signals destined for actuators.

th

nd


Why give an extra dimension to put the input or output planes on? Because in the future we might want to add more sensors and actuators, and have the sensors and actuators stacked on another dimension makes it easy to do so. For example we would simply add the new sensor based input plane on the same 3 dimension, and scoot the others a bit. In this manner we can add new sensors and actuators indefinitely, without changing the substrate topology too much. Also the coordi- nates of the neurodes in the input planes would change only slightly due to scoot- ing, and so the synaptic weights determined by the NN could be more easily and smoothly adjusted through synaptic weight tuning phase.

This is the gist of the idea when forming substrates dynamically, based on sen- sors and actuators used, and expecting to use multiple such sensors and actuators in the future. We will discuss substrate encoding in much more detail in Chapter- 16.

So, now we know how to compute the dimensionality of the input and output hyperlayers. The number of the processing hyperlayers, if any (in the case where only the input and output hyperlayers exist) is determined by the depth value set by the researcher. In DXNN, the hidden processing hyperlayers, their topology and dimensionality, is set to the resolution equal to the square root of the highest resolution between the sensors and actuators of the agent's morphology.

Thus through this process, when creating the seed population of the substrate encoded NN based agents, DXNN can calculate both the dimension of the sub- strate to create, its topology, and the resolution of each dimension. The resolution of each hidden processing hyperlayer is set to square root of the highest resolution of the signals coming from the sensors or towards actuators. The dimensionality is set, as noted earlier, to the highest dimension between the sensors and actuators, +2. The depth, the number of total hidden processing hyperlayers, is set by the re- searcher, usually to 0 or 1. If it is set to 0, then there is only the input and output (which is able to process the sensory signals) hyperlayers, and 0 hidden processing hyperlayers. When set to 1, the full substrate is composed of the input hyperlayer, the hidden processing hyperlayer whose resolution was computed earlier from the resolution of the sensors and actuators, and the processing output hyperlayer whose dimensionality and topology was formed by analyzing the list of available actuators for the agent, and the list of the actuators currently used by the agent.

For example, the substrate created based on the morphology composed from the following sensors and actuators:

rd


sensors:

[#sensor{name=internals_sensor,id=cell_id,format=no_geo,tot_vl=3,parameters=[]},

  1. sensor{name=color_scanner,id=cell_id,format={symmetric,2}, tot_vl=Density, parameters=[Spread,Res,ROffset]} || Spread <-[Pi/2], Res <-[4], ROffset<-[Pi*0/2]],

actuators:

[#actuator{name=two_wheels,id=cell_id,format=no_geo,tot_vl=2,parameters=[]},

  1. actuator{name=create_offspring,id=cell_id,format=no_geo,tot_vl=1,parameters=[]}]

Will have the hidden processing hyperlayer resolutions set to 2, the dimension- ality set to 4 = 2 +2, and the depth set by the researcher to 1. Since the input and output hyperplanes are created when the genotype is converted to phenotype, and based on the number and types of sensors involved, assuming that in this example the agent is using all the sensors and actuators, the substrate will have the follow- ing form:

Fig. 10.10 The substrate belonging to an agent with 2 sensors and 2 actuators, with the di- mensions = 4, and densities = [1,2,2,2]

Once the substrate and its properties are determined, the actual NN is then cre- ated in a fashion similar to one created when standard direct/neural encoding is used. Only in a substrate encoded NN based system, the sensors and actuators


used by the NN are the substrate_sensors and substrate_actuators, because it is the substrate that is using the real sensors and actuators, while the NN gets its input (coordinates and other neurode parameters) from the substrate, and uses its output signals to execute the substrate_actuators, which set up the synaptic weights (and other parameters) between the neurodes.

In the NNs that use substrate encoding, since it is the substrate that accepts in- puts from the environment and outputs signals to the outside world, and the NN is just used to set the weights for the neurodes in the substrate, the system not only has a set of sensors and actuators as in the standard NN system, but also a set of substrate_sensors and substrate_actuators. The substrate_sensors and sub- strate_actuators are used by the NN in the same way the standard, neural encoded NN uses sensors and actuators, and new substrate_sensors and substrate_actuators are also in the same way integrated into the NN as it evolves.

In the standard substrate encoded NN system, the NN is given an input that is a vector composed of the coordinates of the two neurodes that are connected. In DXNN, the set of substrate_sensors are coordinate processors that process the co- ordinate vectors before feeding the resulting vector signals to the NN. The sub- strate_actuators on the other hand process the NN's output, and then based on their function interact with the substrate by either setting the neurode synaptic weights, changes a neurode's currently set synaptic weights (which effectively adds plasticity to the substrate), or performs some other function.

The DXNN system currently has the following list of substrate_sensors availa- ble for the substrate encoded NNs:

1. none: Passes the Cartesian coordinates of the connected neurodes directly to the NN.

2. cartesian_distance: Calculates the Cartesian distance between the neurodes, and passes the result to the NN.

3. polar_coordinates (if substrate is 2d): Transforms the Cartesian coordinate vec- tor to the polar coordinate vector, and passes that to the NN.

4. spherical_coordinates (if substrate is 3d): Transforms the Cartesian coordinate vector to the spherical coordinate vector, and passes that to the NN.

5. centripetal_distance : Transforms the Cartesian coordinate vector from the con- nected neurodes into a vector of length 2, composed of the distances of the two neurodes to the center of the substrate.

6. distance_between_neurodes : Calculates the distance between the two connect- ed neurodes, and passes that to the NN.

This set of substrate_sensors further allows the substrate encoded NN to extract the geometrical patterns and information from its inputs, whatever dimension those input signals have. An example of an architecture of a substrate encoded NN using multiple substrate_sensors and multiple substrate_actuators, with the sub- strate itself using multiple sensors and actuators as well, is shown in Fig-10.11 .


Fig. 10.11 A substrate encoded NN using different types of substrate_sensors and sub- strate_actuators, and standard sensors and actuators.

As can be seen from the figure, it is also possible to have different types of sub- strate_actuators, not just the standard synaptic_weight substrate_actuator which uses the NN's output to set the synaptic weight between the two neurodes based on their coordinates which were passed to the NN's substrate_sensors. The stand- ard substrate_actuator, synaptic_weight setter, is one that simply uses the signal coming from the NN and converts it into a synaptic weight value using the algo- rithm shown in Listing-10.1. In this listing, the substrate_actuator simply takes the NN's output, and computes the synaptic weight to be 0 if the NN's output is be- tween -0.33 and 0.33, and to be between -1 and 1 otherwise, normalizing the syn- aptic weight value such that there is no hole between -0.33 and 0.33 when using this below shown function:

Listing-10.1 The simple synaptic weight setting “substrate_actuator ”.

set_weight(Output)->

[Weight] = Output,

Threshold = 0.33,

Processed_Weight = if

Weight > Threshold ->


(functions:scale(Weight,1,Threshold)+1)/2;

Weight < -Threshold ->

(functions:scale(Weight,-Threshold,-1)-1)/2;

true ->

0

end.

Currently there are a number of other substrate_actuators implemented as well. For example a secondary substrate_actuator called synaptic_expression, decides on whether there is a connection between the two neurodes at all, if there isn't then the weight is set to 0. This is different from the weight being set to 0 by the synaptic_weight actuator, since using this secondary actuator the whole substrate can be made more complex, there can be two different neural circuits, one decid- ing on the synaptic weight, and one deciding on the connectivity pattern of the substrate. Or for example instead of using the synaptic_weight actuator, an itera- tive_plasticity , abc_plasticity, or some other learning algorithm can be used. Us- ing these plasticity substrate_actuators, the NN can change and modify the synap- tic weights after every sensory input is processed. One substrate_actuator could be mutated into another during the topological mutation phase, new ones could be added or removed throughout evolution.

These substrate_actuators further allow one to experiment with different types of learning, adding more agility and robustness to the population and individual agents, providing a greater leverage to evolution to overcome various discontinui- ties and abstractions on the fitness landscape. Combined all together, with the var- ious substrate specific mutation operators which increase the resolution/density of the substrate, add new sensors and actuators, add new substrate_sensors and sub- strate_actuators... the substrate encoding provided by the DXNN system is one of the most advanced substrate encoded neuroevolutionary approaches currently available.

The resolution and dimensionality of the substrate can be further mutated dur- ing the topological mutation stage. When the agent is substrate encoded, the plat- form's standard mutation operator list is further augmented to include the follow- ing substrate specific mutation operators:

1. mutate_resolution

2. mutate_dimensionality

Yes the method and representation is convoluted and could be made simpler. The problem with DXNN, as noted earlier, is that it was built up slowly, evolving through many of my various experiments and tests. And as we know, evolution does not take the cleanest path from genotype A to genotype Z, instead it is all based on the easiest and most direct path, which is based on the agent's environ- ment, and most easily achievable niche based on the agent's genotype/phenotype at that time. Here too, the DXNN is the way it is because of the order in which I got the ideas for its various parts, and the initial, though at times mistaken, repre-


sentations and implementations I used. Once a few hundred or thousand lines of code are written, the amount of motivation to recreate the system in a cleaner manner decreases. But now that we are creating a completely new TWEANN sys- tem together, and have the knowledge of my earlier experience within the field and systems like DXNN to guide us, we can create our new system with foresight, without having to go down the same dark alleys and dead ends I wondered into during my first time around.

10.5.3 Substrate Phenotype Representation

The conversion of genotype to phenotype is similar to one used by the standard direct encoded NNs in DXNN, and thus is similar to what we use in the system we've built so far. As we discussed, in DXNN the cortex process is not a synchro- nizer but instead is the signal gatekeeper between the NN and the sensors and ac- tuators it itself is composed of. In the substrate encoded NNs, the cortex also takes on the role of the substrate itself. In DXNN, the entire [substrate, cortex, sensors, actuators, substrate_sensors, substrate_actuators] system is represented as a sin- gle element/process, because it is possible to encode the substrate in a list form and very efficiently perform calculations even when that substrate is composed of thousands of neurodes.

When the exoself generates and connects all the elements (neurons and the cor- tex), it does so in the same way it does with the direct encoded NN system. Since the cortex knows, based on its parameters, that it is a substrate encoded system, once it is created it builds a substrate based on dimension, densities, sensor, actuator, substrate_sensor, and substrate_actuator list specifications. The neurons and the NN that they compose neither know nor need to know that the agent is substrate encoded. In both versions, the direct encoded and the indirect encoded NN system, the input and the output layer neurons are connected to the cortex, so nothing changes for them. The cortex is the one that needs to keep track of when to use the substrate sensors/actuators, and when to use the actual sensors/actuators.

The algorithm that the substrate encoded cortex follows is specified in the fol- lowing steps, with a follow-up paragraphs elaborating on the more intricate parts.

1. The cortex process is spawned by the exoself, and immediately begins to wait for its initial parameters from the same.

2. The cortex receives all its InitState in the form:

{ExoSelf_PId,Id,Sensors,Actuators,CF,CT,Max_Attempts,Smoothness,OpMode,Type,

Plasticity, Morphology,Specie_Id,TotNeurons,Dimensions,Densities}

3. The cortex checks the agent Type, whether it is neural or substrate. In the steps that follow we assume that the Type is substrate .


4. Cortex constructs the substrate:

5. The cortex reads the number of dimensions, and the densities.

6. The input hyperlayer is built based on the sensors the agent uses, with the neurode coordinates based on the number of dimensions (If the entire sub- strate has 3 dimensions, then each coordinate is [X,Y,Z], if 4d then [X,Y,Z,T]...).

7. If depth > 0 , then hidden processing hyperlayers are constructed based on the densities and dimension specified, and with each neurode in the first hidden processing hyperlayer having the right number of synaptic weights to deal with the input hyperlayer.

8. The output processing hyperlayer is constructed, and each neurode must have the right number of synaptic weights to deal with the signals coming from the hidden processing hyperlayers.

9. The cortex combines the input, processing, and output hyperlayers into a single hypercube substrate.

10. DO Sense-Think-Act loop:

11. DO For each neurode in the substrate:

12. The cortex goes through the substrate_sensors, using the tuples like in the standard sensors, to forward the neurode properties (coordi- nates, and other parameters based on the substrate_sensor used) to the connected neurons in the NN.

13. The output signals of the NN are then used to execute the sub- strate_actuators to set the synaptic weights and other parameters be- tween the neurodes in the substrate.

UNTIL: All neurodes have been assigned their synaptic weights and other parameters.

14. The cortex goes through every sensor, and maps the sensory signals to the input hyperlayer of the substrate.

15. The substrate processes the sensory signals.

16. The output hyperlayer produces the output signals destined for the actua- tors. Since the output hyperlayer is created based on the actuators the agent uses, the output signals are implicitly of the right dimensionality and in the right order, such that the signals are packaged into vectors of proper lengths, and are then used as parameters to execute the actuator functions.

17. The cortex goes through every actuator, executing the actuator function using the output signals produced by the substrate as the parameter of their respective actuators.

UNTIL: Termination condition is reached, tuning has ended, or interrupt signal is sent by the exoself.

During the tuning phase, after every evaluation of the NN, the exoself chooses which neurons should perturb their synaptic weights. After the neurons in the NN have perturbed their synaptic weights, the cortex takes the substrate through the


step 11 loop, updating all the synaptic weights of the neurodes in the substrate by polling the NN for weights.

Thus the cortex first executes all the sensor functions to gather all the sensory signals, then it goes through every neurode in the substrate, until the processing output hyperlayer produces the output signals, which the cortex gathers, packages into appropriate vectors, and executes all the actuators in its actuator list with the appropriate output vector signals.

The phenotypic architecture of the substrate encoded NN based agent, com- posed of the Exoself, Cortex, and Neuron elements, with the Sense-Think-Act loop steps specified, is shown in Fig-10.12 .

Fig. 10.12 The phenotypic architecture of the substrate encoded NN based agent, composed of the concurrent Exoself, Cortex, and Neuron processes, with the processing steps listed.

Let's quickly go over the shortened processing loop shown in the above figure. 1. The exoself creates the cortex.

2. The exoself sends the created cortex its InitState parameters.

3. The cortex creates the substrate based on sensors, actuators, and other specifi- cations.

4. The cortex/substrate uses the substrate_sensors to forward to the NN the coor- dinates and other parameters of the connected neurodes within the substrate.

5. The NN processes the signals passed to it by its substrate_sensors.


6. The substrate_actuators and the signals produced by the NN, used as parame- ters for the substrate_actuators, are used to set the synaptic weights and other parameters of the embedded neurodes.

7. The cortex gathers the sensory signals from its sensors.

8. The cortex maps the sensory signals to the substrate's input hyperlayer.

9. The substrate processes the sensory signals coming from the input hyperlayer. 10.The cortex maps the output signals of the neurodes in the output hyperlayer to

their appropriate actuators.

11.The cortex executes the actuators with the substrate produced output signals as their parameters.

The substrate, due to it being a single process, and capable of being composed of millions of neurodes each with millions of connections, and because each neurode simply does vector multiplication, is a perfect candidate for being accel- erated with a GPU. Substrate encoding is an important field of neurocomputation, it allows for very large NNs to be constructed, for neuroplasticity and geometrical pattern sensitive systems to be composed, and in general substrate encoded NNs are more effective, and perhaps with some new topological structure and with fur- ther expansions, might be the path toward general computational intelligence.

Have you ever seen a PET scan? You know that activity pattern that it shows? It is difficult not to look at the NN computing the synaptic weights and therefore activity pattern on the substrate, as the tool which could carve out that high densi- ty, and highly complex architecture. With a substrate having enough neurons (100 billion let's say), and with the NN, the universal function approximator, having the right function, it could possibly carve out the architecture and the activation patterns similar to something one would see in a PET scan... But we are not at that point just yet.

We will add substrate encoding capabilities to the TWEANN system we are developing together, and thus we will discuss further the algorithms and a way to represent the substrate in great detail. We will of course, having foresight, develop our system to have a more concise and flexible representation. As we develop the next generation TWEANN in this book, we will avoid making the mistakes I made when I first developed the architecture of DXNN.

In the following sections we will discuss the current and ongoing projects that DXNN is being used for, and thus what the system we're developing here (which will replace DXNN, by becoming the new DXNN) will be applied to once devel- oped. The system we're creating here is meant to supersede and replace DXNN, it is the next generation of a fully concurrent, highly general and scalable, Topology and Parameter Evolving Universal Learning Network (TPEULN).

10.6 DXNN Research Group & the NN Research Repository

DXNN Research group [8] is currently a small research group composed of a few mathematicians, computer scientists, and me. We are working on further ex- panding the DXNN platform, and finding new projects to apply it to. One of these projects is the application of DXNN to Cyberwarfare. Another deals with ex- changing the neuron elements with logic gates and transistors, so that the platform can be applied towards the evolution and optimization of large scale digital cir- cuits. The currently explored application of DXNN is towards the evolution and optimization of OpenSPARC [9], some progress has been made, but not enough to publish. The DXNN Research group is also currently working on interfacing the DXNN with the Player/Stage/Gazebo [5] project, allowing it to be used in 3d ALife experiments, and the evolution of robotic systems and neurocontrollers for the same. The Player/Stage/Gazebo robot simulators provide 2d and 3d simulation environments, and the drivers to interface the evolved NNs with actual hardware. The use of Player gives us the ability to evolve systems in artificial environments, and immediately have the ability to apply them to real hardware, and thus usable and applicable in the real world. The current main project and interest in this area is the evolution of neurocontrollers for the control of Unmanned Combat Ariel Vehicles (UCAVs). This is accomplished through the co-evolution of two, forever warring, populations of Combat UAVs in the 3d simulated environment, through Gazebo for example. Due to the use of the Player interface, we can then transfer the evolved intelligence to real UCAV systems.

The main reasons why we are trying to create a highly decoupled neuroevolutionary system is because it will allow us to easily augment it, and then provide it to the public so that crowdsourcing is used to further expand the plat- form, letting anyone with interest and skill to contribute various modules and computational packages to the system, further expanding and augmenting it, mak- ing it more general, and applicable to new projects, which benefits the entire community using the TWEANN system. DXNN Neural Network Research Repos- itory [10] provides the specifications on how to add new modules to the DXNN TWEANN, where to submit them...

The goal of the Neural Network Research Repository (NNRR) is also to be- come the repository of neural network systems evolved through the DXNN sys- tem. NNs are by their very nature blackbox systems, different neural networks can be evolved to solve the same problem, or inhabit same environments (when NN based agents are used in ALife). NNRR provides a place where individuals can submit the NN systems they have evolved, and specify the fitness functions and other parameters they used to evolve these agents. Because everyone else on the NNRR is also using DXNN, they can then try to see what types of NN topologies they can evolve given the same fitness function and TWEANN parameters. Thus, the NNRR should over time accumulate useful NN based agents. Those who wish to simply start using these agents can do so, others can try to download the hun-


dreds of the already evolved NN based systems for some problem, and try to data- mine their topologies, try to see what are the essential parts of these NN based sys- tems, what are the common threads? Through this approach we can try to start building a path towards illuminating the blackbox. These types of databases also provide the data needed to figure out where the DXNN system is perhaps having difficulties when solving problems.

Finally, with the standardized interfaces between the various processes, and with the specified genotypical encoding system, the community can contribute the various activation functions, neural plasticity rules, neuron types, substrate topol- ogies, fitness functions, selection functions... Every decoupled element is a self contained module, and thus anyone can augment the DXNN system by simply conforming to the proper interface specifications. The NNRR will propel us, and allow for the capabilities and applicability of this neuroevolutionary system to ex- pand dramatically, making the evolved systems available globally, providing al- ready evolved solutions to those interested, and giving a place for researchers to contribute, while at the same time giving them a place where they can gather tools and data for their own further research.

10.7 Currently Active Projects

The DXNN research group is currently actively pursuing three projects:

1. Cyberwarfare.

2. Coevolution of Unmanned Ariel Vehicle Combat Maneuvers.

3. CPU Evolution and Optimization.

When successful, the results of these 3 projects could potentially be game changing for the industrial and military sector.

10.7.1 Cyberwarfare

One of the exciting applications the DXNN platform is currently being applied toward is the evolution of offensive and defensive cyberwarfare agents. We are currently trying to evolve agents capable of using parameterized metasploit (a penetration testing program) and other tools to effectively penetrate and attack other machines, and agents capable of defending their host network against such attacks, by monitoring signals on its network for attacks being carried out against it, and then using various available tools and methods to thwart and counterattack. This is done by creating scapes, simulated network environments using network simulators like NS3, with simulated host targets, and then interfacing the NN based agents with programs like metasploit, letting them evolve the ability to


combine the various attack commands to penetrate the simple hosts. With regards to the evolution of defensive agents, the NN based agents are fed signals coming from the ports, and they are required to make a decision of whether they are being actively attacked or not. If they are, they must decide on what they should do, lock the port, fully disconnect, counter-attack...

There are a number of difficulties in evolving cyberwarfare agents, because un- like in the natural environments, there are no smooth evolutionary paths from simply existing on a network, to being able to forge attack vectors using metasploit. Neither is there a smooth evolutionary path leading from mere exist- ence, to the ability to detect more and more complex attacks being carried out against your own host. In standard ALife, there is an evolutionary path from simp- ly running after a prey and then eating it, to trying different approaches, hiding, baiting the prey... it's all a smooth progression of intelligence. That is not the case in cyberwarfare, things are more disconnected, more arcane, requiring beforehand knowledge and experience. Nevertheless, through bootstrapping simple skills, and forging fitness functions, our preliminary results have demonstrated that the goals of this project are achievable.

10.7.2 Coevolving Unmanned Ariel Vehicle Combat Maneuvers

Another exciting application and field where evolved neurocognitive systems can provide a significant advantage is of course robotics. As with cyberwarfare, there is a significant amount of both industrial and military applications, with the successful system and implementation being potentially game changing. Due to the current increased use of unmanned aerial vehicles, particularly in combat, there is a great opportunity in evolving neural network agents specifically for con- trolling such systems. At the moment the UAVs are programmed to scout, or fly to particular way-points. Once the UAV gets there, a real pilot takes over. The pilot sits somewhere in the base and controls the UAV, looking at the screen which is fed by the UAV's camera. This of course provides a much lower level of situa- tional awareness to the pilot when compared to that available when sitting in a cockpit. Also, the maneuvers available to the drone are limited by the human op- erator, and the time delay in the connection due to the distance of the UAV from the human operator. All of this combined, puts the Unmanned Combat Ariel Vehi- cle (UCAV) at a disadvantage in a standard dogfight against a piloted fighter jet. Yet a UCAV can undertake g forces and perform maneuvers that are impossible for a human piloted jet fighter. Furthermore, an evolved NN would be able to in- tegrate the signals from many more sensors, and make the decisions faster, than any biological pilot can. Thus, it is possible for the UCAVs to have performance levels, precision levels, situational awareness, and general capabilities that far sur- pass those of pilots and piloted jets.


This can be mitigated by evolving NN based agents specifically for controlling UCAVs, allowing the NN systems to take full advantage of all the sensory data, and use the UCAV to its full potential with regards to maneuverability. I think that this would give the drone an advantage over standard manned aerial vehicles. To evolve such NN based agents we once again do so through an ALife coevolutionary approach. As discussed in the “Motivations and Applications ” chapter, by creating a detailed simulation through a simulator like Gazebo, and creating the simulated UCAVs with high enough detail, and a set of prepro- grammed or even evolving fighter jet simulations constrained to the physical lim- its of the pilot, it is possible to coevolve UCAV controlling NN systems. To ac- complish this, we can put two populations of forever warring UCAVs into a simulated 3d environment, to coevolve the ever more intelligent digital minds within. This, as in the Predator Vs. Prey [11] simulations, will yield ever more creative NN based agents, evolving neurocontrollers with innovative combat ma- neuvers, and having the ability to use the full potential of unmanned combat air- craft, the full potential of metal that is not limited by flesh.

The preliminary testing in this project has started. At the time of this writing, the interface between the DXNN platform and the Player/Gazebo has been devel- oped, and the work is being concentrated on developing simulations of the UCAVs which are modular enough to allow for morphological evolution. Based on the performance of DXNN in ALife, there seems to be no reason why it would not evolve highly adaptive, flexible, and potent UCAV piloting agents.

10.7.3 Evolving New CPU Architectures & Optimizing Existing Ones

The third project currently being pursued by the DXNN research group, deals with the DXNN platform being applied to the evolution and optimization of digital circuits. Because the neurons in the evolving NN topologies can have any type of connections and activation functions, the DXNN platform does not in reality evolve NNs, but Universal Learning Networks, where the nodes can be anything. In this particular application, the nodes use logic operators and transistor simula- tions as activation functions, thus the evolved topologies are those of digital cir- cuits.

The OpenSPARC project provides the whole architecture and topology of the OpenSPARC T2 CPU, which our team is hoping to take advantage of. The goal of our project is composed of two parts. 1. Create the tuple encoded genotype of a system which recreates the OpenSPARC T2 architecture, and then through its mu- tation operators (complexifying and pruning), optimize the CPU, by reducing the number of gates used while retaining the functionality. 2. By specifying particular goals through the fitness function, such as increased throughput, higher core count

10.9 References

coherency, and certain new features, evolve the existing architecture into a more advanced one.

Because OpenSPARC T2 also provides a testing suit, it is possible to mutate the existing architecture and immediately test its functionality and performance. But due to the architecture's high level of complexity, the project is still in the process of having new mutation operators being developed, the fitness functions being crafted for optimization and evolution of the CPU, and the creation of the genotype representing the OpenSPARC-T2 architecture. DXNN has been used to evolve and optimize much smaller digital circuits, which gives hope that it can successfully be applied here as well. The potential payoffs could be immense, im- proving and optimizing CPUs automatically, and adding new features, would rev- olutionize the landscape of this field. At the moment, we are only beginning to scratch the surface of this project.

10.8 Summary and Future Work

In this chapter we have discussed the DXNN Platform, a general Topology and Weight Evolving Artificial Neural Network system and framework. I briefly ex- plored its various features, its ability to evolve complex NN topologies and its par- ticular approach to the optimization of synaptic weights in the evolved NN topol- ogies. We discussed how DXNN uses the size of the NN in the determination of how long to tune the new synaptic weights, which synaptic weights to tune, and which NNs should be allowed to create offspring and be considered fit. We have also discussed the substrate encoding used by the DXNN, which allows it to very effectively build substrates composed of a very large number of neurodes.

Finally, we have went into some detail discussing the DXNN Research group's current projects. The Neural Network Research Repository, the Cyberwarfare pro- ject, the Combat UAV project, and the CPU Evolution project. DXNN is the first neuroevolutionary system built purely through Erlang, and which was designed from the very beginning to be implemented only in Erlang. Without Erlang, some- thing as complex, dynamic, and general as this neuroevolutionary platform, could not be created by a single individual so easily. There is an enormous room for growth and further improvement in this system. And it is this that you and I are working on in this book, we are building the next phase of DXNN.


[1] DXNN's records.hrl is available at: https://github.com/CorticalComputer/DXNN

[2] Sher GI (2010) Discover & eXplore Neural Network (DXNN) Platform, a Modular TWEANN. Available at: http://arxiv.org/abs/1008.2412


[3] Gauci J, Stanley KO (2007) Generating Large-Scale Neural Networks Through Discovering Geometric Regularities. Proceedings of the 9th annual conference on Genetic and evolution- ary computation GECCO 07, 997.

[4] Siebel NT, Sommer G (2007) Evolutionary Reinforcement Learning of Artificial Neural Networks. International Journal of Hybrid Intelligent Systems 4, 171-183.

[5] Player/Stage/Gazebo: http://playerstage.sourceforge.net/

[6] Risi S, Stanley KO (2010) Indirectly Encoding Neural Plasticity as a Pattern of Local Rules. Neural Plasticity 6226, 1-11.

[7] Woolley BG, Stanley KO (2010) Evolving a Single Scalable Controller for an Octopus Arm with a Variable Number of Segments. Parallel Problem Solving from Nature PPSN XI, 270- 279.

[8] DXNN Research Group: www.DXNNResearch.com

[9] OpenSPARC: http://www.opensparc.net/

[10] DXNN Neural Network Research Repository: www.DXNNResearch.com/NNRR

[11] Prdator Vs. Prey Simulation recording:

http://www.youtube.com/watch?v=HzsDZt8EO70& feature=related

[12] Sher GI (2012) Evolving Chart Pattern Sensitive Neural Network Based Forex TradingAgents. Available at: http://http://arxiv.org/abs/1111.5892 .

Part IV

Advanced Neuroevolution: Creating the

Cutting Edge

The TWEANN system that we've built so far is a good start, it is clean, direct, with great potential. We have tested it on the simple XOR problem, and it does work. We have discussed that what we are building here is the new DXNN, one that is better than the original version in every way. In this part we will begin add- ing new advanced features to the TWEANN we've created thus far, chapter by chapter, increasing its capabilities. We will develop a TWEANN that will be the contender of the bleeding edge in this field.

In the following chapters we will first decouple our TWEANN system, in the sense that we will allow for the different features and functionalities of our plat- form to be held and specified through their own functions and modules. In this manner we can then put our TWEANN system online, as open source, and allow others to concentrate and add various new features and functions (selection func- tions, activation functions, plasticity functions, mutation operators...), without having to worry about modifying, integrating, and in general dealing with the rest of the code. In this way, contributors can just concentrate on particular aspects of the TWEANN, without digging through the rest of the source code.

Then we will modify the population_monitor module so that it can keep track of the evolutionary statistics of the population it is evolving. Afterwards we add the benchmarker module, a new process that can be used to perform experiments by performing multiple evolutionary runs of some particular problem, and then computing averages, standard deviations, max & min values of the various popu- lation parameters, and building graphable files. To actually test the performance of our system after adding these new features, we will require problems more com- plex than the simple XOR mimicking one, thus in Chapter-14 we add two stand- ard, more complex problems. In Chapter-14 we implement the T-Maze navigation problem, and a few variations of the Pole Balancing problem.

Having built all the necessary tools to move forward and be able to keep track of our system and test the new features, we advance and add plasticity to the evolved neural networks. We implement everything from the standard Hebbian plasticity of various forms, to Oja's rule, and neuromodulation. Afterwards, we make a significant leap and add indirect encoding to the type of NN based agents our TWEAN can evolve. The particular indirect encoding we add is that of sub- strate encoding . Afterwards, we add substrate plasticity, all the while testing the performance and capabilities of our TWEANN on the new problems we've added earlier. By the time we add substrate plasticity, we have developed a highly ad- vanced TWEANN platform, capable of evolving advanced NN based agents with plastic and static networks, different learning algorithms and rules, numerous acti- vation functions, highly dynamic mutation operators, and different types of encoding.

Chapter 11 Decoupling & Modularizing Our Neuroevolutionary Platform

Abstract In this chapter we modify the implementation of our TWEANN sys- tem, making all its parts decoupled from one another. By doing so, the plasticity functions, the activation functions, the evolutionary loops, the mutation opera- tors... become independent, each called and referenced through its own modules and function names, and thus allowing for our system to be crowd-sourced, letting anyone have the ability to modify and add new activation functions, mutation op- erators, and other features, without having to modify or augment any other part of the TWEANN. This effectively makes our system more scalable, and easier to augment, advance, and improve in the future.

In Chapter-10 we discussed DXNN, and the Neural Network Research Reposi- tory. I mentioned that it becomes extremely useful and necessary to decouple the TWEANN platform, because it makes it that much easier to later on expand and improve it. Our system already has a number of interesting features that make it somewhat decoupled. For example the activation functions (AFs) that the neurons use, are independent of the neurons themselves. We need only provide the name of the activation function, and if such function exists, the neuron accesses it through functions:ActivationFunctionName(...). We could take this same approach with regards to other features of our TWEANN. As long as we specify a standard interfacing format with those functions, new modules and functions can then be added. Not only would it make the system more modular and upgradeable, but al- so make those features, where a choice in the use of a particular function is pre- sent, mutatable and evolvable . Anything that can be specified in the manner in which we specify activation functions, which provides us with a list of available activation functions, can be used in evolution because it gives us the ability to switch between the functionalities, between the function names available in that list.

For example neural plasticity, which we will discuss later on, is the ability of the neuron to adapt and change/learn based on the signals it is processing. This is not training or tuning, this is true learning, in the same manner that biological neu- rons change and develop additional receptors at a particular place on the dendrite (which is somewhat similar to increasing/decreasing the synaptic weight for some connection)... We can change the neuron to also have a place for plasticity functions. A list like we have for activation functions, can then become available. As soon as new plasticity approaches are developed and added to the plasticity module, the tag/name of the said plasticity function can be added to the plasticity_functions list. They then would become immediately available as mutatable and thus evolv- able features for future and existing neurons. Then, when adding new neurons dur-

DOI 10.1007/978-1-4614- 4463 - 3_11, © Springer Science+Business Media New York 2013


ing evolution to the already existing NN, this approach could give those newly added neurons a chance to use Hebbian [1], or Oja's [2] plasticity functions, as long as the species' constraints allows for it. By adding mutate_plasticity mutation op- erator, we can also allow the already existing neurons to have a chance of mutat- ing, and using the newly added plasticity functions from the available list of said functions.

Another thing that becomes possible when we completely decouple the various features of our TWEANN system, is allowing species to use a different selection function and a different evolutionary loop approach. In this manner the TWEANN can then be easily customizable, and use steady-state evolution, or generational evolution... on different species, depending on just the selection of that particular tag/name of the selection and evolutionary loop function. And it allows us to change between these various evolutionary strategies and approaches mid- evolution.

In the following sections we are first going to analyze which parts and which elements can have their various parameters and functionalities modularized and decoupled. Then, we will modify our existing system to use this new decoupled approach, and develop a modified genotype encoding to incorporate the new fea- tures we will implement.

11.1 What Can be Decoupled?

For the sake of convenience, Fig-11.1 shows again the visualization of our neuroevolutionary platform's architecture from Fig-8.1 . The functional elements in the system we've developed so far are as follows: [polis, scape/sim, popula- tion_monitor, agent/exoself, cortex, neuron, sensor, actuator, genome_mutator] . The functional elements and the processes that have features which can & should be changed are numerous, and we need to figure out what they are. Let's analyze every one of these processes and parts of our neuroevolutionary system, and see what types of features it contains, and whether those features can and should be changed in some way, similar to the activation function example I keep mention- ing. I will put an asterisk in front of those parts for which things could be further decoupled.

Polis: This program and system is simply the general monitoring program, in charge of the different scapes, there is nothing yet that we can modify or add to it.

Scape: Each scape, each simulation, is independent in its own right, so there is nothing to decouple here at the moment.

  • population_monitor: There are numerous features that can be decoupled in the population_monitor system. Selection_Algorithm for example, currently we have two, competition and top3. These are chosen through the case statement, but


instead we could create a new selection_algorithm module, put all the different types of selection algorithms there, specify the particulars and the format that the selection algorithms created in the future must abide to work with the system, and then call these selection algorithms through the use of Mod:Fun(...) . Another ele- ment is the max_attempts number, which basically specifies if the population should be evolved through a genetic algorithm, by setting the max_attempts = 1 , or memetic algorithm, in which case we set max_attempts > 1 . The popula- tion_monitor process is also the one that decides on whether to evolve the popula- tion through steady-state, or through the generational evolutionary loop.

  • genome_mutator: The mutation operators are already in the list form, and new mutation operators can be added to the module to let the genome_mutator use them during the topological mutation phase. But there is another element that can be, but is not yet, decoupled: the manner in which the mutation operators are cho- sen and the percentage with which they are chosen. There can be different func- tions, different ways of choosing the mutation operators, and with different proba- bilities. We can have an approach that uses only complexifying mutation operators, another that uses all available mutation operators, and each of those can either choose the mutation operators with the same probability, or each mutation operator with its own probability. These parameters specify our system's evolu- tionary strategy, and these parameters too can be mutated, and evolved.
  • Exoself: Exoself has a number of jobs that can be decoupled. For example, how do we decide what the MAX_ATTEMPTS value should be? This is the num- ber of times we allow the exoself to fail to improve the fitness of the NN during tuning. This value should perhaps be proportional to the number of the weights in the neurons to be tuned, or perhaps it should be a static value... There are many ways that this value can be calculated, and because there are numerous ways and we do not know which is most effective, it is a good feature to decouple. This will allow people to create new tuning_duration functions, and test and compare them against each other. Another thing that exoself needs to derive is which neurons to choose, how to choose them, and how many of them to choose for synaptic weight perturbation. How to choose those neurons? Again, numerous approaches and functions can be viable, and so it should be made as simple as possible to create and try out different tuning_selection_strategy functions. Another thing that can be decoupled is the actual synaptic weight perturbation intensity. Currently this value is a constant, specified through: -define(DELTA_MULTIPLIER, math:pi()*2, in the neuron module. This means that the perturbation values are chosen to be between -Pi and Pi , but perhaps that is not the best way to go about it. Perhaps simulated annealing would provide a benefit. The neuron's “age ”, the longer neurons are untouched by mutation operators the more “stable ” that we can consider them. In other words, if a NN performs better when some set of neurons is not mutated or perturbed, then it means that this set of neurons has stabilized, works well the way it is, and should continue being left alone in the future. Per- haps the perturbation_intensity , the range of the perturbation possible, should be dependent on the stability and age of the neuron to which it is applied. The more


stable the neuron, the lower the perturbation intensity applied to it. In this manner, the new neurons will have a chance of having their synaptic weights perturbed with full strength, giving those synaptic weights a chance to jump all the way from -Pi to Pi. While at the same time, the already existing and more stable neural cir- cuits, those that have shown to produce a stable cognitive system, or a module of a cognitive system, will have the perturbation intensity that is lower. The synaptic weight perturbations should perhaps be inversely proportional to the age of the neuron they are applied to. Finally, what about the actual local search function, should it be the augmented stochastic hill climber that we're using? Or perhaps some variant of ant colony optimization , or simulated immune system ?... Again, if we decouple this feature, then others can easily form new local optimization func- tions, which the system can then use, and flip/mutate between.

  • neuron: The neuron too has a few things that can be decoupled. The activa- tion functions are already decoupled, the neuron uses the name of the activation function to produce an output. But perhaps the manner in which the inputs are analyzed should be decoupled. Should we use a dot product? Or something else, perhaps a diff function which calculates the difference between the previous input and the current input. Should there be plasticity? Which too can be specified by a name of the function stored in the plasticity_function module. What about prepro- cessing of the input vectors, should normalization be used? Would it add any- thing? What about weight saturation, should it be constant as is currently in our system, specified in the neuron module by: -define(SAT_LIMIT,math:pi()*2). Or perhaps saturation limit should be dependent on the activation function? Finally, what about output and input saturation? Should that also be present and be based on some researcher specified function? All of these things could be decoupled.

cortex: Cortex is a synchronization element, and so at the moment there is nothing that can be decoupled with regards to its functionality.

  • sensor: Sensors and actuators are already represented by their very own func- tions, belonging to their very own modules. Each sensor, based on its name, does the preprocessing. Every actuator does postprocessing of the signals sent to it. Though we could add to the sensor records a parameter element, which could fur- ther specify the type of pre-processing that should be done. In this manner we could have the same sensors whose functionality differs only in the way in which the data is produced, their vector lengths, the type of pre-processing that is con- ducted, or the format in which the data comes. Is the data simply a vector? Or is the data coming from a camera, and is arranged in a 10 by 10 grid, and is thus in possession of geometrical information. In this way, if for example the NN based agent is substrate encoded, it could then, when evolving or being seeded, use the sensor that also provides geometrical information. While a neural encoded NN based agent would request from the sensor of the same name, the data to be pack- aged in a simple vector form.
  • actuator: As is the case with the sensor elements, the actuators too can have the parameter element in their records. The parameter could then differentiate the same actuators by the different types of post-processing they conduct, for example.

11.2 Updating the Genotype Representation

Fig. 11.1 The general architecture of our neuroevolutionary platform.

Certainly other things that should further be decoupled will turn up, as we con- tinue developing and advancing our TWEANN. But for now, the modifications based on these observations will make our system more general, more modular, and more future proof. Though of course there is a downside to it as well, these modifications will also make our system a tiny bit slower, and a bit more complex if we are not careful in the way in which we implement them.


The elements that make up the genotype of our NN based agents are all in the records.hrl file, shown bellow for convenience:

-record(sensor,{id,name,cx_id,scape,vl,fanout_ids=[],generation}).

-record(actuator,{id,name,cx_id,scape,vl,fanin_ids=[],generation}).


-record(neuron, {id, generation, cx_id, af, input_idps=[], output_ids=[], ro_ids=[]}). -record(cortex, {id, agent_id, neuron_ids=[], sensor_ids=[], actuator_ids=[]}).

-record(agent,{id, generation, population_id, specie_id, cx_id, fingerprint, constraint,

evo_hist=[], fitness, innovation_factor=0, pattern=[]}).

-record(specie,{id, population_id, fingerprint, constraint, agent_ids=[], dead_pool=[], champion_ids=[], fitness, innovation_factor=0}).

-record(population,{id,polis_id,specie_ids=[],morphologies=[],innovation_factor}).

-record(constraint,{

morphology=xor_mimic, %xor_mimic

connection_architecture = recurrent, %recurrent|feedforward

neural_afs=[tanh,cos,gaussian,absolute] %[tanh,cos,gaussian,absolute,sin,sqrt,sigmoid]

}).

We need to modify the records for each of these elements such that we can then begin updating the source code to incorporate the decoupled form discussed in the previous section.

11.2.1 The sensor & actuator Records

We modify the sensor and actuator elements first. To do so, we need to modify the records that encode them, making the records also include the parameters el- ement that will further augment their processing ability. Also, we add the format element to the record, which specifies in what format the signals are to be fed to the NN or in what format the NN should feed its output signals to the element (ac- tuator for example). This will allow us to specify whether the input is a standard vector, or whether it is a 2d or 3d data structure, for example an image coming from a camera. This will allow the substrate encoded NN systems to take ad- vantage of this extra information and appropriately set up the input hyperplanes (a structure used in the substrate encoded NN systems, discussed in a later chapter). Keeping an eye on the future expansions and modifications of our system, and the fact that we will at some point apply the NN based agents to ALife or robotics simulations and problems, we should also perhaps add elements that specify the representation-particulars of the sensor and actuator systems, the manner in which they are to be represented in the simulated environment. This will allow the sen- sors and actuators to be integrated and evolved by the artificial organisms and simulated robots, allowing the simulator to simulate the visual and physical repre- sentation of the sensors through their own physical and visual representation pa- rameters. Thus we also add the elements: phys_rep and vis_rep, for which the specification format we can create later on, once we begin delving into that area. Finally, we add to the record the elements that the sensors and actuators can use to specify what pre and post processing functions should be used, if any. Thus the updated sensor and actuator records are as follows:


-record(sensor,{id,name,cx_id,scape,vl,fanout_ids=[], generation,format,parameters, phys_rep,

vis_rep, pre_f, post_f}).

-record(actuator,{id,name,cx_id,scape,vl,fanin_ids=[],generation,format, parameters, phys_rep,

vis_rep, pre_f, post_f}).

The following are detailed descriptions of the just added elements, and the for- mat they should possess:

format : Is a tuple, and specifies the geometrical format of the signal being generated (if in a sensor record), or expected (if in an actuator record). The format can be set to: no_geo, which would specify that there is no geometrical information, it is simply a vector. The specification format of this element will be: {geo_tag,[Resolution1,Resolution2...]} .

parameters : Is a list, and can be used to specify various parameters that the sensor or actuator can use to further augment the manner in which they process the input signals, or produce output signals.

phys_rep : Is a list. Specifies the physical representation, if any, of the sensor or actuator. This could be represented as a list of molecules, or hardware ele- ments, each specifying how it is connected to the others. If for example the agent is used in Alife or robotics experiments, this element would specify the physical properties like mass, volume... of the particular sensor or actuator. vis_rep : Is a list. Specifies the visual representation, if any, of the sensor or ac- tuator. If for example the agent is used in ALife or robotics experiments, this would specify what the sensor or actuator looks like in the simulated world. pre_f : Is an atom, a name of a function. It is the preprocessing function to be used, if any is listed. This could further separate the different types of sensors of the same type, allowing, through evolution, for our TWEANN to explore the different manners in which signal preprocessing is done for the sensor or actua- tor used.

post_f : Is an atom, a name of a function. The postprocessing function to be used, if any. This, as the pre_f, can be used to explore the different ways to postprocess the signals for the same sensor or actuator, and thus eventually landing on a perfect combination.

11.2.2 The neuron Record

We have decided that the neuron element must also specify the plasticity func- tion it uses, not just the activation function. We do this by adding to the record the plasticity function designating, pf element. Finally, we also add the element which will specify which aggregation function to use, should it be a simple dot product? Or perhaps the vector difference between the current input vector and the previous


input vector? Or should all the input signals be multiplied? The modified neuron record is as follows:

-record(neuron, {id, generation, cx_id, af, pf, aggr_f, input_idps=[], output_ids=[], ro_ids=[]}). The added elements and their format and detailed definitions are listed next:

pf : Is an atom, the name of a plasticity function. The plasticity function accepts as input the synaptic weight list, input vector, and output vector. The output of this function is the updated version of the synaptic weights.

aggr_f : Is an atom, the name of the aggregation function. Thus far we have on- ly used the dot function, which simply aggregates the input signals, dots them with the synaptic weights, adds the bias if present, and returns the result, which is to be sent through the activation function. An alternative to the dot product might be as follows: Aggregate the input signals, save the input signals to pro- cess registry, subtract the previous input signals from the current input signals (element by element, the previous vector from the current vector), then calcu- late a dot product of the result and the synaptic weights.

11.2.3 The agent Record

Perhaps it would at some point be a good idea to also allow the agents to choose, or evolve whether to use simulated annealing during the tuning phase or not, and which of the numerous and varied tuning selection functions to use. Final- ly, we also add the element which will specify what function to use for the tuning duration. The modified agent record is shown below:

-record(agent,{id, generation, population_id, specie_id, cx_id, fingerprint, constraint,

evo_hist=[], fitness, innovation_factor=0, pattern=[], tuning_selection_f, annealing_f, tuning_duration_f, perturbation_range, mutation_operators, mutation_selection_f}).

The added elements and their format and detailed definitions are as follows: tuning_selection_f : Is an atom specifying the function name. This function ac-

cepts the list of NIds as input, and then chooses which neurons should be se- lected for synaptic weight perturbation. There can be any number of ways to do this. We can simply select all NIds, or only those which were created or muta- tion effected within the last 3 generations, or we could select randomly. tuning_annealing_f : Is an atom specifying the function name. There are nu- merous ways to implement simulated annealing based on the various properties of the NN system, for example the neuronal or general agent age. This function should accept as input the list of the selected neuron Ids for perturbation, and then based on it and the perturbation_range parameter, calculate the new and


updated perturbation intensity range for each neuron, sending each neuron a signal that it should perturb its weights and the intensity range it should use. tuning_duration_f : Is a tuple composed of an atom specifying the function name, and a parameter: {FunctionName,Parameter} . The Max_Attempts value could also be computed in numerous ways. It could be a constant, independent of the NN's size, which is what we're using now. On the other hand, it could be proportional to the size of the NN, the number of neurons selected for perturba- tion. After all, the greater the number of neurons recently added to the NN, the longer it would take to get the right combination, the longer it would take to tune their synaptic weights. The input to this function should be the NN size. Though it must be ensured that all the agents which belong to the same species or population, use the same tuning_duration_f, otherwise we could end up with certain agents achieving a higher fitness merely due to having the tun- ing_duration_f that gives them a larger Max_Attempts. It is for this reason that this value should be set up by the population_monitor, such that all the agents are evaluated against each other based on fair grounds. We want the NN based agents that learn the fastest, that are the most dynamic and most general, given all other things are equal, including the amount of time they are given to tune in. Using different tuning_duration_f for each different agent in the same popu- lation would be the same as letting different sprint runners being given differ- ent amounts of time to run the track, and then calculating their fitness based on the proportion of time of the total allotted time that they used to run the track. Certainly the one who was given the greatest amount, even if he was a bit slower than the others, would end up wining due to having taken a smaller pro- portion of the allotted time. This does not make winning runner the fastest... For this reason, this element is set up by the population record, and copied to the agent records, ensuring that all agents in the same population use the same tuning duration function.

perturbation_range : Is a float() , a multiplier of math:pi() , which specifies the actual perturbation range. So for example if perturbation_range is set to 2, then the actual (before annealing, if any) perturbation range is 2*math:pi() , thus the random perturbation value is chosen by running: math:random(2*math:pi()) . In this manner, by allowing the constraint record provide a list of perturba- tion_ranges, as it does with activation functions for example, we can have dif- ferent agents using different perturbation_range parameters, which will help in experimentation and also make it easier to test this element and its affect on evolution and performance in different problems and applications. mutation_operators : Is a list of tuples: [{MO,Probability} …] composed of at- oms representing the names of mutation operator functions that are available to the agent during its topological mutation phase, and floating point numbers that specify the proportional probability of that mutation operator being chosen verses another operator in the list. It might be advantageous for different agents, different species, or simply different populations, to have different sets of mutation operators. We could also, through this list, specify a particular set of mutation operators for an agent based on its encoding approach. Or we could


even perturb the mutation probabilities, thus making our system have the functionality of an evolutionary strategies based evolutionary computation algorithm.

tot_topological_mutations_f : Is a tuple which specifies the name of the func- tion that produces the total number of topological mutations to be applied to the NN, and a parameter for that function. Currently, our system does this through the function: random:uniform(round(math:pow(TotNeurons,1/2))). It is a clean approach and works very well, but perhaps we will later wish to try a different method, or allow different agents to use different functions, to see which work better in a single population. We could achieve this through this parameter, by letting different agents use such different functions.

11.2.4 The population Record

There are numerous things that can be decoupled in the population_monitor process, and there are a number of elements we can add to the population record, which will then independently specify some particular feature of the population. By doing this, others can then create new modules, new functions, and easily augment the general manner in which the population of agents is optimized, or controlled and evolved. We need to have an element which specifies whether the evolutionary loop is generational or steady-state, same as in DXNN. ALife would certainly work best when a variation of the steady-state evolutionary loop is used. Another element can be used to specify what function to use with regards to selec- tion of fit against unfit agents. Should it be random? Simply top3, or the competi- tion algorithm we've created? What factors should go into the computation of fit- ness? Just the fitness of the agent, or also its size? Should a NN based agent with 1000 neurons which has a fitness of 100 be considered more or less fit than a NN composed of 10 neurons with a fitness of 99? Thus the selection algorithm and the fitness computation algorithm can be decoupled as well. The updated record should thus be as follows:

-record(population,{id, polis_id, specie_ids=[], morphologies=[], innovation_factor, evo_alg_f, fitness_f, selection_f}).

The added elements, their format, and detailed definitions, are discussed next: evo_alg_f : Is an atom, which specifies the name of the evolutionary loop. This

could be steady_state or generational . The population_monitor process should, based on this element, choose the evolutionary loop function which will then, using the fitness_f and selection_f, deal with the population of agents, select them, mutate them, keep track of the population's progress...

fitness_f : Is an atom, which specifies the name of the fitness function used to calculate the true fitness of the agent based on its fitness score and various


other properties. When for example using the none fitness function, the popula- tion_monitor accepts the fitness score of the agent as its true fitness. If on the other hand we create a function size_proportional , then the population monitor takes the fitness score achieved by the agent, and then scales it down propor- tional to the size of the NN system. The input to the fitness_f is the agent id, since by this time the agent will have its fitness score stored in its record within the database. The output of the function is the agent's true fitness.

selection_f : Is an atom, which specifies the name of the selection function. This function accepts the list of agent Ids and their true fitness, and produces a list of fit agents, and the number of offspring each is allotted. When executed within the steady-state evolutionary loop, this value (allotted number of off- spring) is converted into a percentage of the agent being selected for the crea- tion of a single offspring to replace it, and a percentage that the agent itself will be released back into the simulated world for reevaluation (If ALife application is in question), instead of producing an offspring.

11.2.5 The constraint Record

This particular record is used to initialize the population record, it specifies the constraint of the species to which it belongs, and the choices and functions that the population can use. It is this record that contains the initial list of the available functions for each category: activation functions, plasticity functions, selection functions... We now modify it to accommodate the new decoupled additions, as shown next, in which I also added, in comment form, the possible values from which the parameters for each constraint element can be chosen:

-record(constraint,{

morphology=xor_mimic, %xor_mimic

connection_architecture = recurrent, %recurrent|feedforward

neural_afs=[tanh,cos,gaussian,absolute] %[tanh,cos,gaussian,absolute,sin,sqrt,sigmoid],

neural_pfs=[none], %[none,hebbian,neuro_modulated]

neural_agr_fs=[dot_product], %[dot_product, mult_product, diff]

tuning_selection_fs=[all], %[all,all_random, recent,recent_random, lastgen,lastgen_random]

tuning_duration_f={const,20}, %[{const,20},{nsize_proportional,0.5}]

tuning_annealing_fs=[none], %[none,gen_proportional]

perturbation_ranges= [1], %[0.5,1,2,3...]

agent_encoding_types= [neural], %[neural,substrate]

mutation_operators= [{mutate_weights,1},{ add_bias,1},{ mutate_af,1}, {add_outlink,1}, {add_inlink,1}, {add_neuron,1}, {outsplice,1}, {add_sensor,1}, {add_actuator,1}], %[{mu- tate_weights,1}, {add_bias,1}, {remove_bias,1}, {mutate_af,1}, {add_outlink,1}, {re- move_outLink,1}, {add_inlink,1}, {remove_inlink,1}, {add_sensorlink,1}, {add_actuatorlink,1}, {add_neuron,1}, {remove_neuron,1}, {outsplice,1}, {insplice,1}, {add_sensor,1}, {remove_sensor,1}, {add_actuator,1}, {remove_actuator,1}]


tot_topological_mutations_fs=[{size_power_propotional,0.5}], %[{size_power_propotional,

0.5}, {size_linear_proportional,1}]

population_evo_alg_f=generational, %[generational, steady_state]

population_fitness_postprocessor_f=none, %[none,size_proportional]

population_selection_f=competition %[competition,top3]

}).

As can be noted, the population based parameters are not specified as lists, the way that the neural activation functions are specified for example. Instead, they are specified as single atoms. This is done for consistency. Though it is fine to give entire lists of available functions with regards to activation functions, plastici- ty, tuning... That is not the case when it comes to the population based functions, since when we run simulations and tests, we want those values to be specified, not chosen randomly during every different run. I've also added, as comments, lists of available functions for selection for the particular decoupled feature, some of which are not yet created, like the tuning_duration_fs function by the name size_proportional for example. We will eventually create them, as we continue to advance our neuroevolutionary system.

The added elements and their format and detailed definitions are as follows: neural_pfs : Is a list composed of plasticity function names, such as hebbian,

ojas, and others that get added over time. When a neuron is created, it randomly chooses a plasticity function to use, similar to the way it chooses an activation function. In the population_monitor the researcher can choose to use a list composed of a single plasticity type, for example the none function, which means that the neurons of this NN will not have plasticity.

neural_aggr_fs : Is a list composed of aggregation function names. The func- tions can be dot_product, which is the default, or some other later added func- tion like the diff , or mult function.

tuning_selection_fs : Is a list composed of tuning selection function names. The different agents can start off using different tuning selection functions (neurons chosen for tuning during evaluation), allowing the researcher to de- termine which of the selection functions is more advantages in a particular simulation. This is also simply a good way to specify in the population monitor, when creating a seed population using the SpeCon variable, the selection algo- rithm that one wishes for the agents in the population to use. tuning_duration_f : Is a tuple composed of the tuning duration function name and its parameter. All agents in the population must use the same tun- ing_duration_f function so that their performance is judged on equal grounds. Thus they are all given the same number of chances to improve during their tuning phase. Also, if we set it to {const,1}, then we can effectively convert our neuroevolutionary system to a genetic rather than memetic algorithm based TWEANN, since every agent will have the exoself simply perform a single evaluation to get its fitness, and then immediately terminate, waiting for the topological mutation phase.


tuning_annealing_fs : Is a list composed of annealing function names, which could also be set to none . This is the default, and is the manner in which our current neuroevolutionary system functions. Different agents can start off with different annealing functions if the list is composed of more than one function. perturbation_ranges : This is a list of floats(), each of which specifies the mul- tiplier of math:pi(). The actual perturbation range is: Multiplier*math:pi(). The perturbation_ranges can be composed of a single float, or a list of them if one wishes to experiment with a population composed of agents using different perturbation ranges.

agent_encoding_types : This is a list of atoms, which specify the different types of encodings. At the moment we only have the neural encoding imple- mented. In later chapters we will implement a substrate encoding type, the two types used in DXNN for example. Other researchers will add other encoding approaches over time. This is what is used by the exoself when it is summoning the NN system. Based on the encoding type, it will compose the NN system differently, performing different steps for the different systems. The list can contain multiple types, thus the population could be composed of different types of agents, some neural encoded, some substrate encoded... mutation_operators : Is a list of tuples, composed of function names of the mutation operators available to a particular agent, and the probability of being used, as proportional to other operators. We would usually have a single list for the entire population, or have different populations each with a different set of mutation operator lists available, if for example we wish to test whether a mu- tation operator list containing both, pruning and complexifying operators, pro- duces better results than one containing only complexifying operators. tot_topological_mutations_fs : Is a list of tuples, where each tuple is composed of a function name, and a function parameter. This allows us to have a popula- tion of agents where different agents can use different functions that determine the amount of topological mutations that is used to produce the agent's off- spring.

population_evo_alg_f : Is an atom specifying the evolutionary loop function, generational or steady_state for example. A population should use only a single evolutionary loop function for all its agents. A particular population_monitor must be fair to all its agents, judging them all in the same way (otherwise we cannot know whether the agent is superior due to its fitness, or due to its pref- erential treatment) and so only a single type of evolutionary loop type should be used for a single population. Of course one could run multiple populations, each using a different type of evolutionary loop, existing in the same scape (like in Alife simulation for example).

population_fitness_postprocessor_f : Is an atom specifying the fitness post- processor function. It could simply be none, which would imply that the fitness score achieved by the agent is what its true fitness is, or it could scale its fitness score proportional to the average complexity of the population, which would


give advantage to smaller NNs which perform similarly to larger ones, for ex- ample.

population_selection_f : Is an atom specifying the name of the selection func- tion. A single population uses a single particular selection function to separate the fit agents from the unfit.

We would specify and specialize the INIT_CONSTRAINTS tuple for each ex- periment we'd like to run, starting the population_monitor using the specified con- straints record. We should allow our neuroevolutionary system to start with multi- ple populations, or a single population but multiple species, each specie could then use different constraints parameters. At this time though, we will assume that the population treats all its agents the same, and does not segregate them into particu- lar species, each with its own specie_monitor (something that can be implemented later on).

To specify a population using some particular combination of evolutionary loop algorithms, fitness postprocessor, and selection functions, we would do as follows:

- define(INIT_CONSTRAINTS,

[#constraint{morphology=Morphology, connection_architecture=CA, population_evo_alg_f

=EvoAlg, population_fitness_postprocessor_f =FitPostProc, population_selection_f =Selection} || Morphology [xor_mimic],CA [feedforward], EvoAlg [steady_state], FitPostProc [none], Selection [top3]]

).

As you can see, the lists: [steady_state], [none], [top3], could be composed of multiple function names, in which case multiple constraints would be generated, which would then, after we implement this feature, allow our neuroevolutionary system to start with multiple species, each with its own permutation of these pa- rameters.

11.3 Updating the records.hrl

Because the specifications, formatting, the way in which the various elements, ids, functions, parameters, of all these records are becoming more numerous and complex, we need to create a document where all this information and specifica- tions detail can be found. We will use the records.hrl file, adding a list of com- ments to it which specifies the formatting, type and pattern, of every element in the various records stored. In this manner, as we continue to expand our neuroevolutionary system, we will know exactly what format and in what form, each element of the record comes. Listing-11.1 shows the new updated rec- ords.hrl.


Listing-11.1 The updated and commented records.hrl file

-record(sensor,{id,name,cx_id,scape,vl,fanout_ids=[],generation,format,parameters,

phys_rep,vis_rep,pre_f,post_f}).

-record(actuator,{id,name,cx_id,scape,vl,fanin_ids=[],generation,format,parameters,

phys_rep,vis_rep,pre_f,post_f}).

-record(neuron, {id, generation, cx_id, af, pf, agr_f, input_idps=[], output_ids=[], ro_ids=[]}). -record(cortex, {id, agent_id, neuron_ids=[], sensor_ids=[], actuator_ids=[]}). -record(agent,{id, generation, population_id, specie_id, cx_id, fingerprint, constraint, evo_hist=[], fitness, innovation_factor=0, pattern=[], tuning_selection_f, annealing_parameter, tuning_duration_f, perturbation_range}).

-record(specie,{id, population_id, fingerprint, constraint, agent_ids=[], dead_pool=[], champion_ids=[], fitness, innovation_factor=0}).

-record(population,{id, polis_id, specie_ids=[], morphologies=[], innovation_factor, evo_alg_f, fitness_f, selection_f}).

-record(constraint,{

morphology=xor_mimic, %xor_mimic

connection_architecture = recurrent, %recurrent|feedforward

neural_afs=[tanh,cos,gaussian,absolute], %[tanh,cos,gaussian,absolute,sin,sqrt,sigmoid],

neural_pfs=[none], %[none,hebbian,neuro_modulated]

neural_agr_fs=[dot_product], %[dot_product, mult_product, diff]

tuning_selection_fs=[all], %[all,all_random, recent,recent_random, lastgen,lastgen_random]

tuning_duration_f={const,20}, %[{const,20},{nsize_proportional,0.5}]

annealing_parameters=[1], %[1,0.5]

perturbation_ranges=[1], %[1,0.5,2]

mutation_operators= [{mutate_weights,1},{ add_bias,1},{ mutate_af,1}, {add_outlink,1}, {add_inlink,1}, {add_neuron,1}, {outsplice,1}, {add_sensor,1}, {add_actuator,1}], %[{mu- tate_weights,1}, {add_bias,1}, {remove_bias,1}, {mutate_af,1}, {add_outlink,1}, {re- move_outLink,1}, {add_inlink,1}, {remove_inlink,1}, {add_sensorlink,1}, {add_actuatorlink,1}, {add_neuron,1}, {remove_neuron,1}, {outsplice,1}, {insplice,1}, {add_sensor,1}, {remove_sensor,1}, {add_actuator,1}, {remove_actuator,1}]

population_evo_alg_fs=generational_default, %[generational, steady_state]

population_fitness_fs=size_proportional, %[none,size_proportional]

population_selection_fs=competition %[competition,top3]

}).

%%% sensor:

%id= {{-1::LayerCoordinate, float()::Unique_Id()}, sensor}

%name= atom()

%cx_id= cortex.id

%scape= {private|public, atom()::ScapeName}

%vl= int()

%fanout_ids= [neuron.id...]

%generation=int()

%format= {no_geo|geo,[int()::Resolution...]}


%parameters= [any()...]

%phys_rep= [any()...]

%vis_rep= [any()...]

%pre_f= atom()::FunctionName

%post_f= atom()::FunctionName

%%%actuator:

%id= {{1::LayerCoordinate,generate_UniqueId()},actuator}

%name= atom()

%cx_id= cortex.id

%scape= {private|public, atom()::ScapeName}

%vl= int()

%fanout_ids= [neuron.id...]

%generation=int()

%format= {no_geo|geo,[int()::Resolution...]}

%parameters= [any()...]

%phys_rep= [any()...]

%vis_rep= [any()...]

%pre_f= atom()::FunctionName

%post_f= atom()::FunctionName

%%%neuron:

%id= {{float()::LayerCoordinate, float()::Unique_Id},neuron}

%generation= int()

%cx_id= cortex.id

%af= atom()::FunctionName

%pf= atom()::FunctionName

%aggr_f= atom()::FunctionName

%input_idps= [{Input_Id,Weights},{neuron.id|sensor.id,[float()...]}...]

%output_ids= [neuron.id|actuator.id...]

%ro_ids= [neuron.id...]

%%%cortex:

%id= {{origin, float()::Unique_Id()},cortex}

%agent_id= agent.id

%neuron_ids= [neuron.id...]

%sensor_ids= [sensor.id...]

%actuator_ids= [actuator.id...]

%%%agent:

%id= {float()::Unique_Id(),agent}

%generation= int()

%population_id= population.id

%specie_id= specie.id

%cx_id= cortex.id


%fingerprint= fingerprint()

%constraint= constraint()

%evo_hist= [OperatorAppllied...]

% {atom()::MO_Name, ElementA.id, ElementB.id, ElementC.id}

% {atom()::MO_Name, ElementA.id, ElementB.id}

% {atom()::MO_Name, ElementA.id}

%fitness= float()

%innovation_factor= int()

%pattern= [{float()::LayerCoordinate, N_Ids}...]

%tuning_selection_f= atom()::FunctionName

%annealing_parameter= float()::FunctionName

%tuning_duration_f= {atom()::FunctionName ,any()::Parameter}

%perturbation_range= float()

%mutation_operators= [{atom()::FunctionName,float()}...]

%tot_topological_mutations_f= {atom()::FunctionName,float()}

%%%specie:

%id= atom()|{float()::Unique_Id,specie}

%population_id= population.id

%fingerprint= fingerprint()

%constraint= constraint()

%agent_ids= [agent.id...]

%dead_pool= [agent.id...]

%champion_ids= [agent.id..]

%fitness= float()

%innovation_factor= int()

%%%population:

%id= atom()|{float()::Unique_Id,population}

%polis_id= polis.id

%specie_ids= [specie.id...]

%morphologies= [atom()::Morphology_Name...]

%innovation_factor= int()

%evo_alg_f= atom()::FunctionName

%fitness_f= atom()::FunctionName

%selection_f= atom()::FunctionName

%%%fingerprint:

%generalized_sensors= [sensor()::init...]

% sensor.id = undefined

% sensor.cx_id = undefined

% sensor.fanout_ids = []

%generlized_actuators= [actuator()::init...]

% actuator.id = undefined

% actuator.cx_id = undefined


% actuator.fanin_ids = []

%generalized_pattern= [{float()::LayerCoordinate,int()::TotNeurons}...]

%generalized_evohist= [GeneralizedOperatorApplied...]

% {atom()::MO_Name,{float()::ElementA_LayerCoordinate,atom()::ElementA_Type},

{ElementB_LayerCoordinate,ElementB_Type},{ElementC_LayerCoordinate,ElementC_Type}

},

% {atom()::MO_Name,{float()::ElementA_LayerCoordinate,atom()::ElementA_Type},

{ElementB_LayerCoordinate,ElementB_Type}},

% {atom()::MO_Name,{float()::ElementA_LayerCoordinate,atom()::ElementA_Type}},

% {atom()::MO_Name},

%%%constraint:

%morphology=xor_mimic, %xor_mimic

%connection_architecture = recurrent, %recurrent|feedforward

%neural_afs=[tanh,cos,gaussian,absolute] %[tanh,cos,gaussian,absolute,sin,sqrt,sigmoid],

%neural_pfs=[none], %[none,hebbian,neuro_modulated]

%neural_aggr_fs=[dot_product], %[dot_product, mult_product, diff]

%tuning_selection_fs=[all], %[all,all_random, recent,recent_random, lastgen,lastgen_random]

%tuning_duration_f={const,20}, %[{const,20},{size_proportional,0.5}]

%annealing_parameters=[1], %[1,0.5]

%perturbation_ranges=[1], %[0.5,1,2,3...]

%agent_encoding_types= [neural], %[neural,substrate]

%mutation_operators= [{atom()::FunctionName,float()}...]

%tot_topological_mutations_fs = [{size_power_propotional,0.5}],

%[{size_power_propotional,0.5},{size_linear_proportional,1}]

%population_evo_alg_fs=generational_default, %[generational, steady_state]

%population_fitness_fs=size_proportional, %[none,size_proportional]

%population_selection_fs=competition %[competition,top3]

%%%polis

%id= atom()|float()|{float()::Unique_Id,polis}|{atom()::PolisName,polis}

%%%scape

%id= atom()|float()|{float()::Unique_Id,scape}|{atom()::ScapeName,scape}

Some of the defined elements, like the polis id and the scape id, we are not yet using. But we will eventually start using these elements, and so their format is de- fined here for convenience. Having now discussed in detail the various features and decoupled elements of our system, we will implement them in the following section.

11.4 Updating the Modules

Having discussed the additions we wish to make, and elements of the system we wish to decouple and put into their own respective modules, we are now ready to develop these functions, and update our neuroevolutionary system such that it is capable of using the modified records. We will first update the genotype module, then genome_mutator, then population_monitor, then exoself, and then finally the neuron module.

11.4.1 Updating the genotype Module

In the genotype module we need to modify the functions which set up the rec- ords for the agent and neuron elements. Also, it is in the construct_Cortex/3 func- tion that the actual NN genotype is created in, and the elements are linked together into a topological structure, so it is in this function that we have to set up a case that constructs the NN system based on the actual agent type ( neural or substrate ).

We first update the construct_Agent/3 function to randomly select a tuning, an- nealing, and duration functions, and the tuning perturbation multiplier, as shown next:

construct_Agent(Specie_Id,Agent_Id,SpecCon)->

random:seed(now()),

Generation = 0,

{Cx_Id,Pattern} = construct_Cortex(Agent_Id,Generation,SpecCon),

Agent = #agent{

id = Agent_Id,

cx_id = Cx_Id,

specie_id = Specie_Id,

constraint = SpecCon,

generation = Generation,

pattern = Pattern,

tuning_selection_f = random_element(SpecCon#constraint.tuning_selection_fs),

annealing_parameter = random_element(SpecCon#constraint.annealing_parameters),

tuning_duration_f = SpecCon#constraint.tuning_duration_f,

perturbation_range = random_element(SpecCon#constraint.perturbation_ranges),

mutation_operators = SpecCon#constraint.mutation_operators,

tot_topological_mutations_f = random_element(SpecCon#constraint.tot_topological_

mutations_fs ),

evo_hist = []

},

write(Agent),

update_fingerprint(Agent_Id).


The random_element/1 function simply accepts a list, and returns a random el- ement from the list, chosen with uniform distribution.

We then update the construct_Cortex/3 function, so that it uses a case to select how it builds the seed NN topology and links the elements together, based on the agent encoding type (neural or substrate):

construct_Cortex(Agent_Id,Generation,SpecCon)->

Cx_Id = {{origin,generate_UniqueId()},cortex},

Morphology = SpecCon#constraint.morphology,

case random_element(SpecCon#constraint.agent_encoding_types) of

neural ->

Sensors = [S#sensor{id={{-1,generate_UniqueId()},sensor}, cx_id=Cx_Id,

generation=Generation}|| S<- morphology:get_InitSensors(Morphology)],

Actuators = [A#actuator{id={{1,generate_UniqueId()},actuator},cx_id=Cx_Id,

generation=Generation}||A<-morphology:get_InitActuators(Morphology)],

N_Ids=construct_InitialNeuroLayer(Cx_Id,Generation,SpecCon, Sensors,

Actuators,[],[]),

S_Ids = [S#sensor.id || S<-Sensors],

A_Ids = [A#actuator.id || A<-Actuators],

Cortex = #cortex{

id = Cx_Id,

agent_id = Agent_Id,

neuron_ids = N_Ids,

sensor_ids = S_Ids,

actuator_ids = A_Ids

}

end,

write(Cortex),

{Cx_Id,[{0,N_Ids}]}.

Next, a simple modification is made to the construct_Neuron/6 function, which makes the neuron also randomly select an annealing function and an aggregation function, in addition to a randomly selected activation function:

construct_Neuron(Cx_Id,Generation,SpecCon,N_Id,Input_Specs,Output_Ids)->

Input_IdPs = create_InputIdPs(Input_Specs,[]),

Neuron=#neuron{

id=N_Id,

cx_id = Cx_Id,

generation=Generation,

af=generate_NeuronAF(SpecCon#constraint.neural_afs),

pf=generate_NeuronPF(SpecCon#constraint.neural_pfs),

aggr_f=generate_NeuronAggrF(SpecCon#constraint.neural_aggr_fs),

input_idps=Input_IdPs,


output_ids=Output_Ids,

ro_ids = calculate_ROIds(N_Id,Output_Ids,[])

},

write(Neuron).

The generate_NeuronPF/1 and generate_NeuronAggrF/1 functions are analo- gous to the generate_NeuronAF/1:

generate_NeuronAF(Activation_Functions)->

case Activation_Functions of

[] ->

tanh;

Other ->

lists:nth(random:uniform(length(Other)),Other)

end.

%The generate_NeuronAF/1 accepts a list of activation function tags, and returns a randomly chosen one. If an empty list was passed as the parameter, the function returns the default tanh

tag.

generate_NeuronPF(Plasticity_Functions)->

case Plasticity_Functions of

[] ->

none;

Other ->

lists:nth(random:uniform(length(Other)),Other)

end.

%The generate_NeuronPF/1 accepts a list of plasticity function tags, and returns a randomly chosen one. If an empty list was passed as the parameter, the function returns the default none tag.

generate_NeuronAggrF(Aggregation_Functions)->

case Aggregation_Functions of

[] ->

dot_product;

Other ->

lists:nth(random:uniform(length(Other)),Other)

end.

%The generate_NeuronAggrF/1 accepts a list of aggregation function tags, and returns a ran- domly chosen one. If an empty list was passed as the parameter, the function returns the default dot_product tag.

With these small simple additions, this module is now fully updated, and the agents created through it in the future, will be able to include the newly added con- straint parameters into their genotype.


11.4.2 Updating the genome_mutator Module

With the genotype module updated, we now have a way to create the new ver- sion of genotypes which use the updated building blocks and the updated records. We now need to update the genome_mutator module so that we can mutate such genotypes, and take advantage of the new information available in them. For the genome_mutator we want to add a method that allows different mutation operators to have different probabilities of being selected, a separate module that contains the functions that allow the genome_mutator to choose the neurons to be mutated through different algorithms, a new module that contains the functions which pro- vide different ways of calculating how many mutation operators should be applied to the NN based agent, and finally, a new set of mutation operators that can mutate the new genotypical features (plasticity and aggregation functions for example).

The very first thing that needs to be updated in this module is the ap- ply_Mutators/1 function:

apply_Mutators(Agent_Id)->

A = genotype:read({agent,Agent_Id}),

Cx = genotype:read({cortex,A#agent.cx_id}),

TotNeurons = length(Cx#cortex.neuron_ids),

TotMutations = random:uniform(round(math:pow(TotNeurons,1/2))),

io:format( “Tot neurons:~p Performing Tot mutations:~p on:~p~n ”,[

TotNeurons, TotMutations,Agent_Id]),

apply_Mutators(Agent_Id,TotMutations).

Because we change the very way that TotMutations is calculated, by creating a whole new function which calculates the total topological mutations value. And because apply_Mutators/1 , after this modification, does nothing but execute the tot_topological_mutations function, we can delete it and move the remaining functionality to the mutate/1 function, as shown next:

mutate(Agent_Id)->

random:seed(now()),

F = fun()->

A = genotype:read({agent,Agent_Id}),

{TTM_Name,Parameter} = A#agent.tot_topological_mutations_f,

TotMutations = tot_topological_mutations:TTM_Name(Parameter,Agent_Id),

OldGeneration = A#agent.generation,

NewGeneration = OldGeneration+1,

genotype:write(A#agent{generation = NewGeneration}),

apply_Mutators(Agent_Id,TotMutations),

genotype:update_fingerprint(Agent_Id)

end,

mnesia:transaction(F).


%The function mutate/1 first updates the generation of the agent to be mutated, then calculates the number of mutation operators to be applied to it by executing the tot_topological_mutations:TTM_Name/2 function, and then finally runs the apply_Mutators/2 function, which mutates the agent. Once the agent is mutated, the function updates its finger- print by executing the genotype:update_finrgerprint/1 function.

As can be noted from the above implementation, we now let mutate/1 function call the apply_Mutators/2 directly. Rather than first calling apply_Mutators/1, which then called the apply_Mutators/2 function.

Since there are many ways to calculate TotMutations, we create the tot_topological_mutations module, which can store the different functions which can calculate this value. At this time, this new module will only contain 2 such functions, as shown in Listing-11.2.

Listing-11.2 The implementation of the tot_topological_mutations module, with two available functions.

-module(tot_topological_mutations).

-compile(export_all).

-include( “records.hrl ”).

%ncount_exponential/2 calculates TotMutations by putting the size of the NN to some power: Power

ncount_exponential(Power,Agent_Id)->

A = genotype:read({agent,Agent_Id}),

Cx = genotype:read({cortex,A#agent.cx_id}),

TotNeurons = length(Cx#cortex.neuron_ids),

TotMutations = random:uniform(round(math:pow(TotNeurons,Power))),

io:format( “Tot neurons:~p Performing Tot mutations:~p on:~p~n ”,[TotNeurons,

TotMutations, Agent_Id]),

TotMutations.

%ncount_linear/2 calculates TotMutations by multiplying the size of the NN by the value: Mul- tiplier.

ncount_linear(Multiplier,Agent_Id)->

A = genotype:read({agent,Agent_Id}),

Cx = genotype:read({cortex,A#agent.cx_id}),

TotNeurons = length(Cx#cortex.neuron_ids),

TotMutations = TotNeurons*Multiplier,

io:format( “Tot neurons:~p Performing Tot mutations:~p on:~p~n ”,[TotNeurons,

TotMutations, Agent_Id]),

TotMutations.


The next thing we need to change is the way mutation operators are chosen. In- stead of uniform distribution, we should, with every mutation operator, also pro- vide a value that dictates its percentage of being chosen in comparison to other mutation operators in the list. This can be done by specifying the mutation opera- tors in tuple form: {MutationOperator,RelativeProbability} for example. The probability can be a pie section, where the total is all the RelativeProbabilities added together. So for example if we have the following list of 4 mutation opera- tors:

[{add_neuron,2},{add_sensor,1},{add_outlink,5},{mutate_weights,10}]

Then the total is 2+1+5+10 = 18, and we could then use Choice = ran- dom:uniform(18) to generate uniformly a random number between 1 and 18, and then go in order, through the mutation operators from left to right, and see where the generated value lands. It is like spinning a roulette wheel, where each mutation operator gets a slice on the wheel proportional to its RelativeProbability. Thus, if Choice = 4, then we go past the add_neuron, past the add_sensor, and land on add_outlink, since it is located between 3 and 8 inclusive, and 4 lands between those borders. This is visually demonstrated in Fig-11.2 .

Fig. 11.2 The roulette wheel approach to mutation operator selection probability.

We can implement this approach by first modifying the way in which the muta- tion_operators are specified, changing it from a list of atoms to a list of tuples as discussed above. And then modify the function apply_NeuralMutator/1 , which


randomly picks out a mutation operator from the available list, and applies it to the NN system.

We originally stored the mutation operator function names in the MUTATORS macro in the genome_mutator module. Having now updated the records, all the names of the mutation operators are now listed in the constraint and the muta- tion_operators element of the agent's record. The mutation operators are specified in the records.hrl as follows:

mutation_operators= [{mutate_weights,1},{ add_bias,1},{ mutate_af,1}, {add_outlink,1}, {add_inlink,1}, {add_neuron,1}, {outsplice,1}, {add_sensor,1}, {add_actuator,1}],

Thus we modify the apply_NeuralMutator/1 function from its original form of: apply_NeuralMutator(Agent_Id)->

F = fun()->

Mutators = ?MUTATORS,

Mutator = lists:nth(random:uniform(length(Mutators)),Mutators)

io:format( “Mutation Operator:~p~n ”,[Mutator]),

genome_mutator:Mutator(Agent_Id)

end,

mnesia:transaction(F).

To one that uses a function capable of extracting the mutation operators with the probability proportional to the values specified in the tuples of the muta- tion_operators list:

apply_NeuralMutator(Agent_Id)->

F = fun()->

A = genotype:read({agent,Agent_Id}),

MutatorsP = A#agent.mutation_operators,

Mutator = select_random_MO(MutatorsP)

io:format( “Mutation Operator:~p~n ”,[Mutator]),

genome_mutator:Mutator(Agent_Id)

end,

mnesia:transaction(F).

%The apply_NeuralMutator/1 function applies the available mutation operators to the NN. Be- cause the genotype is stored in mnesia, if the mutation operator function exits with an error, the database made changes are retracted, and a new mutation operator can then be applied to the agent, as if the previous unsuccessful mutation operator was never applied. The mutation opera- tor to be applied to the agent is chosen randomly from the agent's mutation_operators list, with the probability of each mutation chosen being proportional to its relative probability value.

select_random_MO(MutatorsP)->

TotSize = lists:sum([SliceSize || {_MO,SliceSize} <- MutatorsP]),

Chapter 11 Decoupling & Modularizing Our Neuroevolutionary Platform Choice=random:uniform(TotSize),

select_random_MO(MutatorsP,Choice,0).

select_random_MO([{MO,SliceSize}|MOs],Choice,Range_From)->

Range_To = Range_From+SliceSize,

case (Choice >= Range_From) and (Choice =< Range_To) of

true ->

MO;

false ->

select_random_MO(MOs,Choice,Range_To)

end;

select_random_MO([],_Choice,_Range_From)->