Life, Simulated

A lot of programmers write Conway's Game of Life in their first year: a grid of cells, a few rules about neighbors, and gliders, blinkers and other stable shapes start showing up on their own. What caught me back then wasn't that it looks nice. It was that meaningful behavior grows out of rules that nobody wrote for that behavior specifically. Three conditions about live and dead cells, and yet something on the screen seems to crawl around and live its own life.

I wanted to poke at that same self-organization, but on something less abstract than cells. A world with animals that have their own behavior: they eat, drink, breed, hunt each other, and somehow coexist without all dying at once or one species taking over the whole map. Simple idea. I kept coming back to it for about ten years, a couple of evenings a year, whenever I had the time. And for almost all of that time nothing really worked.

Below I go through why it didn't work and what finally did. The short version: I stopped scripting the animals' behavior by hand and started training it.

Why the head-on approach failed

My first attempts went the obvious way for a programmer: if I need a deer to look for food, I just write that. Sees grass, walks to grass. Sees a wolf, runs. Thirsty, heads for water. Every behavior is its own condition, and as I added species and situations, the conditions kept piling up.

As long as there's one animal in an empty field, this works. The trouble starts when there are hundreds of them and the world gets its own resource economy. Grass regrows at some rate, a deer eats it at some rate, starts breeding at some energy level, a wolf catches the deer at some speed. All these numbers are tied together, and the moment you touch one, the rest drift apart. Bump up how fast grass spreads and the deer breed, strip it bare, then starve. Weaken the wolves and the deer flood the map. Strengthen them and the wolves eat everyone in a couple of days and sit alone in an empty world.

I had close to a hundred of these settings, and I tuned them by hand. Fix one and you break the balance you just barely put together with another. The world would live for about ten minutes and then tip into one of the extremes: everyone starves, or a single species eats the rest. And then that one dies too, because there's nothing left to eat =) A steady state, where the species just coexist and none of them wins for good, I never managed to reach.

What annoyed me separately was that rule-based behavior itself came out too branchy. A living animal's decisions depend on many things at once, and that fits poorly into a pile of ifs. A deer grazes, but it should keep glancing around. It runs, but preferably not toward a second wolf. It stays with the herd, but doesn't crowd right up against the water. The code kept growing, and the animals somehow didn't get any smarter for it.

At some point I realized I'd come at it from the wrong end. I was trying to describe smart behavior in words, when I should have gotten it the same way gliders appear in the Game of Life: not by specifying it directly, but by setting up conditions where it emerges on its own (emergence).

The idea: don't describe behavior, train it

Here's the plan I landed on. Each animal has a small neural network, its "brain", a very plain multilayer perceptron. It takes in what the animal sees and feels, and it outputs a decision: where to go, whether to eat, whether to run. I don't write the behavior rules (well, I do, but more on that later). Instead I run selection: whoever behaves better survives and leaves offspring, and its version of behavior sticks.

It's basically evolution, just inside a simulation and faster than in nature. The animal's genes are the network weights here, and the selection criterion is the same as in life: did you manage to feed yourself and leave offspring or not.

This plan runs into a problem. Drop a network with random weights into the world and turn on selection by genetic algorithm (GA), and nothing happens. A random brain does nothing sensible, the animal dies almost immediately, so does the next one, and so does the whole population. Every individual is equally helpless, and selection has nothing to choose between: there's no difference between "bad" and "bad". Evolution stalls and shuffles noise generation after generation. The same trouble hits reinforcement learning (RL) started from scratch, by the way: the reward comes too rarely to tell which step was the right one, this is the credit assignment problem.

In nature this is carried by billions of individuals over millions of years. I don't have that kind of time, so I needed to start from something other than zero: a way to lift the population up to a minimally sensible level, one where selection actually has something to improve. And the push came from the very rule-based code that had been failing me until then.

The dummy: a rule-based teacher

I put my old ifs together into a separate character. In the project I call it the dummy. It's a simple algorithm: sees a wall, turns; sees food, goes to it; sees a predator, runs. It won't get any smarter, but it behaves sensibly from the very first step and it runs reliably.

From there the dummy works as a teacher. It lives in the world and makes a decision every step by looking at its sensors, and I record pairs of "what it saw and what it did next". Hundreds of thousands of these pairs. Then, with gradient descent, I nudge the network weights until it starts giving roughly the same decisions on the same inputs. In machine learning this is called behavior cloning, a special case of supervised learning: you fit the network to repeat the right answers, the way you drill a student on worked problems.

So the dummy turns from a source of problems into a launch pad. On its own it's no solution, but it pulls the network out of random chaos up to the level of "can walk, eat, and run away". The interesting part starts only after it, once selection takes over.

One rule I keep from the start: both the dummy and the network see only their own sensors. No top-down look at the map, no x-ray of the whole map with a breadth-first search, no absolute coordinates, just what the animal can actually sense around itself: what's in front of it, how far the water is, whether there's a wolf nearby, how much energy is left. Otherwise the teacher would decide using something the student can't see, and the student physically couldn't reproduce its moves. And it isn't sporting anyway: in the real world animals rely only on what they directly see and feel. We go the same route.

An exam for the teacher

It turned out pretty fast that the network comes out exactly like the dummy's examples. If the dummy makes a mistake somewhere, walks a deer into a dead end or gets stuck on a rock, the network dutifully picks up that mistake too. So the teacher has to be careful, and it needs checking.

For checking I made a set of small arenas, I call them stands. Each one has a single clear task: go around a rock and reach the grass, get out of a pocket between rocks, spot a predator and run, find water behind an obstacle. So the network doesn't memorize a specific layout, each stand is multiplied into hundreds of variations with the rock shifted, a different start point, a mirror flip, a rotation.

You can run the stands live and watch the dummy and the network take the same arenas. And in the editor you can build your own stand: place rocks, water, food, a predator, and see whether they cope.

The requirement for the dummy is simple: it has to pass all the variations, not most of them. It's the only source of correct answers, and teaching the network on its failures is pointless. Getting the old algorithm to pass every stand across every variation turned out to be quite a task, but there's no point moving on without it.

In other words, we literally keep adding ifs to its code whenever it can't handle something.

The weak spot of learning from examples

Behavior cloning has an unpleasant quirk that I didn't hit right away. In its examples the dummy passes the stand cleanly and never leaves the route, so it never shows what to do once you've already made a mistake and ended up somewhere else.

The analogy is simple. Imagine learning to drive by watching a perfect driver who always stays in the center of the lane. They never once show you how to get back from the shoulder, because they never drift onto it. And when you wander off a little yourself, you land in a situation you've never seen, then make a bigger mistake, and the whole thing falls apart.

That's exactly what happens with the network: it misses a bit, ends up in a position that wasn't in the dummy's examples, and gets completely lost there. The errors stack one on top of another. This trouble has a name, distribution shift: the network is now operating in situations it wasn't trained on.

The fix was fairly elegant, a trick called DAgger (Dataset Aggregation). I let the network itself drive, and it wanders into its own crooked situations, the ones the perfect teacher never visits. Then, from that point, I hand control back to the dummy and record how it gets out. I add those recovery-onto-the-route examples to the training set. Now, if the network drifts sideways, it finds its own way back instead of getting lost for good. Kind of dogfooding on your own mistakes.

The important part is that I don't try to detect when the network "behaved wrong", there's no detector here. I just give it the wheel for part of the way, then hand over to the dummy regardless of whether it drove well or badly. I don't catch the bad situations, I deliberately create them, so I can record the dummy's correct way out. The longer the network has been trained, the later I take over, so the early slip-ups get fixed first and the later ones after that.

All these examples also go into the same supervised-learning pot.

What's in the deer's head

The brain itself is more modest than you'd expect. The input is 59 normalized numbers: directions and distances to food, water, a predator and neighbors; its own hunger, thirst, fatigue and age; plus short "whisker" rays around the body that show whether you'll bump into a wall if you go that way. Plus a few memory cells (a stripped-down gated recurrent unit, GRU-lite), so the animal isn't completely amnesiac from step to step. These are just more sensors. Last turn we went left and it was a dead end, so we remember that we turned left.

The dummy takes the same set of inputs.

And this is the network itself, what goes in and what comes out.

Then a single layer of 96 neurons, and the output is a decision: which way to turn, how fast to move, whether to eat, drink, look for a mate, sleep. The network doesn't output a turn angle directly, it picks a target from a set of meaningful directions (toward food, toward water, away from the wolf) and takes the best one by score (arg max). This turned out to be more reliable: if you average two directions into one, it's easy to get a third that leads nowhere, straight into the rock that both original routes were neatly going around.

Why is the network so small? One layer, and why is that enough when the dummy's behavior looks complicated: if wall, turn, go, if wall again, turn around, a whole chain of conditions. The thing is, that chain unfolds over time, not inside a single pass of the network. Each step the animal looks at its current sensors and makes exactly one move, and "go, then check again" is already the next step. The world plays the role of the loop, so the network doesn't have to hold the whole procedure in its head, it just has to map what it sees now to what to do now. A single condition like "wall ahead" is just a threshold on one sensor, that is one neuron, and you lay such neurons side by side, not on top of each other. The memory sensors help here: they keep it from walking in circles.

Extra layers would be needed for something else, for example to build complex images from a raw picture, or to plan several moves ahead, but the task hasn't grown into that yet, width and a little memory are enough. Don't complicate things without need, Occam's razor.

Selection pulls the network away from the teacher

After learning from examples and working through the mistakes, the network passes almost all the stands, but it's still just a copy of the dummy: it can do what the dummy can, and no more. Now the part it was all set up for kicks in.

I take not one network but a population with small random differences, run them through the trials, keep the ones that did better (fitness function), cross over their weights and add mutations, and so on generation after generation. This is a genetic algorithm on top of the network weights, that is neuroevolution. The difference from learning by examples is fundamental: there the network was rewarded for resembling the teacher, here for the result, that is for the animal surviving, eating, leaving offspring. The goals are different, so the network gradually moves away from how the dummy would act toward what actually works.

There's a lesson tied to this stage that didn't come to me right away. If you drill the network on the stands too hard, it becomes an A student on the stands and helpless in the real world: it simply memorizes the arenas (overfitting). To catch this, I hide part of the variations and test the network on layouts it hasn't seen (held-out set). But even that isn't enough: I pick the final brain not by its stand score but by how it behaves in the big world, whether it stalls in place, clumps into a pile, gets its food. A good grade on a stand guarantees nothing on its own.

The big world and inherited brains

The real test isn't an arena with one rock anymore, it's the big world: hundreds of animals, eight species, a crush at the watering hole, competition for pasture, predators between the trees.

Here evolution shows up in full, and it really happens on the screen. Each animal has its own separate brain, not one shared by all. When two of them produce offspring, the pup gets a mix of the parents' brains and a small mutation, just like genes. There's no separate reward function here anymore, the world plays that role, it's literally natural selection: whoever feeds better, runs away in time, and manages to have offspring, their weights spread through the population, and whoever didn't make it doesn't pass its brain on. The dummy gave the start, selection on the stands brought the brain up to sanity, and after that it changes on its own under the pressure of ordinary life in the world.

What broke along the way

I don't want to leave the impression that all of this works and worked smoothly. Plenty of it cost me a lot of nerves, and people usually don't write about that.

The thing I fought longest was animals that just stood still. On the stands the network passes everything confidently, and in the big world whole groups of animals freeze and starve right in the middle of the grass. I blamed the balance until I realized it was about movement: with no clear target the network produced a jittery course, a step left, a step right, and the animal shuffled in place until it died. The dummy didn't have this, because it moved using tricks that don't carry over into a bare network: set a course, move without thinking. I had to bring some of those tricks back separately, into the code, as innate reflexes (subsumption architecture, reflexes at the bottom, thinking on top): push apart in a crowd, hold the chosen course while you go looking for food. These aren't brain decisions, they're something like instincts the animal doesn't learn, it has them by default.

A similar story happened with fear. When I moved the herbivores onto the network, they started grazing calmly next to wolves, and from the outside it looked like they'd stopped being afraid. They hadn't, they were just noticing the predator too late, when it was already right on top of them and there was nowhere to run. The dummy had a longer-range sense, the network didn't, and once I brought that long-range sense back, the herd started scattering ahead of time again.

And, of course, the very balance this all started over. Even with trained animals the eight species wouldn't coexist on their own: either the herbivores had a population explosion, or the predators ate the rare species down to zero. I worked through it slowly and in pieces, cutting fertility here, adding stamina to the predators there, reworking how plants regrow somewhere else. Right now the world holds: all eight species coexist for tens of thousands of steps, and none of them wins for good. This is the steady state I couldn't put together by hand in all my previous attempts. And it's still not endless development. The end of the world in this particular little place happens annoyingly often =)

Where it is now and what's next

What used to be just an idea in my head now works. The world has grass, deer, hares, mice, horses, wolves, foxes, bears and lynxes, each with its own brain, each deciding for itself and passing its brain to its offspring with mutations. The world runs on its own, holds its balance and doesn't fall apart, the predators hunt, the prey scatter ahead of time, the herds don't clump into a pile.

From here it's interesting to dig in two directions. In depth, that's more complex behavior: memory, differences in temperament, pack hunting, something like passing on experience. This is probably where the extra network layers I've managed without so far will finally be needed. In breadth, that's more species and strategies, and maybe several different planets, each with its own climate and its own separate line of evolution. And I'd like it to be pleasant to watch, so part of the time goes into how the world looks.

It's funny that the problem got solved in a way I wasn't reaching for at the start. Who had even heard about AI ten years ago? While I was trying to script the behavior and the balance by hand, nothing stable came out. It started working only once I stopped describing behavior and started training it, and stopped setting the balance by hand and started searching for it with selection. Exactly the self-organization from Conway's Game of Life that got me into all this, only now on animals with neural networks instead of cells. There's even a name for this sort of thing, artificial life.

And the rise of coding agents, which let me try every idea and every fix as fast as possible. Sometimes firing off five subagents at once to research something.

---

The world is spinning right now, you can drop by and watch: the live-world demo.

An honest disclaimer: I haven't reached full autonomy or a permanent balance yet. Left completely alone, the world still slides into extinction sooner or later. So the demo runs with "Agent Smith" mode on, they nudge things a little to keep the world from collapsing. Waiting for Neo =)