Train for Persistence

Growing a target once is not enough.

A useful self-organizing system should continue to satisfy its objective after it arrives.

That means the target should behave more like an attractor than a timestamped frame. Operationally: low loss sustained over an extended interval past the growth horizon — not exact state invariance, not biological homeostasis.

The failure mode

Suppose training always evaluates at step 64.

The model can learn:

seed
  ↓
grow
  ↓
target at step 64
  ↓
overshoot
  ↓
disintegrate

The loss never sees the failure after step 64.

So the model is not wrong according to the objective.

The objective is wrong according to us.

Train over a time window

Instead of one terminal loss, evaluate multiple later states:

def persistence_loss(model, initial, target, warmup=64, checks=8, interval=8):
    x = initial

    for _ in range(warmup):
        x = model(x)

    losses = []
    for _ in range(checks):
        for _ in range(interval):
            x = model(x)
        losses.append(F.mse_loss(x[:, :4], target))

    return torch.stack(losses).mean()

Now a transient match is not enough.

Use a pool of states

Another powerful training pattern is to maintain states from previous rollouts.

seed
partly grown state
nearly complete state
mature state
slightly degraded state

Sample from that pool, evolve for a random number of steps, compute loss, then put the resulting state back.

# Pool holds unbatched (channels, height, width) states;
# stacking a sampled batch restores the batch dimension.
def pool_step(pool, model, target, batch_size=8, steps=(32, 96)):
    idx = torch.randint(len(pool), (batch_size,)).tolist()
    batch = torch.stack([pool[i] for i in idx])

    n = torch.randint(*steps, ()).item()
    for _ in range(n):
        batch = model(batch)

    loss = F.mse_loss(batch[:, :4], target.expand(batch.shape[0], -1, -1, -1))

    with torch.no_grad():
        for i, b in zip(idx, batch):
            pool[i] = b.detach().clone()

    return loss

This exposes the rule to many positions along its own trajectory rather than restarting from the seed every time.

Why a state pool changes the learning problem

Without a pool:

learn seed -> target

With a pool:

learn many nearby states -> target region

    flowchart LR
    P[pool of states] --> B[sample batch]
    B --> R[roll out N steps]
    R --> L[loss vs target]
    L --> U[update rule]
    R --> W[write finals back to pool]
    W --> P
  

The latter encourages corrective dynamics.

If the state wanders slightly away from the desired morphology, the update rule has experience pushing it back. Three rungs, three setups, three different claims:

Measure persistence explicitly

Define a survival window:

def persistence_curve(model, seed, target, total_steps=512):
    x = seed.clone()
    losses = []

    for step in range(total_steps):
        x = model(x)
        losses.append(float(F.mse_loss(x[:, :4], target)))

    return losses

(Verified: the harness runs end to end at small scale and returns the full 512-length curve.)

Plot loss against time.

A persistent model should not merely hit one low point.

It should remain in a low-loss region for an extended interval — with the guardrail that low loss over a finite horizon is not indefinite stability, and shape persistence is not state invariance.

Persistence is dynamic maintenance

A mature organism-like pattern need not be frozen.

Hidden channels can continue changing while the visible structure remains approximately stable.

So persistence can mean:

stable visible morphology
+
ongoing internal dynamics

That distinction is important. A fixed point is only one kind of attractor.

Persistence helps recovery, but does not replace damage training

One of the most important lessons from the original Growing NCA experiments is that explicitly training systems to remain near their target can also improve their ability to recover from perturbations, even before strong damage training is introduced.

But incidental recovery is not enough.

If regeneration matters, damage needs to become part of the training distribution itself.

That is the next chapter.

Research

• Mordvintsev, A., Randazzo, E., Niklasson, E. & Levin, M. — Growing Neural Cellular Automata (Distill, 2020). Experiments 1–2 are this chapter’s training curriculum: naive fixed-horizon growth goes unstable past its horizon, while pool training molds the target into an attractor — with the honest observation that persistent models often regenerate incidentally, which motivates but never substitutes for damage training. https://doi.org/10.23915/distill.00023