1
00:00:00,050 --> 00:00:34,345
Deep dive number nine. The Surrogates tab. Some elements are expensive to track. A three dimensional field map means a Runge Kutta integration through measured field data for every particle, every step. A surrogate is a small neural network that learns that element's transfer behaviour once, and then answers in microseconds. This tab trains them, scores them, and swaps them in, and this deep dive walks every control on it. Our demo machine is a medium energy beam transport with four real buncher cavities driven by field maps.

2
00:00:34,445 --> 00:01:09,012
The idea, honestly stated. A surrogate does not replace physics. It memorises one element's response over a declared range of energies and settings, and it is only trusted inside that range. HELIX trains on data it generates from its own full tracking, validates on held out samples, and reports the validation error to you before you decide to use it. The tab's own hint line states the contract: tick Use to engage a surrogate in envelope mode runs, and Compare runs the envelope twice, baseline against surrogate, and shows the difference.

3
00:01:09,112 --> 00:01:38,078
Before training anything, the economics, and the manual's overview chapter has a table for it. A single envelope pass on a novel lattice is not worth surrogating: training costs more than it saves. A parameter scan of the same elements breaks even at five to ten runs, a matching loop with hundreds of forward passes pays off handsomely, and a tolerance study across many seeds is where the payoff dominates. Everything in this episode has a written reference in this chapter.

4
00:01:38,178 --> 00:02:00,671
The heart of the tab is the trained surrogates table, and it fills automatically from cached weights when you load a lattice. We go to the empty tab first, then load the M E B T deck on camera. Watch the table. It found all four bunchers, trained earlier, one row each with the element name, its validation error, its scope, a Use checkbox and a Compare button.

5
00:02:00,771 --> 00:02:12,695
The status line at the foot of the tab is the evidence. Auto loaded four cached surrogates from the weights directory named by the lattice hash. Nothing was retrained; the rows came from disk.

6
00:02:12,795 --> 00:02:35,117
The element dropdown lists what can be surrogated. Each entry shows the lattice index, the element name, its concrete class and its length: four FieldMap3D bunchers of two hundred forty millimetres. Only field map classes appear. R F Q elements carry internal state a surrogate cannot replicate yet, so they are excluded by construction.

7
00:02:35,217 --> 00:03:23,912
Two columns deserve a closer look. Val M A P E is the mean absolute percentage error over all thirty six transfer matrix entries, measured on held out samples, never on the training set. It is a strict metric: entries that are nearly zero inflate it, which is why these smoke trained values sit between zero point four and zero point nine rather than near zero. Scope is the surrogate's contract. It answers only inside these windows: kinetic energy from two to two point five M e V, cavity amplitude k e plus or minus twenty percent around the card value, and phase from minus one hundred ten to minus seventy degrees. Outside scope it raises an out of scope error and the solver falls back to full physics. Slower, never wrong.

8
00:03:24,012 --> 00:03:58,567
Use is the engagement switch. Ticking it registers the surrogate in the runtime registry, and the status line confirms it: registered surrogate F MAP one. The next zero current envelope run routes that element's full matrix through the network automatically; runs with space charge stay on full physics. You can mix and match, two cavities surrogated, two on full physics. Select all and Deselect all fire the same per row toggle for every row, and the status line reports registered all four surrogates, then unregistered all four.

9
00:03:58,667 --> 00:04:22,457
Now training. Clicking Train surrogate on an element that already has cached weights asks first. The dialog quotes the cache location, the cached validation error, and the sample and epoch budget it was trained with. Open loads the cached weights instantly, Retry retrains from scratch, Cancel aborts. Retraining is a deliberate act, never the default. We choose Retry.

10
00:04:22,557 --> 00:05:23,291
The training dialog, every field. Samples: two hundred Latin hypercube points, the smoke default. Epochs: forty. Hidden dims: two layers of sixty four. Workers: C P U processes for data generation, defaulting to the core count minus two, twelve on this machine. The energy window is auto detected, and the label says so: an envelope forward pass brackets this element's entry and exit energy with a five percent margin. Then the sweep rows, which are dynamic per element. This cavity sweeps amplitude, k e, and phase, because it has an active electric channel; a pure magnetic solenoid would show only k b. This deck declares no ADJUST cards, so the defaults are plus or minus twenty percent and twenty degrees around the current values; with ADJUST bounds the row label would quote them instead. Setting a sweep to zero drops that knob from training entirely. Watch the phase row. We restore it and accept.

11
00:05:23,391 --> 00:06:03,957
Training runs on a background thread and this live dialog tracks it. Stage one, data generation: the bar reports samples done, the rate and the E T A, while the training panels say waiting for training to start. Each sample is a full Runge Kutta integration of the field map at a different energy, amplitude and phase. Stage two fits the network: train loss and held out validation M A P E animate per epoch, with the best so far tracked, and the six by six heatmap shows which transfer matrix entries the network finds hardest. When both stages complete the weights and metadata land on disk and the window title reads done.

12
00:06:04,057 --> 00:06:15,791
Back on the tab the row for F MAP one has updated in place with the fresh validation error, and the status line names the saved weights path under the lattice hash directory.

13
00:06:15,891 --> 00:06:42,691
Closing the live dialog is the cancel gesture. We start a second run, on the second buncher, and close the window during data generation. The trainer polls a stop flag per sample and per epoch, stops, and reports cancelled, never failed. The status reads training cancelled, nothing was saved, and the title gains a cancelled suffix. Weights are only written after both stages complete, so nothing partial ever reaches the disk.

14
00:06:42,791 --> 00:07:07,457
Train all FieldMap batches the whole machine. Shared settings on top, and note the production grade defaults: five thousand samples, two hundred epochs, two layers of one hundred twenty eight. Below, a checkbox per candidate element with Select all and Select none. For the demo we select none, tick the third and fourth bunchers, and drop to the smoke budget of two hundred samples and forty epochs.

15
00:07:07,557 --> 00:08:05,857
Elements train sequentially on purpose: the data generation inside each is already multi process, so parallelising across elements would oversubscribe the C P U. The window title counts one of two, then two of two, and each element still gets its own auto detected energy window and sweep bounds. A failed element would be skipped, not fatal, and closing the dialog stops the current element and skips the rest. At the end the status reads batch training, two of two elements done.

16
00:08:05,957 --> 00:08:28,249
Trust, then verify. The Compare button on a row runs the same beam twice in a background worker, pure Runge Kutta first, then with the surrogate registered, and the status line reports comparing F MAP one while the training buttons grey out. Compare swaps the process wide registry, so it is guarded: it refuses to overlap another solve.

17
00:08:28,349 --> 00:08:58,983
The summary reports the wall clock of both runs, the speedup, the end of line sigma differences, and an honesty line: the number of N N full element queries. This compare ran at the configured five milliamps, so the N N query line reads zero, every relative difference is exactly zero and the speedup is 1.00 times: identical envelopes, no gain, and the note under the table says why. We will unpack that in a moment. First, the plot you were prompted to save.

18
00:08:59,083 --> 00:09:19,983
The saved figure overlays the two envelopes along the full line, sigma x, sigma y, sigma phi and sigma W, with the speedup and the worst relative difference in the title. Disagreement is shown, not claimed. This plot is the honest gate before any production use, and here the two curves lie exactly on top of each other.

19
00:09:20,083 --> 00:10:19,538
Why identical? Because of the current. We ran the same compare at zero and at five milliamps at capture time. At zero current the envelope solver takes each element's full end to end matrix, and the registry serves exactly that from the network: one query per surrogated cavity, a real speedup, and a small nonzero difference at the network's own accuracy, all printed in the summary. With current, the cavity is sliced into substeps for space charge kicks, and the surrogate delegates every partial slice back to Runge Kutta, because a linear end to end matrix cannot be cut into honest sub slices of a time varying R F field. So with space charge the network is never asked, the N N query line reads zero, and the compare shows identical envelopes. The speedup case with current is multi particle tracking, coming up next. Either way the answer stays physically defensible: unsupported or out of scope queries fall back to full physics.

20
00:10:19,638 --> 00:10:43,504
Now the safety rule. All four surrogates are engaged. We load a different lattice on camera, episode one's F O D O cell, which has no field maps at all. The table and the dropdown empty, and the runtime registry is cleared with them, because the registry is keyed by element name, and a surrogate trained for one machine's F MAP one must never answer for another's.

21
00:10:43,604 --> 00:11:00,038
Press Train on this lattice and the tab refuses with a plain warning: the loaded lattice contains no FieldMap or FieldMap3D elements. Pure magnet lines do not benefit from surrogates; the speedup case is field maps, where Runge Kutta is expensive.

22
00:11:00,138 --> 00:11:15,424
Switch back to the M E B T and the cached weights are rediscovered from the lattice hash directory: four rows return, and the status line says auto loaded four cached surrogates. But the Use ticks stay off until you re engage them deliberately.

23
00:11:15,524 --> 00:12:00,257
Where surrogates live on disk. The weights root holds one directory per lattice, named by the first sixteen hex digits of the S H A two fifty six hash of the dot dat file. Edit a single number in the lattice and you get a fresh directory; old weights are never silently reused. Inside, one folder per element with weights dot p t and metadata dot json. The metadata records the element, its class, the training seed, sample and epoch counts, the validation error, the HELIX commit and the creation time: full provenance. A weights directory is self contained; copy it to another machine with the same dot dat and it just works. Rows persist across restarts; the Use ticks deliberately do not.

24
00:12:00,357 --> 00:12:32,157
Below the table, the multi particle hybrid section, and notice that it is honest about its own maturity. The hint says it in plain text: safe mode, the default, delegates to the native integrator, bit identical, no speedup; the experimental fast path transports particles through the surrogate's linear matrix; and the planned residual mode is not implemented, so the substeps control is reserved. Four controls: the master toggle, the fast path opt in, the reserved substeps spinner, and Compare M P.

25
00:12:32,257 --> 00:13:07,324
Operate them. Engage in M P runs alone changes nothing physical, and the status says so: registered surrogates delegate to native Runge Kutta, bit identical to baseline, no neural inference. The fast path is a second, separate opt in, because its accuracy depends on training quality; its status says fast path engaged, run Compare M P to validate. The substeps spinner reports its value propagated to four registered surrogates, though no tracking path reads it yet. We put it back to fifteen and switch the fast path off again.

26
00:13:07,424 --> 00:13:24,757
These three settings persist across sessions. They are written to the application settings file the moment you touch them, here the surrogates group with m p enabled, m p fast path and the residual substeps. The Use ticks are not stored anywhere; engagement is always explicit.

27
00:13:24,857 --> 00:13:51,324
Compare M P with the fast path off. The beam is the project's own: H minus at two point one two M e V, five milliamps, one thousand particles for the demo. Two complete multi particle simulations run back to back, baseline against hybrid. The button reads running comparison, a thin busy bar appears, and the accent status names the particle count, the four surrogates and the substep setting.

28
00:13:51,424 --> 00:14:09,890
The summary. In safe mode every relative difference is exactly zero: sigma x, sigma y, sigma z, both emittances and transmission, at a speedup of 0.95 times. The delegate really is bit identical, and now you have seen it measured rather than promised.

29
00:14:09,990 --> 00:14:19,623
Tick the fast path and run the comparison again. Now the per substep Runge Kutta inside each buncher is replaced by a cached matrix apply.

30
00:14:19,723 --> 00:14:54,590
The price tag arrives with the speedup. Speedup 1.27 times on this short line with four cavities. Sigma x now differs by 0.12 percent, sigma z by 0.77 percent, and the worst entry, sigma z, by 0.77 percent; the note records that the fast path was enabled for the run. Smoke trained surrogates are not accurate enough for science; the manual's measured table shows the longitudinal error dropping six fold with better training. Run Compare M P after every toggle change.

31
00:14:54,690 --> 00:15:08,057
The saved multi particle figure: sigma x, y and phi, both normalised emittances and transmission, baseline solid against hybrid dashed, with substeps, speedup and worst difference in the title.

32
00:15:08,157 --> 00:15:31,985
Everything this tab does is scriptable. The surrogates C L I has four subcommands: train, compare and run envelope mirror the buttons, and register multi shares one trained surrogate across several identical cavities, a cryomodule of eight driven by the same field map, for instance. The Python A P I underneath is a third entry point, documented in the manual.

33
00:15:32,085 --> 00:16:13,151
The manual's surrogates section has five chapters: overview, G U I walkthrough, C L I, Python A P I, and the training guide. The overview spells out which envelope runs engage the network and which fall back to slice delegation, and its measured table sets the fast path's accuracy against training quality. The training guide's cycle table tells you what a budget buys: two hundred samples for pipeline checks, fifty thousand for science. Its worker benchmark shows parallel data generation is bit identical to serial at any worker count. And the troubleshooting section covers the out of scope flood, high validation error, and the no field maps case.

34
00:16:13,251 --> 00:16:32,618
That is the Surrogates tab. Learn the expensive parts once, verify against full physics, engage deliberately, and let the compare tools tell you the truth about the shortcut. Next, space charge and real beams, the physics that makes intense linacs hard, and how HELIX computes it. See you there.