Repository navigation
Conversation
…and training-loop changes Uncommitted work accumulated by earlier sessions, plus this session's changes to the session event record. Pages the pool with a starvation hatch and its telemetry log; train.py carries the hatch plumbing, the '# event' session record and the plotting hooks; corpora and config carry the matching settings. Not included on purpose: weights/ (gitignored), tools/ (the project keeps it untracked), runs/samples.txt and OPs.samples.txt (the upstream log), and the upstream docs/assets block.
Applies upstream PR volotat#20 by @dreddnafious (unmerged upstream at the time of writing), which fixes volotat#19: the flattened gradient vector changed length whenever a parameter sat a step out, so the running mean no longer matched it and read crashed on long runs. Recurrence-depth sampling makes that routine - a depth-1 step forces the halt and the halting head (d_model + 1 elements) gets no gradient that step. Ported verbatim from refs/pull/20/head (cb36633) rather than retyped. Co-authored-by: dreddnafious
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What I measured
In the continuous-reading run (Qwen-free, 8 GB card, ~437M chars read), expert
residency is decided by a demand score, and a challenger must beat the incumbent by
prune.margin = 0.10before it displaces it.The measured problem: the demand scores are near-tied. At the live dwell, the best
non-resident candidate sits within 3% of the weakest resident. A 10% margin
therefore hands essentially every near-tie to the incumbent, and the effect compounds —
of 89 experts that had never been seated, 97% never got a slot on their own.
That is not routing collapse (which the MoE literature covers well); it is an
admission pathology that only exists once experts are paged, and it is invisible
unless you instrument the near-tie distribution.
What I changed
A bounded re-entry route (
prune.hatch_every,prune.hatch_cooldown,0disables):at most one forced replacement every N admission decisions, restricted to growth-born
or low-admission experts, fewest-admissions-first, with the victim chosen as the
weakest evictable resident so churn lands at the bottom of the working set rather
than on the core.
Measured at the live dwell over 127 decisions, of the same 89 starved experts:
hatch_every16is the setting in this branch: ~4.5× the baseline's movement, still bounded.Also in this branch:
dwell_charsprotection for newcomers, pool paging changes, and aGradSNR fix that counts a missing gradient as zeros rather than dropping the observation.
Why it might matter beyond this repo
Expert offloading is now the standard answer for memory-constrained inference
(MoE-CORE, MoBiLE, CoX-MoE, S³). Every one of those systems has an admission step with
some hysteresis, and this failure mode follows from the combination of near-tied scores
and a displacement margin rather than from anything specific to mini-AGI.