Observe learning
in full.
A visual library for navigating nanoGPT from its architecture to the internal signals that change while it learns.
Why more observables?
The phenomena we know are bounded by what we routinely measure.
Grokking and scaling laws have already shown that training contains structure far richer than a smooth fall in loss. Yet most training dashboards still return to a small set of familiar signals: loss, accuracy, learning rate, and parameter norms.
Our premise is simple: more phenomena may be waiting in observables we inspect less often. The example below starts with attention entropy and asks what becomes visible when we look beyond the usual dashboard.
Start with attention entropy
Block 1 moves through three distinct regimes.
Mean attention entropy measures how spread out attention is across key positions: higher values indicate more diffuse attention, while lower values indicate more concentrated attention. In the baseline run, Block 1 first settles near 4, drops to a second plateau near 2, then gradually rises toward 2.5.
The curve therefore suggests a sequence of internal reorganizations, not one smooth trend. A standard L2 norm of Block 1's QKV projection weights does not separate the same regions, so the structure is easy to miss in familiar aggregate metrics. The next comparisons ask whether it is shared by other blocks or stable across training setups.
Block 1 is zero-indexed: it is the second transformer block.
Loading processed curve…
One curve, three regimes. Block 1 pauses near 4, drops to a second regime near 2, then rises toward 2.5.
Loading supporting curve…
A supporting comparison. The QKV weight norm captures a broad trend, but not the same three regions visible in attention entropy.
Compare depth
The other blocks do not take the same path.
We next hold the setup and observable fixed, then overlay all 12 blocks to ask whether the trajectory is shared across depth. Block 1 is highlighted; the remaining blocks are drawn in neutral tones.
Loading layer comparison…
Block 1 is the depth-wise outlier. Other blocks settle along smoother paths; none reproduces its plateau–drop–recovery sequence.
Compare setups
The trajectory depends on the training setup.
Block 1 is the depth-wise outlier in the baseline run. The final check holds the block and observable fixed while changing model depth or learning-rate warmup.
Loading setup comparison…
The levels are distributed across different trajectories. The baseline plateaus around 4 and then around 2; the 6-layer run plateaus around 2 early, while the no-warmup run rebounds toward 4.
| Setup | Layers | Warmup | Change from reference |
|---|---|---|---|
| Baseline | 12 | 2,000 steps | Standard Karpathy nanoGPT |
| 6-layer nanoGPT | 6 | 2,000 steps | Model depth only |
| No LR warmup | 12 | 0 steps | Warmup schedule only |
All other recorded settings follow the same reference setup where applicable.
What the comparison reveals
A different observable produces a different research question.
Taken together, the comparisons show a three-regime trajectory that is unusual across depth and sensitive to controlled changes in the training setup. They do not yet provide a causal explanation—but they turn an otherwise hidden pattern into a concrete question that can be tested.
Explore the library
Follow a signal from first glance to a shared finding.
The example above is one path through the library. Use the same tools to move from a training setup to a module, inspect an observable, and compare the pattern across depth or experimental conditions.
-
01
Choose a setup
Open the exact model, optimizer, and schedule configuration for every recorded run.
-
02
Navigate the model
Move through embeddings, every transformer block and submodule, layer normalization, and the output head.
-
03
Inspect a signal
Read definitions and recorded histories for weights, gradients, updates, activations, attention, and loss.
-
04
Compare and share
Overlay layers or setups on shared axes, focus a step range, and preserve an annotated finding as a discovery.
Navigate from a transformer module to its recorded observables, then inspect the complete curve on shared training-step axes.
Open this curveThe current reference dataset
- Reference setup
- Standard Karpathy nanoGPT GPT-2 training on OpenWebText
- Architecture
- 12 layers · 12 heads · 768 embedding dimensions · 1,024-token context
- Training
- AdamW · 2,000-step warmup · cosine learning-rate decay
- Coverage
- 1,820 curves recorded through step 100,000
Close the feedback loop
See something unexpected? Put the evidence where others can inspect it.
You do not need to leave the observation in a private screenshot or guess where to send it. Choose the path that matches how developed the finding is.
Leave a note on the curve
- Open a curve in the Explorer and expand the chart.
- Use the Notes panel to tag the whole curve, one step, or a step range.
- Post the observation. Others can jump to the same region and reply in place.
Publish a discovery
- Open Discoveries and describe what you found and why it matters.
- Add up to six curve regions across setups, layers, or observables as evidence.
- Publish it to the research log, where the evidence and discussion stay together.
Have an idea about the software itself? Use Suggestions so feature requests and curator replies remain public.
The library will keep growing.
Next we plan to add more training setups and record runs for more steps at finer intervals, making transitions easier to resolve and compare.
Over time, the library can also grow into a lightweight experiment platform: inspect a signal, form a hypothesis, and test a small intervention without leaving the same interface.
Start exploring