Where we left off

In July, we distilled a Chinese frontier teacher into an American open base and investigated the extent to which censorship transferred along with the capability gain. While increased finance capability was acquired, we found no transfer of censorship. Every model we trained on DeepSeek V4 Flash was as uncensored as the untouched base on LineageEval. This is in contrast to DeepSeek V4 Flash itself, which scored 45.45 points more censored on core-political China-sensitive prompts than on their matched controls. We also reported that distillation with a model writing its own hints matched a model with hints from Flash on every seed.

We have since conducted thorough ablations of the method we introduced, Fin-SD, with some unexpected results.

Overview of Fin-SD. For problems the student gets wrong, a frozen copy of the student receives the query, the failing trajectory, and the answer, and returns the first incorrect step k, a short hint h, and an error type (pure arithmetic slips are dropped without a gradient). The trajectory is truncated right before k, so both policies receive a correct, on-policy prefix, and the teacher also receives h. The student minimizes the per-token KL between the two next-token distributions over an L = 100 rollout.

A brief aside

In our last experiment, we investigated training GPT-OSS-120B with expert LoRA. Our best run scored 84.45% accuracy / 99.16% completion rate / 350,812 tokens vs attention-only’s 83.61% accuracy / 98.74% completion rate / 363,144 tokens, but further testing revealed an average within seed noise of attention-only. Consequently, using expert LoRA provides no significant gain for Fin-SD at the 120B scale.

Finding 1: Privileged information can be minimal

In Fin-SD, we gave the hint-writer the ground truth answer, and asked it to write hints of the form “Your logic diverged when X…Assume Y”. For example, a hint might point out that a one-off restructuring charge sits outside the expense base for a given margin calculation.

To measure how much of the method's benefit came from that content, we replaced the entire hint with the vague sentence “Your mistake is conceptual.” Ablations are run with 20B unless noted. 

‍

We found that the placebo performed comparably to the full hint, with accuracy at 72.52% against the full hint's 72.19% and the base model’s 69.16%, completion rate 0.17 pp higher than full hint, and token count 4.7k lower. Similarly, swapping the hint writer for a same-family model roughly 6× the student’s parameter count didn’t boost accuracy, with both arms scoring 72.19%. Finally, the hint-based variants all matched standard on-policy self distillation (OPSD), which uses a full solution. Fin-SD reached 72.19% against OPSD's 70.42% while generating 9.5% fewer tokens at evaluation and completing 2.44 pp more attempts inside budget.

Ablation arms, relative to the base model. [[fn:When analyzing runs, we found a minor error in student rollout generation that caused some rollouts to be only one token long. This impacted all training runs equally, but did not impact evaluation. We fixed the bug and re-ran training; results showed no significant change.]]

‍

A naive reading of these results is that privileged information does not matter. Zhao et al.,[[fn:arXiv:2601.18734]] Shenfeld et al.[[fn:arXiv:2601.19897]] and Hübotter et al.[[fn:arXiv:2601.20802]] established the setting on the premise that it does. Since then, Li et al.[[fn:arXiv:2604.13016]] documented stronger teachers failing to help, Yang et al.[[fn:arXiv:2604.03128]] formalized the leakage gap when the teacher conditions on information unseen by the student, and Kaur et al.[[fn:arXiv:2607.05184]] found privileged context degrades thinking models by suppressing forks. Closest to our result, two concurrent studies replace the reference with a solution to an unrelated problem and find the gain largely survives.[[fn:arXiv:2608.09228]][[fn:arXiv:2608.18271]] Our placebo is a stricter form of that control, since it carries no problem content at all, and we add the size of the hint writer as a second variable. Intuitively, choosing the right amount of privileged information should play a key role in shaping the target distribution to have high performance and minimal distortion.

The optimal level of privileged information specificity likely varies per task. For privileged information to be effective, it must induce a general reasoning structure, as opposed to one-off problem-specific facts and stylistic artifacts. PS-OPSD[[fn:arXiv:2608.01589]] shows structured problem-space guidance outperforms the full solution, and the relevance of that guidance contributes to the gain. The placebo sits at the far end of the specificity spectrum, with no problem-specific facts to leak. For a small model or more difficult task, it may be too vague.

Our claim is accordingly narrower in scope: for finance reasoning domain problems with models of capability on par with or exceeding GPT-OSS-20B, the marginal value of hint content is smaller than the leakage cost of hint content.

Two scope conditions for using a minimal hint follow:

• The student has to already have pre-training exposure to the material. Our critiques name conventions that a Western-pretrained model of this size likely already encodes and simply fails to reach for. Shrestha and Tessier's analysis points the same way, with OPSD gains tracking reasoning the base model already had.[[fn:arXiv:2608.18271]] As an example of the converse, HINT-SD [[fn:arXiv:2605.17873]] applies the same idea to much smaller students in agentic environments and reports the opposite result, namely that a stronger external feedback writer helps substantially. In HINT-SD’s domain, feedback is largely environmental information the student does not have (e.g. a missing credential or renamed endpoint). Ours is latent in the weights, so we would expect that placebo hints work for us but not for HINT-SD.
• The student has to be able to localize. GPT-OSS-20B can identify its mistake given the answer, but a smaller model might not be able to. Writer-invariance should only hold up to a capability threshold.
  
  

Example 1: Economic balance sheet for a family.

Key error: a $600k family foundation the family plans to endow in 25 years is a goal (a liability), not an asset.

Base and epoch 1 produce identical traces. The model raises the right question and then answers it wrong: "…that's an asset? They plan to establish a foundation, so it's an asset? Actually it's a future asset, but present value is $600k, so it's an asset."

Epoch 2 mostly paraphrases base, but a verification step appears that doesn’t exist in the base trace. This is not enough to stop it from making the same mistake.

Epoch 3 starts off the same way, but comes to an opposite resolution. The self-check survives and now takes effect: "…that's a liability? Actually it's a future outflow? … meaning the present value of the future cost? Might be a liability." This time, the model gets it right.

Note that the model was always asking the right question. It didn’t acquire the concept of a financial goal; in fact, the concept surfaced in every base run. Training just increased its probability for “liability”. Full trace in Appendix A.

Finding 2: Windowing is still too crude

We distilled over a 100-token rollout starting at the located error, and found that truncating from 8192 to 100 improved all three metrics (accuracy 72.19% vs 71.09%, completion rate up 0.42 pp, 5.0k fewer tokens). This matches Early Stopping Rollout,[[fn:arXiv:2605.27028]] which restricts on-policy distillation to the first response tokens because the teacher's corrective signal decays later in the rollout. Incidentally, truncating also has the benefit of cutting training memory usage, as the backward buffer is proportional to rollout length.

Unfortunately, this still doesn’t cure the underlying problem of overindexing on style. Within the windowed region, most tokens are still stylistic artifacts (hedges, transitions, restatements, etc), and without granular token filtering we see an increase in language like "but", "maybe", and "however". We also observed significant length increases throughout training. RLCSD[[fn:arXiv:2606.11709]] calls this privilege-induced style drift; conditioned on a full solution, the teacher turns shorter and more assertive and suppresses hedges, a shift the hint induces regardless of its correctness.

Example 2: Put-call parity arbitrage.

Key error: The t=0 inflow must be compounded forward to expiration.

Epochs 1 and 2 both error on the last step. While they correctly calculate a $5,000 net inflow at t=0, they never carry that inflow forward to expiration at the risk-free rate. Notably, epoch 1 actually notices this, but waves it off inside a parenthetical: "(actually realized at expiration but we have cash now)."

Epoch 2, meanwhile, re-derives the discounting three times, confirms the rate is effective rather than continuous, but never mentions compounding the t=0 inflow.

Epoch 3 finally corrects the error, carrying the $50 per contract forward at the 5% risk-free rate. While epoch 3 switches from per-share to per-contract framing, epoch 4 reverts to the per-share frame and still gets it right (albeit with a significantly longer reasoning chain).

We observe a few phenomena here. First, because windowing is a crude instrument, the initial and persistent change to the model is to aggressively check its assumptions, since it receives a signal to course-correct at each token. It stumbles upon the correct reasoning structure only after a few epochs of training. Second, the concept persists despite a complete change of the surrounding derivation, so it’s not memorizing (in contrast to e.g. SFT).

Epoch 5 also gets the question correct, but is twice the length of epoch 3 for the same answer. That space is filled by:

• A full exploration of the opposite arbitrage direction, even after the reasoning chain had already verbally ruled it out.
• Six separate deliberations on output formatting, e.g. "So answer should be '$5,250'. But they might expect just 5250? Usually they'd want $5,250. We'll output $5,250."
• Significant reasoning about per-contract rounding. In contrast, epoch 3 was able to figure out the rounding in a few lines.

Full trace in Appendix B.

‍

The natural next step here is selective distillation, which lets gradients reach only the most promising tokens. A promising approach here is token-teachability OPD (TA-OPD),[[fn:arXiv:2605.26844]] which identifies the most "teachable" tokens based on top-k support and directs training to that set.

Finding 3: Reverse KL works better on short horizons

Reverse KL beat forward KL in our setup: accuracy +1.26 pp under the self writer (72.19% vs 70.93%) and +0.93 pp under the larger writer, with completion rate and token count moving in the same direction under both.

This decision does not come without compromises. Reverse KL is mode-seeking, which makes it fast to converge over short, localized rollouts. Ours are 100 tokens each over five epochs, with around 25 training items per epoch after arithmetic slips are filtered out. This short training horizon naturally benefits mode-seeking, as it takes advantage of the fast convergence and doesn’t expose the degradation. When we trained for more epochs, the aggressive trimming of probability mass became problematic, causing a length explosion.

Neither objective is optimal. Forward can be brittle, and its natural long convergence time as well as the greater resource requirements it demands for extended tuning run counter to self-distillation’s promise of efficiency. Objective design remains a task-specific choice.

What's next

Many of the factors we’ve studied, like the importance of tuning privileged information to align distributions, style-token drift as a failure mode, teachable tokens as a promising method, limitations of raw accuracy as a metric, point to interpretability as a key toolkit for optimization of self-distillation. A greater understanding of model geometry allows for more sophisticated tuning of PI, detection of distortion and drift, and identification of tokens where the geometries align well. This representational work is forthcoming.

Citation

Yu, Johnny and Gorlla, Cyril, "Privileged Information in Self-Distillation: An Empirical Study", CTGT Research, September 2026.

@article{yu2026privileged,
 author  = {Johnny Yu and Cyril Gorlla},
 title   = {Privileged Information in Self-Distillation: An Empirical Study},
 journal = {CTGT Research},
 year    = {2026},
 month   = {September},
 note    = {https://ctgt.ai/research/privileged-information-self-distillation}
}

Appendix A

20B Vague Case Study: test-2060 (run 20260808-035913, 20B expert LoRA)

We scanned all 238 hard questions for a clean wrong→right transition using the 6 base replicates + 2 replicates each at e1/e3. Exactly one problem is unambiguous:

FinanceMath-validation-161: build an economic balance sheet for the Smith family; ground truth 365,000. The whole problem turns on one classification: the $600k family foundation they plan to endow in 25 years is a goal (liability), not an asset.

Base / epoch 1: byte-for-byte identical traces. The model raises the right question and answers it wrong:

Foundation in 25 years... that's an asset? They plan to establish a foundation, so it's an asset? Actually it's a future asset, but present value is $600k, so it's an asset.

Assets 2,760k − liabilities 1,195k = 1,565k.

Epoch 2: the trace is no longer a copy of base (rewording throughout), and a verification step appears that exists in no base trace, but it re-confirms the error:

Let's double-check: Are we missing any other liabilities? The future expected consumption... is a liability. Yes. Also the foundation is an asset. Good.

Epoch 3: same opening, opposite resolution. The self-check survives and now bites:

...establish a family foundation in 25 years, projected to be worth $600k at present: that's a liability? Actually it's a future outflow? ... meaning the present value of the future cost? Might be a liability.

Assets 2,160k − liabilities 1,795k = 365,000. Epoch 3 spends fewer tokens here than four of six base runs.

Appendix B
‍

120B Vague Case Study: Item #103, put-call parity arbitrage (0814, all 5 epochs)

Epochs 1–2: the derivation is entirely correct except the last move.

Both chains are correct on setup: per-share prices $10 and $11.50, PV(K) = 105/1.05 = 100, parity gap of $0.50/share, 10,000 shares → $5,000 net inflow at t=0. Then:

ep1: "Thus profit $5,000 risk-free at expiration (actually realized at expiration but we have cash now). So answer $5,000."

The parenthetical is the model noticing the exact discrepancy and waving it off. The state-finder cut lands at 97% of the chain (94% in epoch 2); nearly everything before it is right.

Epoch 2's self-checking is notable. It runs three verification passes, and every one targets a step that was already correct:

"But we must verify that the risk-free rate is effective annually, not continuous. We used PV(K) = K/(1+r) = 105/1.05 = 100. That's correct."

It re-derives the discounting three times and never once asks whether cash received today equals profit at expiration. Zero mentions of carrying the inflow forward across both epochs. The model cannot self-locate this error; it took the external hint.

Epoch 3: the fix doesn't arrive where training put it.

"At expiration, the positions will offset, leaving risk-free profit of 50 × (1+r)… we receive cash now, invest at risk-free… 50 × 1.05 = 52.5. So total profit = 100 × 52.5 = 5,250."

Correct, but note how it arrives:

• Training distilled a corrected tail onto the epoch-2 prefix, cut at 6,035 of 6,449 chars. The epoch-3 chain never reproduces that prefix; it switches from per-share to per-contract framing ($50/contract × 1.05 = $52.50 × 100) and reaches the answer at 2,869 chars, less than half way through a prefix the correction was grafted onto.
• Epoch 4 then reverts to the per-share frame (0.5 × 1.05 = 0.525 × 10,000 = $5,250) and still gets it right.

So what persisted was the concept (carry the t=0 inflow to expiration), not the corrected text. The insight survived a complete change of the surrounding derivation twice.
‍
The late error position is not a contributing factor: across 0812+0814, trained items with errors in the last quarter fix at 33.3% (n=15) vs 51.5% for errors at 25–50% depth. No clean relationship. Errors in the first quarter do fare worst (29.9%).

Epochs 4–5: the answer is settled, the chain keeps growing.

Epoch 5 is 2.1× the length of epoch 1 for the same answer. What fills it:

• A full exploration of the opposite arbitrage direction (buy call, sell put, short stock), computed and rejected, after the chain had already established C − P > S0 − PV(K).
• Six separate deliberations about output formatting: "So answer should be '$5,250'. But they might expect just 5250? Usually they'd want $5,250. We'll output $5,250."
• Epoch 3 correctly considered and dismissed per-contract rounding (52.5 → 53 → 5,300) in a few lines; epoch 5 spends far more on questions of that kind.
  

The single useful change in this item happened once, between epochs 2 and 3, and cost ~100 characters of actual content. Everything after it is elaboration around a fixed answer. Epoch 3 was also the most efficient chain of the five. It reached the right answer at 44% depth, earlier than any other epoch reached anything. Then the chains got progressively longer while staying exactly as correct.