USTC and Tencent Team Proposes SimCT to Recover Lost Supervision in Cross-Tokenizer Distillation

Researchers from the University of Science and Technology of China (USTC), Tencent, and other institutions have proposed SimCT (Simple Cross-Tokenizer On-Policy Distillation), a new method designed to address the loss of supervision signals in knowledge distillation between models with different tokenizers. The paper has been accepted by NeurIPS 2026 and was last revised on October 15, 2026.

Standard On-Policy Distillation (OPD) assumes that teacher and student predictions are comparable token-by-token, an assumption that fails when models use heterogeneous tokenizers. Existing cross-tokenizer methods either incur significant computational overhead or supervise at coarser granularities, discarding a large fraction of the teacher's signal. SimCT restores this signal by constructing a common supervision space containing shared tokens and 'minimal aligned units'—the finest text continuations expressible by both tokenizers.

Across three heterogeneous teacher-student pairs on mathematical reasoning and code-generation benchmarks, SimCT demonstrates consistent improvements. According to the authors, the method achieves an average relative gain of 6.0% over Supervised Fine-Tuning (SFT) and outperforms representative cross-tokenizer baselines. Ablation studies confirm that these gains stem from recovering supervision discarded by exact shared-token matching. SimCT requires no additional trainable parameters or transport computation, leaving the standard OPD loss form unchanged.