독서리서치레이더투자 프레임워크
로그인 / 회원가입
← 읽기로 돌아가기
뉴스주제 · SimCT跨分词器蒸馏중대1 분 소요DisconfirmAI 편집부

USTC and Tencent Team Proposes SimCT to Recover Lost Supervision in Cross-Tokenizer Distillation

SimCT achieves a 6.0% average relative gain over SFT by expanding the supervision space without adding parameters.

검증됨 2026-10-16 13:10 GMT+8

Researchers from the University of Science and Technology of China (USTC), Tencent, and other institutions have proposed SimCT (Simple Cross-Tokenizer On-Policy Distillation), a new method designed to address the loss of supervision signals in knowledge distillation between models with different tokenizers. The paper has been accepted by NeurIPS 2026 and was last revised on October 15, 2026.

Standard On-Policy Distillation (OPD) assumes that teacher and student predictions are comparable token-by-token, an assumption that fails when models use heterogeneous tokenizers. Existing cross-tokenizer methods either incur significant computational overhead or supervise at coarser granularities, discarding a large fraction of the teacher's signal. SimCT restores this signal by constructing a common supervision space containing shared tokens and 'minimal aligned units'—the finest text continuations expressible by both tokenizers.

Across three heterogeneous teacher-student pairs on mathematical reasoning and code-generation benchmarks, SimCT demonstrates consistent improvements. According to the authors, the method achieves an average relative gain of 6.0% over Supervised Fine-Tuning (SFT) and outperforms representative cross-tokenizer baselines. Ablation studies confirm that these gains stem from recovering supervision discarded by exact shared-token matching. SimCT requires no additional trainable parameters or transport computation, leaving the standard OPD loss form unchanged.

출처

  • arxiv.org
로그인 / 회원가입
독서리서치레이더투자 프레임워크
↗
주제

SimCT跨分词器蒸馏

中科大与腾讯团队提出SimCT方法,旨在解决大模型知识蒸馏中因教师与学生模型使用不同分词器(Tokenizer)导致的监督信号丢失问题。传统On-Policy Distillation (OPD) 假设双方预测可逐Token比较,但在异构分…

3 단계0 건업데이트됨 2026-10-16 19:39

현재까지의 과정

  1. 2026-05-08SimCT论文v1版本提交至arXiv,首次提出通过扩展监督空间解决跨分词器蒸馏信号丢失问题。arxiv.org ↗
  2. 2026-10-15论文更新至v3版本,确认被NeurIPS 2026接收,并在数学和代码基准上验证了比SFT平均6.0%的提升效果。arxiv.org ↗
  3. 2026-10-16媒体广泛报道SimCT成果,强调其无需额外参数即可恢复教师信号的技术优势。arxiv.org ↗

아직 해결되지 않은 문제

  • SimCT在更大规模或更多样化的模型家族组合中的泛化能力如何?
  • 该方法是否会影响训练速度或计算开销的具体量化数据?
  • 社区对‘最小对齐单元’这一理论定义的接受度及后续改进方向?
전체 주제 읽기 →
리서치 계속하기 →