Korean AI Infrastructure Firm MOREH Unveils TIDE Framework at SC26, Claiming 1.66x Throughput Gain
MOREH introduced TIDE at SC26, a framework that uses target model hidden states to retrain draft models online, addressing speculative decoding pattern mismatches.
South Korean AI infrastructure software company MOREH unveiled TIDE, an inference framework designed to optimize "speculative decoding," at the Supercomputing 2026 (SC26) conference held in Chicago. Speculative decoding accelerates Large Language Model (LLM) generation by having a smaller draft model predict multiple tokens which a larger target model then verifies. However, traditional implementations often suffer from performance degradation when user request patterns shift, as the statically trained draft model's predictions become inaccurate.
TIDE addresses this limitation by leveraging the "intermediate hidden states" generated by the large model during inference to retrain the draft model in real-time. This approach eliminates the need for separate data generation passes, reducing computational overhead while allowing the draft model to adapt to changing traffic patterns. The system monitors prediction accuracy and effectiveness continuously, pausing speculative decoding when benefits diminish and triggering retraining when patterns change.
According to results presented by MOREH at SC26, TIDE achieved up to 1.66 times higher inference throughput compared to baseline methods without speculative decoding. Additionally, the training time for the draft model was reduced by up to 3.02 times compared to existing online learning approaches. The company also demonstrated a heterogeneous computing strategy, running the large model's inference on Nvidia H100 GPUs while training the draft model on AMD Instinct MI250 GPUs, which reportedly improved overall system throughput by up to 1.22 times compared to using a single GPU type for all tasks. These figures are self-reported vendor measurements and have not been independently reproduced.