ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
NewsMaterial2 min readDisconfirmAI Editorial

Qdrant Team Benchmarks: Retrieval Quality Surpasses 0.6B Original Precision After Compressing Code Embedding Model from 4B to 1 Bit

The Qdrant team evaluated Papr's code-specific embedding model, papr-embed-code-1, finding that code fine-tuning improved average nDCG@10 by approximately 0.03 on unseen data. However, the more critical conclusion is that quantization changes selection logic: the 4B model compressed via 1-bit TurboQuant achieved higher retrieval scores than the 0.6B model's float32 original precision across seven code tasks, while using only one-twelfth of the memory. This evaluation is vendor self-testing and applies primarily when the corpus consists mainly of code.

2026-11-18 08:00 GMT+8

The Qdrant team conducted an independent evaluation of Papr's papr-embed-code-1 code embedding model, separate from the model card. The model is based on Qwen3-Embedding and was further trained on code retrieval data, offering two sizes (0.6B and 4B) with open weights. The core finding of the evaluation is not merely that code fine-tuning is effective—both sizes outperformed their base models on code tasks they had not been trained on (HumanEval, MBPP, FreshStack, WikiSQL), with the 4B gaining an average of 0.034 nDCG@10 and the 0.6B gaining 0.026—but rather that quantization alters the logic for choosing model size.

The 4B model has a vector dimension of 2560, storing approximately 10 KB per vector in float32. Using Qdrant's 1-bit TurboQuant compression, each vector requires only 328 bytes, a reduction of 31 times. Across seven code retrieval tasks, 1-bit compression retained 99.7% of the 4B model's float32 score without rescoring, and all precision with rescoring at 2x oversampling. This means the compressed 4B model achieves higher retrieval quality than the uncompressed 0.6B (0.877 vs. 0.838 nDCG@10), while consuming only one-twelfth of the RAM. Based on this, the authors recommend: if embedding compute allows, select the 4B model stored at 1 bit, enable rescoring, and prioritize increasing hnsw_ef over adding quantization bits.

This conclusion has clear boundaries. On prose retrieval tasks, code fine-tuning actually caused both sizes to score slightly lower than their base models (the 0.6B dropped by 0.003, the 4B by 0.016); on TREC-COVID, the 0.6B even dropped by 0.065. BM25 hybrid search failed to provide gains across the tested query types (questions, descriptions, code); equal-weight fusion decreased performance on all tasks by 0.02 to 0.17, and even reducing the BM25 weight to 0.1 only brought results close to parity. The entire evaluation was completed by the Qdrant team on its own infrastructure, constituting vendor self-testing, and has not yet been independently reproduced.

Sign in / Sign up
ReadingResearchRadarInvestment framework

Sources

  • qdrant.tech↗
Continue researching →