ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
News4 min read·2026-10-19

How a German Civic Data Platform Made All Local Politics Searchable on One Server

Poliscope co-founder and CTO Kolja Martens describes how the platform reduced hybrid-search latency over 130 million vector points from 20 seconds (measured on an earlier ~70 million-point index) to 0.87 seconds end-to-end on a single Hetzner bare-metal server in Germany, using tenant partitioning by state, a prefix index on hierarchical administrative IDs, and quantization migration that shrank the in-RAM index by 24% while actually improving recall. He also explains why an EU-only self-hosted stack was a jurisdictional requirement, not just a cost decision.

ImportanceMaterialEvidenceE3 inspectableWrite-upStandard

2026-10-15 08:00 GMT+8

German administrative IDs have an engineering property that turns out to be extremely useful: every sub-entity's ID starts with its parent's ID. Poliscope co-founder and CTO Kolja Martens exploited this structure in Qdrant, partitioning by state into tenants so that a query can discard up to 80% of the corpus before vector search even begins. After layering tenant partitioning, a prefix index on the region hierarchy, and quantization migration, the current 130-million-point collection returns Qdrant queries at 8.5 ms P50 and 0.87 seconds end-to-end; the first version, running on roughly 70 million points with little tuning, took up to 20 seconds per search.

Poliscope crawls about 4,800 active council information portals (Ratsinformationssysteme) covering roughly 90% of Germany's municipalities and 98% of its population. These portals run on a dozen different software products that do not talk to each other, publishing agendas, minutes, proposals, and planning documents—the layer of politics people feel most directly: a wind-farm zoning decision, a new school building, whether the bus route survives. Poliscope runs OCR over 4.3 TB of PDFs, chunks the results into 1,536-dimensional embeddings, and stores them in Qdrant. The collection now holds 130 million points, 117 million of them document chunks, with about 20,000 new meetings arriving each month.

Sign in / Sign up
ReadingResearchRadarInvestment framework

Martens lists three key optimizations, each measured on the production corpus. First, tenant partitioning: Germany's 16 states are grouped into 12 tenants matching the collection's 12 shards. Because a query almost always carries a region, the search can exclude 80% of data before starting, bringing typical term queries from 1,315 ms down to 235 ms. Second, a prefix index on the region hierarchy: a single prefix match covers everything inside a district in one condition, replacing the slowest filter in the system and cutting single-town searches from 6–10 seconds to 0.3–0.8 seconds. Third, quantization migration: on September 26, 2026, Poliscope switched the in-RAM index from 2-bit binary quantization to 1.5-bit TurboQuant—a configuration change requiring no re-ingest—shrinking the index from 45.9 GiB to 35.1 GiB, reducing P50 latency from 16.8 ms to 11.7 ms, and actually improving Recall@10 from 69.5% to 75.1%. Two days later it moved original vectors on disk from float32 to the turbo4 datatype, cutting disk footprint from 1,174 GiB to 531 GiB and P95 latency from 23.9 ms to 12.5 ms, with recall practically unchanged.

Self-hosting was not just about cost. Martens required an EU-only data stack, with no US vendor holding the index and no exposure to the US CLOUD Act. Data sovereignty comes up in Poliscope's own sales conversations, and a managed service in a US hyperscaler's region did not answer that requirement. The final deployment is a Hetzner bare-metal server in Germany with 256 GB of RAM, running Qdrant 1.19 open source as a single node with one collection of 130 million points across 12 shards, database and API sharing the same machine. Martens says the team decides which cores answer queries and which run the optimizer, scheduling large backfills around production traffic rather than around a provider's limits.

All performance figures are Poliscope's own measurements published through Qdrant's blog; no independent reproduction exists. Poliscope's data starts in January 2024, the year the company was founded. Collaborations with research centers will extend coverage back to 2016, multiplying the corpus several times over. Martens plans a second node and sharding across two machines when growth pushes past single-server capacity, and will revisit the self-hosted-versus-managed question then. The team has started measuring recall for each configuration change and is considering shorter dense vectors, since hybrid retrieval with BM25 has not produced known misses at 1,536 dimensions. Martens sums it up: "Every scaling problem we ran into had a knob for it, and the knob was well documented."

On the query path, crawlers pull meetings, agenda items, proposals, and PDFs from the portals into Poliscope's database. PDFs go through an OCR queue on a separate GPU machine where a vision model turns every page into Markdown. A worker chunks the Markdown, embeds each chunk in 1,536 dimensions, and upserts it into Qdrant with built-in BM25 as the sparse vector using the IDF modifier. At query time, a hybrid prefetch runs BM25 and dense search in one request—the dense side served from the 1.5-bit TurboQuant index and rescored against turbo4 originals at 3× oversampling—fused with distribution-based score fusion, then reranked by a Jina multilingual cross-encoder. Every hit carries its PDF page range, deep-linking to the exact page in the source document.

Poliscope started with a monitoring problem that companies used to solve by handing interns a list of portals. Martens says that now one search box shows you every council in the country that touched your topic last week—municipal-level monitoring went from practically impossible to something as easy as a Google search.

Sources

  • 原始出处qdrant.tech↗
Continue researching →