How a German Civic Data Platform Made All Local Politics Searchable on One Server
Poliscope co-founder and CTO Kolja Martens describes how the platform reduced hybrid-search latency over 130 million vector points from 20 seconds (measured on an earlier ~70 million-point index) to 0.87 seconds end-to-end on a single Hetzner bare-metal server in Germany, using tenant partitioning by state, a prefix index on hierarchical administrative IDs, and quantization migration that shrank the in-RAM index by 24% while actually improving recall. He also explains why an EU-only self-hosted stack was a jurisdictional requirement, not just a cost decision.
German administrative IDs have an engineering property that turns out to be extremely useful: every sub-entity's ID starts with its parent's ID. Poliscope co-founder and CTO Kolja Martens exploited this structure in Qdrant, partitioning by state into tenants so that a query can discard up to 80% of the corpus before vector search even begins. After layering tenant partitioning, a prefix index on the region hierarchy, and quantization migration, the current 130-million-point collection returns Qdrant queries at 8.5 ms P50 and 0.87 seconds end-to-end; the first version, running on roughly 70 million points with little tuning, took up to 20 seconds per search.
Poliscope crawls about 4,800 active council information portals (Ratsinformationssysteme) covering roughly 90% of Germany's municipalities and 98% of its population. These portals run on a dozen different software products that do not talk to each other, publishing agendas, minutes, proposals, and planning documents—the layer of politics people feel most directly: a wind-farm zoning decision, a new school building, whether the bus route survives. Poliscope runs OCR over 4.3 TB of PDFs, chunks the results into 1,536-dimensional embeddings, and stores them in Qdrant. The collection now holds 130 million points, 117 million of them document chunks, with about 20,000 new meetings arriving each month.
Martens lists three key optimizations, each measured on the production corpus. First, tenant partitioning: Germany's 16 states are grouped into 12 tenants matching the collection's 12 shards. Because a query almost always carries a region, the search can exclude 80% of data before starting, bringing typical term queries from 1,315 ms down to 235 ms. Second, a prefix index on the region hierarchy: a single prefix match covers everything inside a district in one condition, replacing the slowest filter in the system and cutting single-town searches from 6–10 seconds to 0.3–0.8 seconds. Third, quantization migration: on September 26, 2026, Poliscope switched the in-RAM index from 2-bit binary quantization to 1.5-bit TurboQuant—a configuration change requiring no re-ingest—shrinking the index from 45.9 GiB to 35.1 GiB, reducing P50 latency from 16.8 ms to 11.7 ms, and actually improving Recall@10 from 69.5% to 75.1%. Two days later it moved original vectors on disk from float32 to the turbo4 datatype, cutting disk footprint from 1,174 GiB to 531 GiB and P95 latency from 23.9 ms to 12.5 ms, with recall practically unchanged.