How a German Civic Data Platform Made All Local Politics Searchable on One Server
Poliscope co-founder and CTO Kolja Martens describes how the platform reduced hybrid-search latency over 130 million vector points from 20 seconds (measured on an earlier ~70 million-point index) to 0.87 seconds end-to-end on a single Hetzner bare-metal server in Germany, using tenant partitioning by state, a prefix index on hierarchical administrative IDs, and quantization migration that shrank the in-RAM index by 24% while actually improving recall. He also explains why an EU-only self-hosted stack was a jurisdictional requirement, not just a cost decision.
ImportanceMaterialEvidenceE3 inspectableWrite-upStandard
German administrative IDs have an engineering property that turns out to be extremely useful: every sub-entity's ID starts with its parent's ID. Poliscope co-founder and CTO Kolja Martens exploited this structure in Qdrant, partitioning by state into tenants so that a query can discard up to 80% of the corpus before vector search even begins. After layering tenant partitioning, a prefix index on the region hierarchy, and quantization migration, the current 130-million-point collection returns Qdrant queries at 8.5 ms P50 and 0.87 seconds end-to-end; the first version, running on roughly 70 million points with little tuning, took up to 20 seconds per search.
Poliscope crawls about 4,800 active council information portals (Ratsinformationssysteme) covering roughly 90% of Germany's municipalities and 98% of its population. These portals run on a dozen different software products that do not talk to each other, publishing agendas, minutes, proposals, and planning documents—the layer of politics people feel most directly: a wind-farm zoning decision, a new school building, whether the bus route survives. Poliscope runs OCR over 4.3 TB of PDFs, chunks the results into 1,536-dimensional embeddings, and stores them in Qdrant. The collection now holds 130 million points, 117 million of them document chunks, with about 20,000 new meetings arriving each month.