Manufacturing Company Bright Machines Pushes Data Out of Air-Gapped Factories: Edge Capture and Federated Query Replace Nine-Service Pipeline
Manufacturing company Bright Machines designed a one-way outbound data path for air-gapped factories that cannot open inbound ports: Debezium captures PostgreSQL changes at the edge and writes them to Apache Iceberg, version progression is controlled via AWS Glue catalog pointers, and Unity Catalog federation provides a read-only catalog for in-place querying in Databricks. This architecture replaces a previous event-pushing pipeline composed of nine loosely coupled services, reducing customer dashboard data freshness from hours to minutes and compressing fault detection time from days to under thirty minutes via heartbeat monitoring.
When customers visit Bright Machines' factories, they stand before assembly dashboards watching production lines run, then request specific data: yield by station, cycle time trends, downtime by cause, and batch genealogy. Bright Machines has this data internally; operations teams view it daily. However, customers cannot access it directly over the network because the factory's security design is air-gapped: there is no inbound access, very limited outbound paths, and zero tolerance for risk. Previously, the only way to share data was to rerun reports locally and send them via email. Each new report required the data engineering team to reshape data through multiple ETL jobs, taking hours to days.
Bright Machines builds software-defined automated production lines for manufacturing clients, running inside client factories. Each line runs a full stack of local services—including proprietary MES, integrations with client MES, operator screens, robot workflow orchestrators, and assembly dashboards—all persisted to a local PostgreSQL cluster on-site. The data drives KPIs for yield, finance, operations, quality, vendor component quality, NCR tracking, batch genealogy, OEE, and shift-level downtime and cycle-time analysis.
The team initially attempted to rebuild a cloud copy of each factory database based on application events, spanning nine loosely coupled services: a message broker, custom shoveler, queue, object storage, several serverless functions, a cloud relational database, a scheduler, an ETL pipeline, and a popular open-source BI tool with basic RBAC support. While this architecture worked fine on the factory floor, it failed off-site: pipelines silently dropped data when schemas changed frequently; dead-letter queues were difficult to monitor, so missed or duplicated events went unnoticed; once data landed in a bucket, no one could confirm its accuracy or source; and there was a lack of demonstrable governance and role-based access control.