Manufacturing Company Bright Machines Pushes Data Out of Air-Gapped Factories: Edge Capture and Federated Query Replace Nine-Service Pipeline
Manufacturing company Bright Machines designed a one-way outbound data path for air-gapped factories that cannot open inbound ports: Debezium captures PostgreSQL changes at the edge and writes them to Apache Iceberg, version progression is controlled via AWS Glue catalog pointers, and Unity Catalog federation provides a read-only catalog for in-place querying in Databricks. This architecture replaces a previous event-pushing pipeline composed of nine loosely coupled services, reducing customer dashboard data freshness from hours to minutes and compressing fault detection time from days to under thirty minutes via heartbeat monitoring.
When customers visit Bright Machines' factories, they stand before assembly dashboards watching production lines run, then request specific data: yield by station, cycle time trends, downtime by cause, and batch genealogy. Bright Machines has this data internally; operations teams view it daily. However, customers cannot access it directly over the network because the factory's security design is air-gapped: there is no inbound access, very limited outbound paths, and zero tolerance for risk. Previously, the only way to share data was to rerun reports locally and send them via email. Each new report required the data engineering team to reshape data through multiple ETL jobs, taking hours to days.
Bright Machines builds software-defined automated production lines for manufacturing clients, running inside client factories. Each line runs a full stack of local services—including proprietary MES, integrations with client MES, operator screens, robot workflow orchestrators, and assembly dashboards—all persisted to a local PostgreSQL cluster on-site. The data drives KPIs for yield, finance, operations, quality, vendor component quality, NCR tracking, batch genealogy, OEE, and shift-level downtime and cycle-time analysis.
The team initially attempted to rebuild a cloud copy of each factory database based on application events, spanning nine loosely coupled services: a message broker, custom shoveler, queue, object storage, several serverless functions, a cloud relational database, a scheduler, an ETL pipeline, and a popular open-source BI tool with basic RBAC support. While this architecture worked fine on the factory floor, it failed off-site: pipelines silently dropped data when schemas changed frequently; dead-letter queues were difficult to monitor, so missed or duplicated events went unnoticed; once data landed in a bucket, no one could confirm its accuracy or source; and there was a lack of demonstrable governance and role-based access control.
The requirements for the new architecture were prioritized as follows: access control must match the existing role restrictions within the factory; there must be a single auditable outbound path explainable to any security team; freshness must be measured in minutes; and the edge must use open formats. The core rule of the threat model is: data can be pushed out, but nothing external can reach in.
This ruled out two obvious pull-based solutions: Lakeflow Connect ingesting directly from PostgreSQL, and PostgreSQL exports picked up by Auto Loader. Pull-based ingestion requires a network path from the cloud to the factory database and running ingestion compute in an environment outside the account where the data resides. The question became: how do we achieve the same governed result while keeping every component touching raw factory data within our own environments?
The answer was to flip the direction: do not pull from the cloud, but push from the edge. Bright Machines' architecture consists of five steps. First, Debezium Server tails the PostgreSQL replication slot at the factory edge, writing changes as an append-only Apache Iceberg change log to local object storage. If the WAN drops, the backlog becomes local object files rather than write-ahead logs piling up on the production database. Second, an edge component copies each object to a bucket in the site's cloud account—this is the factory's sole outbound data path, with credentials restricted to writing only to this bucket. Cross-account replication moves the objects to the analytics account. Third, Iceberg tables are registered in AWS Glue, and Unity Catalog federates Glue as a read-only catalog, allowing over a hundred tables from the factory's PostgreSQL databases to be queried directly in Databricks without copying. Fourth, in Databricks SQL, Bronze views collapse the change log to the current state, and a Lakeflow job rebuilds Silver models every ten minutes. Fifth, AI/BI dashboards read data from the Silver layer with role-based permissions, and OpenSharing shares the same governed data zero-copy to customers.
A key design decision was why register Iceberg tables in AWS Glue rather than pointing Databricks directly at the storage bucket. A change log on object storage is a stream of newly added metadata files. A small function advances the pointer for each table in Glue only after the new metadata file has fully replicated, and only forward. Glue holds the pointer, Unity Catalog holds the permissions, and Databricks reads only the validated version. This gives Bright Machines precise control over when each table moves forward—a catalog hop owned by them, with guards on the pointer.
Another easily overlooked operational issue is distinguishing between an idle production line and a broken pipeline. When a line stops for a shift change, no new rows arrive, which looks exactly like a pipeline break. Row counts cannot distinguish these two extremes. Debezium updates a one-row heartbeat table in each source database every five minutes, regardless of whether the line is running. This heartbeat update travels the same path as production data. The first task of the Lakeflow job checks the heartbeat; if any heartbeat is older than thirty minutes, the task fails, turning the entire job red. Dashboards continue to display the last good data rather than presenting stale data as current.
Actual results: A row committed on the production line is queryable in Databricks within one to two minutes at steady state, with Silver models and dashboards refreshing every ten minutes. Heartbeat checks detect pipeline faults within thirty minutes, compared to days previously. The first customer-visible dashboard went live about two months after kickoff. The proof of concept was completed by two engineers in approximately six weeks.
The migration was phased into four stages: piloting on a PoC line; going live on the highest-value production line; running parallel against the old snapshot path for row-by-row validation; and finally performing a hard cutover for customer dashboards.
An unexpected finding was that scale appeared in the wrong dimension: moving hundreds of gigabytes per day, but in the form of hundreds of thousands of small objects, meaning every design decision ultimately concerned object rate rather than byte volume. The team stated that if they were to start over, they would build a reconciliation path on day one—a sync driven by create events never goes back to retrieve missed objects, and downstream systems won't notice until a query touches them.
Bright Machines summarized five lessons useful for other teams needing to share data from restricted sites: choose a platform starting from your existing security model rather than approximating it; design the unique outbound path first and buffer behind it; maintain control over when tables move forward; build data once and let permissions handle the rest; and assert freshness on heartbeats, not on row counts.
It should be noted that this is a self-reported case study by the Bright Machines team on the official Databricks blog; all performance figures are self-reported and not independently verified. The architecture is deeply tied to the AWS, PostgreSQL, and Databricks stacks, but the authors note that the Databricks-side design is identical across any cloud, so migration only requires changing the edge sync target, cross-account replication, and catalog hop resources.
Next, Bright Machines plans to build natural language analytics (Genie), custom AI agents for operations, and predictive analytics on top of the governed data, enabling production line operators, quality engineers, and maintenance technicians—who have domain expertise but are not SQL users—to get answers grounded in the same governed Silver data by asking questions.