NVIDIA Principal Cloud Architect Mark Chmarny: AICR Validates Configuration, k8s-aibom Tracks Workloads; The Boundary Between Them Is the AI Governance Gap
NVIDIA Principal Cloud Architect Mark Chmarny explains the integration of AICR and Google's open-source project k8s-aibom: the former answers whether GPU infrastructure comes from known and validated configurations, while the latter answers what AI assets are actually running on that infrastructure. Neither is sufficient alone to support AI governance, and the boundary between them is precisely where accountability is currently missing.
When teams deploy AI workloads using NVIDIA Dynamo, Kubeflow, or Slurm, a reproducible deployment bundle does not tell you what is actually running on the cluster. Application teams define and update workloads independently of platform recipes, and mutable image tags like :latest add further complexity, meaning two nodes under the same tag may be running different code. NVIDIA Principal Cloud Architect Mark Chmarny explains in an NVIDIA technical blog how this gap is closed by integrating Google's open-source project k8s-aibom.
Chmarny describes the two systems as opposite ends of the same problem. AICR operates at the runtime deployment layer—drivers, operators, kernels, Kubernetes components, and deployment configuration—and asks: Is the GPU infrastructure built from a known and validated configuration? k8s-aibom sits higher up the stack and later in the cluster lifecycle; it observes Kubernetes workload and Pod APIs and generates a CycloneDX 1.6 AIBOM record for each identified AI workload, asking: What AI assets are actually running on that infrastructure?
Chmarny points out that neither system alone is sufficient to support AI governance, and the boundary between them is exactly where accountability is currently missing. AICR records qualified deployment intent and compares it against the cluster at a point in time; k8s-aibom observes supported workloads and records what it finds, including the confidence level with which each attribute was determined. The former provides control and reproducibility, while the latter provides workload discovery and traceability.
Specifically, AIBOM helps teams discover unregistered or shadow AI workloads in selected namespaces, identify models and inference runtimes, track external AI APIs and vector databases, detect changes or drift in AI dependencies, and support security reviews, incident response, and compliance audits.
The integration works by making k8s-aibom a qualified component within an AICR recipe. Two stock recipes select it by default: h100-gke-cos-inference and h100-gke-cos-inference-dynamo; other GKE recipes can opt in during generation via the --runtime-inventory flag. Chmarny emphasizes that qualification is not a rubber stamp: AICR binds upstream source, chart, CRD, and digest-pinned controller images into one tested set, verifies provenance against Google’s expected build identity, reviews rendered RBAC and security defaults, and exercises installation, upgrade, health checks, and uninstallation on the tested deployment paths.
Using AICR v1.2.0 pinning k8s-aibom 1.6.2 as an example, the workflow proceeds from recipe generation through packaging, deployment, and validation without manual assembly or tuning. After deployment, the k8s-aibom controller runs in the k8s-aibom-system namespace. During validation, the controller must be available, the cluster-scoped AIBOMControllerConfig must report Ready=True, and both CRDs must report the storage version expected by the pinned chart.
Namespaces opt in via the label aibom.k8saibom.dev/enabled=true. Chmarny specifically notes that this label controls which namespaces emit AIBOM records but does not narrow the controller’s read scope—the informers still cache cluster-wide workload and Pod specs, including container arguments and inline environment variables. No sink or secret access is configured by default, so no data leaves the cluster without an explicit decision.
Regarding resource impact, upstream measured k8s-aibom v1.6.1 on GKE using NVIDIA Dynamo and AICR configurations. The test used CPU-only nodes with workload Pods remaining in pending state, so these figures describe controller inventory overhead: approximately four API server requests per minute, average controller CPU usage of 2.2 millicores, and a controller memory working set of 28 to 34 MiB.
Chmarny also clarifies where the integration deliberately stops. Runtime AIBOM writes are Kubernetes resources, but the observations themselves are unsigned; signing would make captured observations tamper-evident but could not prove every workload declaration true or establish successful execution. k8s-aibom can verify Sigstore signatures, enforce signer-identity constraints, check Rekor inclusion, and assign qualifying claims verified confidence, but AICR disables this verification by default. Correlating platform sources with runtime AI inventory remains future integration work.
A notable boundary: AIBOM is built from workload specs, so it can exist before any Pod is running. An AIBOM report may show Ready=True while the Pod is still pulling its image. It tells you what was declared and what was resolved once the Pod exists, but it does not prove successful scheduling or execution.