Bronze, Silver and Gold Aren't the Hard Part: Building a Restartable Manufacturing Lakehouse

The short answer: Bronze, Silver and Gold tell you how a lakehouse is organized. They say nothing about what happens when a quality result arrives a day late, a write half-completes at 2 a.m., or a failed pipeline has to run again. In automobile manufacturing, those moments decide whether a shift report can be trusted. We are building a metadata-driven Databricks lakehouse prototype for automotive production data, and the design choices that matter most are about recovery and reconciliation, not layer names. This post walks through eight of them, including the ones we have not proven yet.

Why does pipeline reliability decide whether plant data can be trusted?

Because every production KPI on a dashboard inherits the failure modes of the pipeline that built it. When the pipeline quietly drops, duplicates or reorders records, the numbers change and nobody notices.

An automotive plant describes the same event in several systems at once. The ERP holds the production order. The MES records what actually ran on the line. The WMS tracks the parts and finished goods that moved. Each system captures the event at a different time and at a different level of detail, and none of them was designed to agree with the others.

Getting all three into a lakehouse is the easy part. The consequences show up downstream:

• A missing production record understates a shift's output.
• A duplicated event inflates it.
• A late-arriving quality result rewrites yesterday's performance figures after the morning meeting has already used them.

So the engineering problem is wider than ingestion throughput. It covers data freshness, reconciliation between systems, consistent business keys and clean recovery from partial failures. For a plant or operations leader, those are not technical details. They decide whether the people reading the dashboard act on it or quietly keep their own spreadsheet.

What does a metadata-driven manufacturing lakehouse look like?

It is one reusable processing framework driven by configuration, rather than a separate notebook written for every table. Our prototype combines four logical layers with a three-level execution hierarchy.

The four layers are Raw, Source Image, Silver and Gold. Our version of the medallion pattern splits the usual Bronze layer in two. Raw captures data exactly as it arrived, and Source Image represents the current state of each source system. Keeping capture separate from state means we can always rebuild one from the other. Silver handles cleansing and curated processing. Gold organizes manufacturing facts and dimensions for downstream analytics and BI.

Execution is tracked at three levels: Pipeline → Stage → Job. Table names, business keys and file paths all resolve from configuration. When a new entity fits a processing pattern that already exists, onboarding it is mostly a metadata change.

We are careful about the limits of that claim. Metadata-driven processing does not make every integration configuration-only. A source with new semantics, or a transformation the framework has not seen before, still needs code. What changes is that the code gets written once, in one place, and every entity that follows benefits from it.

Why separate processing logic from entity configuration?

So that validation, logging and recovery behave the same way for every entity, instead of drifting across dozens of hand-built notebooks.

Reusable PySpark components hold the execution logic. Control tables decide which entities to process and how to resolve their inputs and outputs. That splits the system into a control plane (configuration, dependencies and execution state) and a data plane (the code that reads, transforms and writes records).

Fewer notebooks is a pleasant side effect. The real gain is consistency. When a validation rule or a recovery fix lands, it lands for every entity at once, which is the kind of property an auditor or a new engineer can check in a single place.

Why isn't a restartable pipeline automatically an idempotent one?

Because knowing which steps finished is not the same as knowing that running a step twice is safe. Restartability is about skipping completed work. Idempotency is about replays producing the same result. A production pipeline needs both, and they come from different parts of the design.

Our execution hierarchy records completed processing units, so a failed run can be retriggered without repeating every successful step. Suppose 18 of 20 independent entities complete before a failure. A selective retry skips those 18. That is 90% fewer entity executions on the retry. It is not necessarily 90% less runtime or cost, because the two failed entities may be the heaviest ones.

A success flag alone does not make the retry safe. Consider a job that commits its data to the target table and then fails before the audit table records success. On retry, the job looks incomplete even though its data is already written. If the write logic is not deterministic, the retry writes it again.

Safe replay depends on how data is written:

• stable business keys
• appropriate deduplication
• carefully designed Delta MERGE logic
• explicit batch or source-version boundaries

In our implementation, the merge design provides the guarantee, not the audit layer. Silver processing uses a hash-based SCD Type 2 merge, so replaying it against unchanged source data is a safe no-op rather than a duplicate write. We treat that as an invariant the team has to protect. Replace the merge with something less deterministic, such as a blind append, and the guarantee disappears without any visible error.

Does a green job status mean the data is complete?

No. A green status proves the code ran to the end. It does not prove that every expected business record arrived, or that the quantities match the source.

Pipeline → Stage → Job tracking gives us operational auditability: what ran, when, and with what outcome. Audit-balance controls need a different kind of evidence. That means source counts, accepted and rejected records, duplicate handling and business control totals.

Take a validation boundary where every record is either accepted or quarantined:

1,000,000 input records = 997,500 accepted + 2,500 quarantined

That is a 99.75% acceptance rate, which looks healthy on a summary page. The remaining 0.25% still needs investigating, because a small rejection rate can still be material. If those 2,500 records are an entire production line's output for a shift, the plant-level total is wrong in a way no percentage will reveal.

Manufacturing adds one more trap. Matching record counts do not guarantee matching production quantities. A reconciliation that only counts rows can pass while the units built, scrapped or reworked disagree with the MES. Quantity reconciliation matters as much as row counts.

How do you avoid reprocessing the full history on every run?

With Delta Change Data Feed (CDF), which lets the pipeline read only the records inserted, updated or deleted in a Delta table since a given table version or processing point.

If a production table holds 10 million records and 50,000 have changed since the last successful run, CDF supplies just those 50,000 to downstream processing. On large manufacturing datasets, rerunning the full history every cycle is wasted compute and wasted time.

CDF also fits the restartability design. The orchestration layer knows which jobs completed. CDF knows which data changed. Together they answer the two questions a rerun has to ask: what work is left, and on which records.

CDF does not settle every edge case on its own. The implementation still has to define how it handles updates, deletes, late-arriving changes and the replay of change ranges it has already processed. Those rules belong in the design up front, before the first late quality result shows up.

Where do Lakeflow Connect and Databricks Asset Bundles fit?

Lakeflow Connect standardizes how source data gets in, and Databricks Asset Bundles standardize how the framework gets deployed. Neither replaces the processing framework. They remove two common sources of hand-built inconsistency around it.

Lakeflow Connect for source ingestion

Lakeflow Connect provides managed connectors for bringing data from supported databases, enterprise applications and operational systems into Databricks. In an automotive environment, that cuts the amount of custom ingestion code the team has to write and maintain. The flow looks like this:

Source systems → Lakeflow Connect → Raw → Source Image → Silver → Gold

Lakeflow Connect gets the data into Databricks. The metadata-driven framework then owns validation, processing, dependencies, audit tracking and recovery. Sources without an available connector can still use custom ingestion.

Databricks Asset Bundles for repeatable deployment

Databricks Asset Bundles (DAB) package notebooks, jobs, pipelines, configuration and other Databricks resources as code. Rather than creating and configuring jobs by hand in each environment, the team keeps one version-controlled project and deploys it consistently across development, testing and production.

In our prototype, DAB standardizes how the orchestration framework and its supporting resources are deployed. Environment-specific values live in targets and variables, while the project structure stays the same everywhere. Deployments become repeatable, and configuration drift between environments becomes much less likely.

Why does the grain of the Gold layer matter to plant leaders?

Because the grain decides what a number means. If nobody has written down what one row of a fact table represents, two correct tables can be joined into an incorrect answer.

Our dimensional model covers plants, production lines, vehicles and shift-level production execution. The critical design question for every fact table is what a single row stands for.

A line, shift and product aggregate has a different grain from an individual downtime event. Join them without controlling cardinality and records multiply, so measures such as units produced or downtime minutes are overstated. The dashboard still renders, and the number is simply wrong.

Grain also limits what a table can answer. Shift-level downtime totals are good for trend analysis, but they cannot tell you why a line stopped. Root cause needs finer-grained events, reason codes and the operational context around them.

An analytics-ready star schema needs an explicit grain, consistent dimension keys and defined aggregation rules. Labelling tables "facts" and "dimensions" is not enough.

How will we know whether the framework works?

By measuring operational outcomes rather than counting features. The next validation step benchmarks the framework against five measures:

• entity onboarding effort, in engineering hours
• end-to-end data freshness, including p95 latency
• recovery time after an injected failure
• duplicate and reconciliation discrepancies after replay
• compute consumption per successful processing cycle

These will show whether metadata-driven processing and selective recovery deliver practical improvements, or only look good in an architecture diagram.

Here is where things stand today. We have a demoable manufacturing prototype with metadata-driven processing, layered tables, execution tracking and an initial dimensional model. Production readiness needs more evidence: failure-injection tests, replay validation, reconciliation controls and benchmarks on representative workloads. We would rather say that plainly than claim results we have not measured.

Where should a manufacturing data leader start?

Start with recovery, not layers. Most manufacturing lakehouse programs can already draw the Bronze, Silver and Gold diagram. Fewer can answer three harder questions:

1. If a job fails after its data commits but before it is marked complete, what happens on the rerun?
2. When the job status is green, what evidence shows that the right records and the right quantities arrived?
3. What does one row of each Gold fact table represent, and is that written down anywhere?

If those answers live in one engineer's head, the dashboard is only as reliable as that engineer's memory.

We build Databricks data platforms where restartability, reconciliation and governance are designed in from the first pipeline. If you are planning or rescuing a manufacturing lakehouse and want a second opinion on how it handles failure, book a 30-minute discovery call. It is a working conversation about your sources, your pipelines and your numbers, not a pitch deck. If your question is mainly about lineage, access control and audit evidence, see how we approach governed data platforms.