ENGINEERING NOTE · BATCH RECOVERY

A batch is finished only after its output commits.

A manifest can make monthly backfills resumable, but only when it records the truth. I designed the load so a failed month remains eligible for retry and a successful retry replaces only the partitions it owns.

System: PySpark bronze → silver → gold
Unit of work: One source month
Validation: 49 offline tests

01 · THE FAILURE WINDOW

A status file can lie before the data exists.

The pipeline processes one month at a time and uses a JSON manifest to skip months that already completed. The dangerous sequence is to mark a month complete before its gold output is durable.

If the process crashes between those two actions, the next run trusts the manifest and skips unfinished work. Recovery appears successful while the dataset remains incomplete.

UNSAFE ORDERRecord month firstA crash can leave missing output

SAFE ORDERCommit output firstRecord the month only afterward

The manifest is evidence of a completed write, not a promise that one will happen.

02 · THE WRITE

Replace the touched partitions, not the table.

Each month can contain several pickup_date partitions. Dynamic partition overwrite replaces only those dates touched by the current batch, preserving partitions written by other months.

That makes a retry deterministic: rerunning a month repairs its own dates instead of appending duplicates or deleting unrelated history.

spark.conf.set(
    "spark.sql.sources.partitionOverwriteMode",
    "dynamic",
)

frame.write.mode("overwrite") \
    .partitionBy("pickup_date") \
    .parquet(target)

03 · THE COMMIT

Advance the manifest only after gold succeeds.

The month is added to the manifest after the final gold write completes. If any earlier stage fails, the month stays absent and the next run retries it.

The manifest itself is written through a temporary file and rename, so an interrupted update does not destroy the last known-good state. Together, output ordering and atomic replacement make the resume point trustworthy.

Write and validate bronze, silver and gold outputs.
Leave the month unrecorded if any stage raises an error.
Replace the manifest only after the new content is complete.
Skip a month only when the durable manifest says it committed.

04 · THE SCOPE

Plain Parquet can still have a clear recovery contract.

This project deliberately avoids Delta Lake and Iceberg so the recovery mechanics remain visible. That means the manifest and partition strategy carry responsibilities that a transactional table format would normally handle.

For this bounded monthly pipeline, the design is easy to inspect and test. At larger scale or with concurrent writers, I would use a transactional table format and keep the same core rule: completion metadata follows a successful commit.