- What: Databricks announced significant updates to AUTO CDC (Change Data Capture) within Apache Spark Declarative Pipelines (SDP).
- Key Feature: Introduction of Bitemporal AUTO CDC for dual-axis history tracking.
- Availability: These capabilities are being expanded into open-source Apache Spark 4.2.
- Compliance Focus: Designed to meet SEC Rule 17a-4 and FINRA recordkeeping requirements regarding point-in-time record reconstruction.
Databricks has announced a major evolution of its AUTO Change Data Capture (CDC) capabilities, introducing bitemporal data tracking to solve complex real-world data engineering challenges. The update, revealed in a company blog post, aims to replace hundreds of lines of fragile, hand-coded logic with declarative configurations that automate data history management. These advanced features are also being contributed to the open-source community through Apache Spark 4.2.
The move addresses a significant pain point for data engineers who traditionally rely on Apache Spark to build CDC pipelines. While standard CDC patterns handle basic updates, they often struggle with out-of-order data, partial record updates, and strict auditability requirements. Databricks' enhanced AUTO CDC is specifically designed to automate Slowly Changing Dimension (SCD) Type 1, SCD Type 2, and Snapshot CDC patterns without the need for manual "MERGE" logic.
Dual-Axis History and Bitemporal Tracking
The centerpiece of this update is Bitemporal AUTO CDC, which implements dual-axis history tracking. While standard SCD Type 2 tables track when a fact changed in the real world (business time), they often fail to record what the system believed to be true at a specific point in processing (system time).
Bitemporal tracking manages two independent timelines:
- Business Time (Event Time): When a fact actually occurred in the real world (e.g., a stock symbol becoming reportable).
- System Time (Processing Time): When the system of record actually learned about and processed that data.
According to Databricks, this dual-axis approach is critical for industries facing heavy regulation. Under SEC Rule 17a-4 and FINRA recordkeeping rules, firms must be able to reconstruct records exactly as they existed at any point in the past. The company noted that the SEC's recordkeeping sweep has resulted in over $2 billion in fines across more than 100 firms since 2021, highlighting the high stakes of accurate data reconstruction.
Technical Implementation and Data Integrity
To facilitate this tracking, the AUTO CDC engine manages four system columns for each target table: __START_AT and __END_AT for business time, and __SYSTEM_START_AT and __SYSTEM_END_AT for system time. This allows a single logical fact to be represented by several physical rows, each capturing a specific combination of business and system versions.
A key technical guarantee of the new engine is its ability to handle out-of-order events. If a correction arrives with an earlier timestamp than data already processed, the engine automatically rewrites the affected history rather than simply appending a new record. This ensures that the bitemporal timeline remains accurate regardless of the order in which data lands in the pipeline. Databricks states this functionality is applicable to both dimension tables, such as symbol masters, and fact tables, such as trade histories or sensor readings.
Transition to Open Source
In a significant move for the broader data ecosystem, Databricks is expanding these AUTO CDC capabilities into the open-source Apache Spark 4.2 release. By moving these features into the open-source domain, Databricks is positioning declarative pipelines as a standard for handling complex CDC use cases across the industry.
The expansion aims to resolve three primary challenges that standard CDC patterns currently face:
- Handling out-of-order bitemporal timelines.
- Processing partial record updates without corrupting existing data sets.
- Maintaining auditability that remains accessible beyond standard storage retention windows.
Impact on Data Engineering and Compliance
For developers and data engineers, the primary impact is a drastic reduction in code complexity. By replacing manual "MERGE" statements with simple declarations of sequencing columns, teams can reduce the likelihood of "fragile" pipelines that break when data arrives late or out of sequence.
For the financial services industry and other regulated sectors, the update provides a more robust path to compliance. The ability to answer what a system "believed" on a specific reporting date—even if corrections were made later—provides the audit trail necessary to satisfy federal recordkeeping mandates.
What’s Next
Databricks has confirmed that these bitemporal AUTO CDC capabilities will be available in the upcoming Apache Spark 4.2 release. Users of Spark Declarative Pipelines (SDP) can begin utilizing the dual-axis tracking by declaring the two sequencing columns within their pipeline configurations. No further timeline for additional features was announced.