Module delta_io 

Delta Lake I/O integration module.

This module provides the actual I/O operations for Delta Lake tables via the deltalake crate. All functions are feature-gated behind delta-lake.

Architecture

The I/O module is separate from the business logic in delta.rs to allow:

• Testing business logic without the deltalake dependency
• Clean separation of concerns (buffering/epoch management vs. actual writes)
• Easy mocking for unit tests

Exactly-Once Semantics

Delta Lake’s transaction log supports application-level transaction metadata via the txn action. We use this to store (writer_id, epoch) pairs, enabling exactly-once semantics:

1. On recovery, read txn metadata to find the last committed epoch for this writer
2. Skip epochs <= last committed (idempotent replay)
3. Each write includes the epoch in txn metadata

Structs

CompactionResult

Result of a compaction (OPTIMIZE) operation.

MergeResult

Result of a MERGE (upsert) operation.

Functions

get_last_committed_epoch

Retrieves the last committed epoch for a writer from Delta Lake’s txn metadata.

get_latest_version

Returns the latest committed version via the log store.

get_table_schema

Extracts the Arrow schema from a Delta Lake table.

map_cdf_to_changelog

Maps CDF _change_type → _op (I/U/D), drops update_preimage rows and CDF metadata columns (_change_type, _commit_version, _commit_timestamp). Returns None if all rows were preimages.

merge_changelog

Atomic changelog MERGE: inserts, updates, and deletes in one Delta commit.

open_or_create_table

Opens an existing Delta Lake table or creates a new one.

read_batches_at_version

Reads record batches from a specific Delta Lake table version.

read_cdf_batches

Reads CDF batches for a version range via scan_cdf().

read_version_diff

Reads only the rows added in a specific Delta Lake version.

resolve_catalog_options

Resolves catalog-aware table URI and merges catalog-specific storage options.

run_compaction

Runs an OPTIMIZE compaction on a Delta Lake table.

run_vacuum

Runs VACUUM on a Delta Lake table, deleting old unreferenced files.

write_batches

Writes batches to a Delta Lake table with exactly-once semantics.