Skip to main content

scan_incremental

laminar_connectors::lakehouse::iceberg_incremental

Function scan_incremental 

Source 
pub async fn scan_incremental(
    table: &Table,
    old_snapshot_id: i64,
    new_snapshot_id: i64,
    select_columns: &[String],
) -> Result<Vec<RecordBatch>, ConnectorError>
Expand description

Reads only data files added between old_snapshot_id and new_snapshot_id.

Walks the new snapshot’s manifest list, selects manifests where added_snapshot_id > old_snapshot_id, loads those manifests, and reads the ADDED data files via the table’s scan API.

If any manifest entry has a Deleted status (compaction, overwrite, or explicit delete), the incremental path is unsafe — falls back to a full scan of the new snapshot so the caller gets the complete table state.

§Errors

Returns ConnectorError::ReadError on I/O or scan failure.