Struct StreamingTableProvider 

pub struct StreamingTableProvider { /* private fields */ }

A DataFusion table provider backed by a streaming source.

This allows streaming sources to be registered as tables in DataFusion’s SessionContext and queried using SQL. The provider handles:

• Schema exposure to DataFusion’s catalog
• Projection pushdown to the source
• Filter pushdown when supported by the source

Usage

let source = Arc::new(ChannelStreamSource::new(schema));
let provider = StreamingTableProvider::new("events", source);
ctx.register_table("events", Arc::new(provider))?;

let df = ctx.sql("SELECT * FROM events WHERE id > 100").await?;

Implementations

impl StreamingTableProvider

pub fn new(name: impl Into<String>, source: StreamSourceRef) -> Self

Creates a new streaming table provider.

Arguments

• name - Name of the table (used for display/debugging)
• source - The streaming source backing this table

pub fn name(&self) -> &str

Returns the table name.

pub fn source(&self) -> &StreamSourceRef

Returns the underlying streaming source.

Trait Implementations

impl Debug for StreamingTableProvider

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl TableProvider for StreamingTableProvider

fn as_any(&self) -> &dyn Any

Returns the table provider as Any so that it can be downcast to a specific implementation.

fn schema(&self) -> SchemaRef

Get a reference to the schema for this table

fn table_type(&self) -> TableType

Get the type of this table for metadata/catalog purposes.

fn supports_filters_pushdown( &self, filters: &[&Expr], ) -> Result<Vec<TableProviderFilterPushDown>, DataFusionError>

Specify if DataFusion should provide filter expressions to the TableProvider to apply during the scan.

fn scan<'life0, 'life1, 'life2, 'life3, 'async_trait>( &'life0 self, _state: &'life1 dyn Session, projection: Option<&'life2 Vec<usize>>, filters: &'life3 [Expr], _limit: Option<usize>, ) -> Pin<Box<dyn Future<Output = Result<Arc<dyn ExecutionPlan>, DataFusionError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait, 'life3: 'async_trait,

Create an [ExecutionPlan] for scanning the table with optionally specified projection, filter and limit, described below.

fn constraints(&self) -> Option<&Constraints>

Get a reference to the constraints of the table. Returns:

fn get_table_definition(&self) -> Option<&str>

Get the create statement used to create this table, if available.

fn get_logical_plan(&self) -> Option<Cow<'_, LogicalPlan>>

Get the [LogicalPlan] of this table, if available.

fn get_column_default(&self, _column: &str) -> Option<&Expr>

Get the default value for a column, if available.

fn scan_with_args<'a, 'life0, 'life1, 'async_trait>( &'life0 self, state: &'life1 dyn Session, args: ScanArgs<'a>, ) -> Pin<Box<dyn Future<Output = Result<ScanResult, DataFusionError>> + Send + 'async_trait>>

where 'a: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, Self: 'async_trait,

Create an [ExecutionPlan] for scanning the table using structured arguments.

fn statistics(&self) -> Option<Statistics>

Get statistics for this table, if available Although not presently used in mainline DataFusion, this allows implementation specific behavior for downstream repositories, in conjunction with specialized optimizer rules to perform operations such as re-ordering of joins.

fn insert_into<'life0, 'life1, 'async_trait>( &'life0 self, _state: &'life1 dyn Session, _input: Arc<dyn ExecutionPlan>, _insert_op: InsertOp, ) -> Pin<Box<dyn Future<Output = Result<Arc<dyn ExecutionPlan>, DataFusionError>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, Self: 'async_trait,

Return an [ExecutionPlan] to insert data into this table, if supported.

fn delete_from<'life0, 'life1, 'async_trait>( &'life0 self, _state: &'life1 dyn Session, _filters: Vec<Expr>, ) -> Pin<Box<dyn Future<Output = Result<Arc<dyn ExecutionPlan>, DataFusionError>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, Self: 'async_trait,

Delete rows matching the filter predicates.

fn update<'life0, 'life1, 'async_trait>( &'life0 self, _state: &'life1 dyn Session, _assignments: Vec<(String, Expr)>, _filters: Vec<Expr>, ) -> Pin<Box<dyn Future<Output = Result<Arc<dyn ExecutionPlan>, DataFusionError>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, Self: 'async_trait,

Update rows matching the filter predicates.