Expand description
InferenceProvider trait and request/response types.
Providers are I/O only: inputs in, outputs + usage out. Task framing and response parsing belong to the adapter.
Structs§
- Inference
Params - Parameters that shape a request and version the result cache.
- Inference
Request - One homogeneous batch of inputs for a single task and model.
- Inference
Response - Result of a batch inference call.
- Usage
- Token and cost accounting for a batch call. Local backends report
Usage::ZERO.
Enums§
- Inference
Outputs - Per-row outputs of a batch. Shape is homogeneous within a request.
- Provider
Error - Errors a provider can return for a batch call.
Traits§
- Inference
Provider - I/O transport over a model backend. Shared as
Arc<dyn InferenceProvider>; driven from Ring 1, never Ring 0.