I’ve been building LaminarDB, an embedded streaming SQL database in Rust. Define streaming SQL (tumbling/hopping/session windows, materialized views), push data in, subscribe to results. Everything in-process, no JVM, no cluster.
I come from financial systems where the options for real-time aggregation were kdb+ (expensive licensing) or Flink (JVM cluster). Many use cases just need a library.
Some technical details:
∙Three-ring architecture. Ring 0 is the hot path: mmap’d state store, zero heap allocations, no locks. Ring 1 handles async checkpointing and compaction via RocksDB. Ring 2 is the control plane. Background work never blocks data processing.
∙Thread-per-core execution. Each core gets a dedicated Tokio runtime and owns its data. No shared thread pools, no work stealing.
∙DataFusion for SQL parsing and optimization, extended with custom streaming window operators and incremental materialized view maintenance. Arrow RecordBatch is the internal format, no conversion at boundaries.
Python bindings via PyO3 + maturin with zero-copy Arrow interop for pandas, Polars, PyArrow.
Currently at v0.13.1 with pre-built binaries. Performance targets in the README are all marked “In Progress,” no published benchmarks yet.
I’ve been building LaminarDB, an embedded streaming SQL database in Rust. Define streaming SQL (tumbling/hopping/session windows, materialized views), push data in, subscribe to results. Everything in-process, no JVM, no cluster.
I come from financial systems where the options for real-time aggregation were kdb+ (expensive licensing) or Flink (JVM cluster). Many use cases just need a library. Some technical details:
∙Three-ring architecture. Ring 0 is the hot path: mmap’d state store, zero heap allocations, no locks. Ring 1 handles async checkpointing and compaction via RocksDB. Ring 2 is the control plane. Background work never blocks data processing.
∙Thread-per-core execution. Each core gets a dedicated Tokio runtime and owns its data. No shared thread pools, no work stealing.
∙DataFusion for SQL parsing and optimization, extended with custom streaming window operators and incremental materialized view maintenance. Arrow RecordBatch is the internal format, no conversion at boundaries.
Python bindings via PyO3 + maturin with zero-copy Arrow interop for pandas, Polars, PyArrow.
Currently at v0.13.1 with pre-built binaries. Performance targets in the README are all marked “In Progress,” no published benchmarks yet.
Happy to go into detail on any of the internals.
[dead]