I'm the author of Polars.NET, a project which aimed to bring pola-rs to .NET world.
Recently I got a issue that asked me to add delta table support for Polars.NET, that is really a good idea, then I tried my best. But since there is no direct rust API from pola-rs let me to call like scan_delta or sink_delta, I have to build the solution by combining pola-rs and deltalake rust crate.
Scanning is not that hard but for write side, I tried to export polars dataframe to arrow record batch to let deltalake help me to write, but I have to turn on datafusion as an extra dependency and SQL parser is also needed here, which will make the final binary size huge(near 80MiB). The final solution I chose is to build delta calculation layer by pure polars expression engine, and use deltalake crate to commit transactions.
Thing turns out that everything goes well since the benchmark result is quite amazing. Because I'm gluing everything in Rust rather than C# side, Z-ordering and deletion vector can also be handed by polars and some rust functions, which is efficient
I'm the author of Polars.NET, a project which aimed to bring pola-rs to .NET world. Recently I got a issue that asked me to add delta table support for Polars.NET, that is really a good idea, then I tried my best. But since there is no direct rust API from pola-rs let me to call like scan_delta or sink_delta, I have to build the solution by combining pola-rs and deltalake rust crate. Scanning is not that hard but for write side, I tried to export polars dataframe to arrow record batch to let deltalake help me to write, but I have to turn on datafusion as an extra dependency and SQL parser is also needed here, which will make the final binary size huge(near 80MiB). The final solution I chose is to build delta calculation layer by pure polars expression engine, and use deltalake crate to commit transactions.
Thing turns out that everything goes well since the benchmark result is quite amazing. Because I'm gluing everything in Rust rather than C# side, Z-ordering and deletion vector can also be handed by polars and some rust functions, which is efficient