LanceDB • 📍 US / Canada Remote •
$180k - $245k USD
Implement custom DataFusion physical execution nodes, disk-based IVF-PQ vector indices, and high-concurrency S3 byte-range schedulers.
#datafusion • $183k-$245k USD Base
DataFusion in Production Rust: Architecture & Open Roles
Extensible SQL & DataFrame query planner and vectorized execution engine
1. Why Teams Deploy DataFusion
Apache DataFusion is an extensible, vectorized SQL query engine written in Rust on top of Apache Arrow. Instead of spending five years writing a custom SQL parser, cost-based optimizer, and hash-join execution engine from scratch, database companies embed DataFusion as their query layer. InfluxDB 3.0 IOx, GreptimeDB, LanceDB, and Databricks' native Spark accelerator (Comet) all build directly on DataFusion.
2. DataFusion Kernel & Memory Internals
Queries enter DataFusion via `sqlparser-rs` and convert into a `LogicalPlan`. Custom analyzer and optimizer rules perform predicate pushdown, projection pruning, and join reordering before producing an executable `ExecutionPlan`. Physical operators run in a push-based, morsel-driven volcano pipeline over partitioned `SendableRecordBatchStream` channels, spilling to disk gracefully when group-by hash tables exceed memory budgets.
3. DataFusion Interview & Hiring Bar
Database startups actively poach contributors who have written custom `TableProvider` implementations, physical optimizer rules, or User Defined Aggregate Functions (UDAFs) in DataFusion.
Open DataFusion Engineering Positions (2)
InfluxData • 📍 US / Europe Remote •
$185k - $245k USD
Contribute upstream to Apache DataFusion, reduce memory pressure during wide time-series compactions, and optimize Arrow Flight SQL throughput.