LanceDB • 📍 US / Canada Remote •
$180k - $245k USD
Implement custom DataFusion physical execution nodes, disk-based IVF-PQ vector indices, and high-concurrency S3 byte-range schedulers.
#parquet • $183k-$247k USD Base
Parquet in Production Rust: Architecture & Open Roles
Columnar cloud storage format with dictionary, RLE & Zstd compression
1. Why Teams Deploy Parquet
The official Rust `parquet` crate (maintained inside the `arrow-rs` monorepo) powers petabyte-scale data lakes and observability backends that store cold data directly on Amazon S3, GCS, or Cloudflare R2. Modern zero-disk architectures like Quickwit, InfluxDB 3.0, and WarpStream bypass local attached storage entirely by writing compacted Parquet files to object storage.
2. Parquet Kernel & Memory Internals
Reading Parquet fast over high-latency S3 requires aggressive metadata pruning. The `ParquetRecordBatchStreamBuilder` inspects footer `RowGroup` min/max statistics, page indexes, and Bloom filters to issue coalesced HTTP byte-range requests (`GetObject` ranges) for only the exact column chunks needed by a query. Late materialization decodes filter columns first before decompressing wide payload columns.
3. Parquet Interview & Hiring Bar
Storage engine teams seek developers who understand Parquet page encoding (Delta-binary-packed, RLE_DICTIONARY), async byte-range coalescing over `object_store`, and background compaction scheduling.
Open Parquet Engineering Positions (5)
Polars • 📍 Worldwide Remote •
$170k - $225k USD
Optimize out-of-core streaming hash joins, cardinality estimation in the lazy query planner, and Parquet predicate pushdown.
Neon (Databricks) • 📍 Worldwide Remote •
$190k - $265k USD
Improve L0-to-L1 compaction throughput over `tokio-epoll-uring`, lower tail latency on cold page reconstructions, and harden Paxos WAL quorum recovery.
Materialize • 📍 US / Canada Remote •
$190k - $255k USD
Reduce arrangement memory footprints, optimize multi-way stream join planning, and tune S3 Parquet compaction inside the Persist storage engine.
InfluxData • 📍 US / Europe Remote •
$185k - $245k USD
Contribute upstream to Apache DataFusion, reduce memory pressure during wide time-series compactions, and optimize Arrow Flight SQL throughput.