Systems Architecture at LanceDB
Lance achieves 100x faster random access than Parquet on cloud object storage (S3/GCS) using a Rust async I/O scheduler, custom bit-packing encodings, and zero-copy Arrow integration.
Design columnar storage formats (Lance) in Rust built on Apache Arrow and DataFusion for petabyte-scale LLM training datasets and vector search. Base pay for this remote role sits at $180,000 to $245,000 USD plus equity.
What You Will Own in Production
- Implement custom DataFusion physical execution nodes, disk-based IVF-PQ vector indices, and high-concurrency S3 byte-range schedulers.
- Profile CPU cache misses, syscall bottlenecks, and tail latency spikes across Apache Arrow, DataFusion, Parquet services using hardware perf counters and flamegraphs.
- Review architectural RFCs and maintain strict memory-safety invariants in async and unsafe Rust blocks.
Background We Look For
- Production track record shipping low-level systems in Rust, C, or C++ with deep fluency in Apache Arrow, DataFusion, Parquet, PyO3, Tokio.
- Hands-on experience debugging Linux kernel networking, NVMe storage, or GPU compute pipelines under heavy load.
- Compensation: $180,000 to $245,000 USD per year, equity grant, and full remote workstation budget (US / Canada Remote).
