Lance Format Systems at LanceDB
Multimodal AI models need fast random access to high-dimensional embeddings and raw images without scanning entire dataset partitions. Standard Parquet formats fail under random point lookups on cloud storage. Lance solves this. Speed matters. The format organizes vector and columnar data into indexed micro-batches optimized for random disk seeks on NVMe and S3. Built completely in Rust on top of Apache Arrow and DataFusion, Lance provides native vector indexing via IVF-PQ algorithms. The engine avoids serialization penalties by passing Arrow record batches straight into user processes via PyO3. You will architect enterprise retrieval pipelines that ingest terabytes of vector embeddings daily while keeping query latencies under 15 milliseconds.
Disclosed base compensation ranges from $180,000 to $250,000 USD with equity participation.
Core Production Responsibilities
- Guide engineering teams integrating LanceDB into large-scale multimodal retrieval stacks.
- Tune query execution plans in DataFusion for high-concurrency vector filtering.
- Optimize PyO3 zero-copy memory transfers between Rust native cores and Python ML frameworks.
- Conduct deep-dive benchmarks evaluating disk read amplification across object stores.
Technical Background
- Strong proficiency in Rust and Python systems development.
- Hands-on experience with Apache Arrow, DataFusion, Parquet, or columnar database internals.
- Understanding of vector search concepts such as HNSW, IVF-PQ, and cosine distance indexing.
- Disclosed base salary: $180,000 to $250,000 USD plus equity.
