Correctness Engineering at LanceDB
LanceDB stores petabyte-scale multimodal datasets using Lance, an open-source columnar format engineered for fast random access and vector queries. Zero-copy access is standard. Lance avoids Parquet decoding overhead by reading Arrow record batches directly from storage via memory maps and asynchronous prefetching. Concurrency bugs in distributed object storage can corrupt version manifests or invalidate IVF-PQ index trees. That cannot happen. As Senior System Test Engineer, you build deterministic chaos testing harnesses that simulate object store network partitions, abrupt process aborts, and concurrent multi-writer conflicts. Your test suites run against real S3 and Azure Blob backends to confirm ACID guarantees under real stress.
This role offers $180,000 to $250,000 USD in base compensation along with equity ownership.
Core Production Responsibilities
- Develop stress testing frameworks that probe data integrity across large Arrow record batches.
- Write Jepsen-style automated scenarios evaluating conflict resolution during concurrent table compaction.
- Identify memory leaks and allocation hotspots within Lance Rust storage engines under prolonged read loads.
- Integrate continuous regression suites validating SIMD vector search accuracy and IVF-PQ clustering.
Technical Background
- Proficiency writing production or verification software in Rust, C++, or Python.
- Experience verifying distributed systems, database engines, or complex storage formats.
- Solid grasp of transactions, serializability models, and cloud object store consistency semantics.
- Disclosed base salary: $180,000 to $250,000 USD plus equity.
