Distributed Storage Architecture at Modal
Disk seeks kill inference throughput. Modal built a custom distributed storage substrate in Rust to stream checkpoints across thousands of ephemeral nodes simultaneously. Every byte counts. Instead of routing through generic NFS mounts or slow object store wrappers, worker daemons pull chunked blobs directly from a local tiered NVMe cache backed by NVMe-oF. Metadata coordination runs over an embedded Raft consensus group implemented in asynchronous Rust. The storage engine uses io_uring for non-blocking direct I/O (O_DIRECT), preventing kernel buffer copies and eliminating cache thrashing. Read pipelines achieve line-rate saturation across 100GbE NICs.
Base compensation ranges between $250,000 and $300,000 USD plus meaningful equity ownership.
Core Production Responsibilities
- Build and maintain high-performance distributed object stores and caching proxies in Rust.
- Implement Raft state machines governing partition placement, failure detection, and log compaction.
- Leverage io_uring and O_DIRECT to maximize local NVMe SSD read speeds without memory amplification.
- Profile network bottlenecks and optimize peer chunk distribution across multi-cloud regions.
Technical Background
- Demonstrated track record of delivering production distributed storage systems in Rust or C++.
- Solid grasp of consensus algorithms (Raft, Paxos), replication protocols, and fault tolerance.
- Familiarity with Linux block storage, file systems, and asynchronous I/O architectures.
- Disclosed base salary: $250,000 to $300,000 USD.
