Inference Runtime at Modal
Cold boot latency hurts. At Modal, containers boot in under 800 milliseconds on warm node pools. Speed is survival. You will build core runtime components in Rust that interface directly with Linux cgroups v2, namespaces, and NVIDIA driver user-space libraries. Standard OCI container runtimes add hundreds of milliseconds in filesystem overlays, seccomp filters, and cgroup reconciliation loops. Modal cuts through this complexity with lean daemonlets that map pre-warmed memory segments directly into guest environments. Fast. Custom eBPF programs trace socket transitions and balance incoming requests across GPU instances without intermediate proxy hops. The stack relies heavily on Tokio for event scheduling and io_uring for non-blocking disk reads when loading multi-gigabyte model weights into pinned memory buffers.
Disclosed base salary spans $220,000 to $300,000 USD plus equity ownership in Modal.
Core Production Responsibilities
- Design and operate the low-latency Linux container runtime managing GPU worker processes.
- Write asynchronous Rust services handling worker handshakes, health checks, and task dispatching.
- Optimize CUDA memory allocation routines and direct NVLink peer-to-peer transfers during container boot.
- Instrument host kernel paths using eBPF to diagnose scheduling jitter and packet drop anomalies.
Technical Background
- Deep experience in systems programming with Rust, C, or modern C++.
- Hands-on knowledge of Linux internals: namespaces, cgroups v2, epoll, and memory management.
- Familiarity with GPU architectures, CUDA runtime calls, or serverless execution platforms.
- Base compensation range: $220,000 to $300,000 USD with equity.
