GPU Systems Architecture at Baseten
Model serving bottlenecks live in kernel launches and GPU memory stalls. Baseten couples raw CUDA and OpenAI Triton operator graphs with an asynchronous Rust execution harness over PyO3. The orchestrator dispatches token generation steps using pre-pinned host memory rings. No Python GIL overhead intrudes here. Instead, an event-driven worker loop manages CUDA streams directly from Rust, executing speculative decoding passes without thread context switches. This architecture keeps GPU compute units saturated during traffic spikes.
You join an engineering unit operating model inference for companies like Cursor, Notion, and OpenEvidence. Base compensation ranges from $180,000 to $360,000 USD alongside equity in San Francisco.
Key Technical Problems You Solve
- Profile GPU kernel execution using Nsight Systems and PyTorch profiler to isolate SM occupancy bottlenecks.
- Implement custom Triton and CUDA flash attention operators optimized for FP8 and FP4 numerical formats on H100 and B200 hardware.
- Design zero-copy serialization between Tokio network listeners and pinned device buffers.
- Maintain deterministic tail latency during dynamic batch expansion and asynchronous speculative token verification.
Qualifications
- Demonstrated ability writing high-performance Rust and modern C++ interfacing with CUDA or Triton runtimes.
- Deep understanding of GPU memory hierarchies, asynchronous copy engines, and warp-level primitives.
- Comfortable navigating Linux low-level systems debugging with perf, bpftrace, and gdb.
- Disclosed compensation: $180,000 to $360,000 USD base salary plus comprehensive equity and healthcare benefits.
