Why Together AI Uses Rust Here
Latency spikes kill throughput. Together Inference Engine combines FlashAttention-3/FP8 CUDA kernels with a lock-free Rust scheduler and HTTP/3 gRPC streaming gateway serving 100k+ concurrent token streams. Sub-millisecond token dispatch matters.
We are hiring a Senior Rust AI Runtime Engineer (vLLM / TensorRT-LLM Backend) (US / EU Remote) at $195,000 to $265,000 USD. Work directly on speculative decoding tree verification, PyO3 bindings for custom CUDA graphs, and Tonic bi-directional gRPC streaming backpressure.
Day-to-Day Engineering Scope
- Architect and benchmark core crates built around CUDA and Tokio and Tonic and PyO3 and Serde.
- Replace allocation-heavy paths with zero-copy buffers, arena allocators, or lock-free ring queues.
- Pair with upstream open-source maintainers and internal platform teams to ship fault-tolerant releases.
Technical Expectations & Pay
- Demonstrated mastery of Rust ownership, pinning, lifetimes, and concurrency primitives (CUDA, Tokio, Tonic).
- Comfort reading assembly output (`cargo asm`), kernel tracepoints, or wire protocol captures when abstractions leak.
- Verified package: $195,000 to $265,000 USD base salary plus competitive equity and global remote healthcare.
