Systems Architecture at Together AI
Inference latency kills throughput. Our production stack pairs custom FlashAttention kernels with an asynchronous Rust execution harness over PyO3 to maximize tensor core utilization. The orchestrator dispatches incoming requests to GPU device memory through lock-free ring buffers without heap allocations. Python stalls. We bypass its runtime entirely by parsing incoming gRPC frames directly into pre-allocated pinned host memory. Every microsecond counts. This design keeps scheduler latency deterministic under peak load.
Forward deployed systems engineers partner directly with engineering leaders to scale demanding model serving pipelines. Annual base pay ranges from $270,000 to $300,000 USD plus equity.
What You Will Own in Production
- Profile end-to-end token generation pipelines to diagnose CUDA stream serialization, PCI bus contention, and memory bandwidth bottlenecks.
- Implement zero-copy serialization paths between Tonic gRPC workers and device memory using pinned host buffers.
- Tune speculative decoding loops and KV-cache compaction routines for large context windows up to 1M tokens.
- Diagnose distributed deadlock conditions across NCCL collectives during live fine-tuning jobs.
Background We Look For
- Demonstrated experience writing systems-level Rust or modern C++ interfacing directly with Nvidia CUDA or ROCm runtimes.
- Comfort profiling Linux system metrics with perf, bpftrace, and Nsight Systems under high concurrency.
- Practical knowledge of PyO3 or C foreign function interfaces for binding native code to Python deep learning workflows.
- Compensation: $270,000 to $300,000 USD base salary plus comprehensive equity and benefits (San Francisco, CA or US Remote).
