Inference Observability at Baseten
Large language model inference pipelines demand granular telemetry without degrading time-to-first-token or throughput. Every millisecond counts. Baseten deploys custom Rust collector daemons on each GPU node to record model server events, KV cache utilization, and queue times. Rather than polling external endpoints, collectors attach to kernel tracepoints and NVML counters via non-blocking channels. Fast zero-copy ring buffers hold data points before flushing upstream to time-series storage. No CPU overhead leaks into model evaluation. You will build high-throughput OpenTelemetry ingestion pipelines that ingest millions of spans per second while preserving microsecond timestamp accuracy across distributed GPU workers.
Base compensation ranges between $165,000 and $330,000 USD depending on seniority, complemented by equity ownership.
Core Production Responsibilities
- Write low-footprint Rust telemetry agents that run alongside tensor inference engines.
- Construct distributed tracing pipelines tracking requests across ingress proxies, model shards, and KV caches.
- Develop eBPF sensors monitoring socket buffer queues and PCIe bus memory transfers.
- Scale ingestion gateways handling millions of metrics per second with minimal CPU overhead.
Technical Background
- Demonstrated experience writing production systems code in Rust, Go, or C++.
- Familiarity with distributed tracing standards such as OpenTelemetry, Jaeger, or Prometheus.
- Understanding of Linux performance tooling, asynchronous IO, and multi-threaded event loops.
- Disclosed base salary: $165,000 to $330,000 USD plus equity.
