Senior Staff Engineer - Infrastructure Systems

2 days, 4 hours ago
Full-time
Lead
Software Development
RUNWARE

RUNWARE

RUNWARE provides an affordable API that enables AI developers to efficiently run image, video, and custom generative AI models without the need for extensive infrastructure or machine learning expertise.

Internet Software & Services
1-10
Founded 2023

Description

  • Own versioned, reproducible GPU software baselines spanning firmware, kernels, NVIDIA drivers, CUDA libraries, and container tooling.
  • Qualify driver and CUDA releases across GPU generations, frameworks, and model families before fleet-wide rollout.
  • Investigate GPU failures to root cause across hardware, firmware, drivers, runtimes, and applications.
  • Build GPU acceptance, burn-in, diagnostic, interconnect, and real-workload tests for nodes and racks.
  • Unify GPU telemetry from DCGM, drivers, kernel logs, and BMC systems to detect, classify, and remediate faults automatically.
  • Manage GPU exposure to workloads through container runtimes, device plugins, GPU Operator, MIG, sharing policies, and tenant isolation.
  • Lead software and validation bring-up for new GPU generations, pods, NVLink, and NVSwitch environments.
  • Benchmark inference workloads across GPU hardware and software stacks to inform placement, pricing, and purchasing.
  • Work with NVIDIA and upstream open-source projects to report, reproduce, and resolve driver and firmware issues.
  • Use AI tooling to accelerate log triage, test generation, release qualification, and runbook automation.

Requirements

  • 12+ years of systems software development and operations experience, including several years operating production GPU software at scale.
  • Deep hands-on knowledge of the NVIDIA stack, including kernel drivers, CUDA, NVML, DCGM, Container Toolkit, and version compatibility.
  • Proven ability to diagnose GPU failures across hardware, firmware, drivers, runtimes, and applications, including Xid, ECC, PCIe, NVLink, thermal, and memory issues.
  • Strong Linux internals expertise, including kernels, device drivers, PCIe, NUMA, IOMMU, cgroups, containers, perf, eBPF, and ftrace.
  • Fluency in Python and at least one systems language: C, C++, Rust, or Go; able to use CUDA for testing and bug reproduction.
  • Experience operating GPUs on Kubernetes and bare metal, including device plugins, GPU Operator, MIG, image builds, and large-scale provisioning.
  • Working knowledge of GPU inference frameworks such as PyTorch, TensorRT, and ONNX Runtime, including memory management and multi-GPU execution.
  • Clear technical writing skills for design documents, postmortems, and operational updates.
  • Experience using AI tools to improve engineering productivity and automation.
  • Experience with Blackwell hardware, NVLink/NVSwitch, liquid-cooled racks, NCCL, GPUDirect, RDMA, firmware qualification, or inference-serving systems is preferred.

Benefits

  • Remote-first work environment with twice-yearly in-person company retreats.
  • Flexible working hours outside core collaboration periods.
  • Generous paid time off, including vacation, sick days, and public holidays.
  • Meaningful stock options.
  • Paid maternity, paternity, and caregiver leave.
  • Dedicated downtime after intensive release cycles.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Senior Systems Engineer, Cameras/Imaging [Remote]

DigitalFish 11-50 Media

DigitalFish is seeking an imaging simulation engineer to develop high-fidelity camera simulators and validation tools that support next-generation consumer camera architecture, image quality, and prototype development.

C C++ Machine Learning MATLAB Python
3 hours, 57 minutes ago

Senior System Engineer

Pingwind 51-200 Internet Software & Services

PingWind is seeking a Senior Systems Engineer to design, integrate, and maintain secure, reliable enterprise IT systems supporting federal government mission objectives across multiple locations.

Cybersecurity
3 hours, 57 minutes ago

PEGA-Senior System Architect

Qualysoft 201-500 Internet Software & Services

As a Lead System Engineer, lead the planning, delivery, and operation of complex system landscapes for a large cross-departmental public-sector IT project, ensuring secure, stable, high-performing systems.

3 hours, 57 minutes ago

Sr. Systems Developer

True 51-200 Professional Services

True is seeking a mid-level IT Support professional to resolve complex technical issues, improve support operations, and deliver high-quality service to internal and external stakeholders across its global talent advisory business.

Domo Salesforce
3 days, 4 hours ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers