Principal Site Reliability Engineer, Infrastructure Architecture- AI Infrastructure Operations

1 day, 1 hour ago
Full-time
Lead
DevOps and Infrastructure
Nscale

Nscale

Nscale is building the engine of superintelligence — full-stack AI infrastructure powering the world's most powerful systems, from ground to cloud.

information technology & services
501-1000
Founded 2024
$4185M raised

Description

  • Own the end-to-end reference architecture for compute, network fabric, storage, control plane, and core infrastructure services.
  • Design multi-site and multi-region topologies, including failure domains, tenant isolation, secure enclaves, and trust boundaries.
  • Define target-state architecture and a multi-year roadmap for platform retirement, consolidation, and rebuilds.
  • Lead cross-functional architecture reviews and author or approve RFCs and architecture decision records.
  • Set pre-production architectural standards for SLOs, observability, capacity models, dependency maps, and failure-mode documentation.
  • Conduct pre-mortems and red-team design reviews to identify single points of failure and hidden dependencies.
  • Co-own operational readiness, incident severity, post-incident review, and production-access policies with the SRE team.
  • Support architectural incident investigations and carry design fixes through implementation.
  • Plan and review site bring-ups, fleet upgrades, live workload migrations, disaster-recovery tests, and game days.
  • Mentor senior engineers and use incident and on-call data to prioritize architectural improvements.

Requirements

  • 10+ years of experience building and operating large-scale production infrastructure, including on-call ownership for systems you designed.
  • Several years of experience as a staff, principal, or architect-level engineer.
  • Demonstrated record of production architectures, with the ability to explain design tradeoffs, failures, and improvements.
  • Experience leading major incidents, large migrations, or site bring-ups.
  • Deep expertise in distributed systems design, including consistency, consensus, replication, partition handling, and graceful degradation.
  • Strong design-level networking knowledge, including L3 fabrics, BGP, EVPN/VXLAN, segmentation, and multi-site connectivity.
  • Experience designing secure multi-tenant platforms, including identity, PKI, secrets management, network isolation, and zero-trust boundaries.
  • Knowledge of change management, staged rollouts, SLO-based alerting, and post-incident reviews.
  • Strong technical writing skills for design documents, RFCs, and ADRs, plus the ability to influence architectural decisions across teams.
  • Strong software engineering skills and sound judgment balancing reliability, delivery velocity, and cost.
  • Experience architecting AI or HPC platforms with GPU clusters, InfiniBand or RoCE, and Slurm or Kubernetes GPU scheduling (preferred).
  • Experience with sovereign, regulated, air-gapped, bare-metal, hybrid-cloud, high-scale observability, or multi-cluster Kubernetes environments (preferred).

Benefits

  • Base salary range of $150,000–$250,000 USD, with potential bonus, equity, and/or commission.
  • Equity package and compensation reviews every 12 months.
  • Remote-first, flexible work environment with autonomy to structure your day.
  • Medical, dental, and vision insurance.
  • Flexible paid time off and parental leave.
  • Retirement plan participation.
  • Tailored career progression and development opportunities.
  • Workplace accommodations and an inclusive equal-opportunity environment.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Sr. Site Reliability Engineer (Remote, Mexico)

Join IO Connect Services as an SRE to build, scale, and operate reliable cloud services and production infrastructure for global customer deployments.

Ansible AWS Azure Chef CI/CD Datadog GCP HDFS Java JavaScript Kubernetes PowerShell Puppet Python Ruby Terraform
3 hours, 34 minutes ago

Senior SRE/DevOps Engineer

Metabase 51-200 IT Services

Metabase is seeking an Operations Engineer to build, operate, and scale the AWS-based infrastructure supporting its rapidly growing hosted Metabase Cloud analytics product.

AWS CI/CD Datadog Go Grafana Kubernetes Prometheus Python Terraform
3 hours, 49 minutes ago

Site Reliability Engineer

Filevine 501-1000 Specialized Consumer Services

Filevine is hiring a Site Reliability Engineer to improve the reliability, scalability, and operational maturity of its Legal AI platform for legal teams.

AWS Bash CI/CD Kubernetes Pulumi Python Terraform
2 days, 3 hours ago

Senior Site Reliability Engineer

Filevine 501-1000 Specialized Consumer Services

Filevine is seeking a Site Reliability Engineer to strengthen the reliability, scalability, and operational automation of its Legal AI platform.

AWS Bash CI/CD Datadog Go Grafana Kubernetes New Relic Python
2 days, 3 hours ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers