ML Platform Engineer, Distributed Training

21 hours, 40 minutes ago
Full-time
Senior
DevOps and Infrastructure
Egen.ai

Egen.ai

Egen.ai specializes in providing technology services that leverage cloud computing, data analytics, and artificial intelligence to enhance document intelligence and drive productivity and growth for its clients.

IT Services
Founded 2000

Description

  • Partner with the client infrastructure team to support GKE GPU training workloads, including node pools, scheduling, quotas, and capacity.
  • Diagnose and resolve root causes of training-platform failures.
  • Tune RDMA/GPUDirect and NCCL networking to improve multi-node GPU training throughput.
  • Run, scale, and debug distributed PyTorch and Ray Train workloads.
  • Profile training jobs and remove compute, memory, input-pipeline, and interconnect bottlenecks.
  • Design and execute cluster and training-stack benchmarks for throughput, scaling efficiency, and cost per run.
  • Build and maintain model evaluation harnesses to measure platform and training-recipe changes.
  • Implement telemetry, monitoring, and end-to-end tests for real training pipelines.
  • Proactively identify platform gaps and risks, communicate status, and support scientist users.
  • Explain and defend technical design decisions to client engineers and scientists.

Requirements

  • 5+ years of engineering experience, including recent hands-on work with GPU training infrastructure and large-scale training.
  • Strong Python and shell scripting skills, including production-quality code, testing, and code review.
  • Experience with GKE/Kubernetes on Google Cloud, GPU node pools, job queueing such as Kueue, and quota and capacity management.
  • Experience running and debugging multi-node distributed training with PyTorch, including FSDP, DeepSpeed, Megatron, or NeMo-style frameworks, and Ray Train.
  • Knowledge of RDMA/GPUDirect, NCCL tuning, and multi-node GB200 or H200 GPU topology.
  • Experience designing performance benchmarks and model evaluations to guide technical decisions.
  • Ability to reason deeply about system design and operation without relying on AI coding tools.
  • Demonstrated ownership, accountability, and clear communication in demanding, changing team environments.
  • Experience with Google Gemini, GPT-4, or LLaMA; prompt engineering; and fine-tuning.
  • Experience with Vertex AI, vector databases, semantic search, and orchestration frameworks such as LangChain or LlamaIndex.
  • Healthcare or life-sciences data experience, including HIPAA environments (preferred).
  • Experience with observability and telemetry for training platforms, including metrics, tracing, alerting, and end-to-end testing (preferred).

Benefits

  • Competitive salary and comprehensive benefits package.
  • Comprehensive health insurance.
  • Paid vacation/PTO, holidays, and sick leave.
  • Parental and bereavement leave.
  • 401(k) employer match.
  • Employee referral bonuses.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Principal / Staff Data Scientist

Xsolla 201-500 Internet Software & Services

Xsolla is seeking a Principal Machine Learning Engineer to lead the architecture and delivery of scalable production ML systems that advance its global video game commerce platform.

Deep Learning LLM Machine Learning MLflow MLOps Neural Networks Statistics
21 hours, 25 minutes ago

Machine Learning Engineer Lead

MUTT DATA 51-200 Internet Software & Services

Muttdata is seeking a remote Machine Learning Engineer Lead to guide teams in delivering scalable data and ML solutions for client decision-making, forecasting, automation, and personalization use cases.

Apache Airflow AWS Azure Databricks dbt Deep Learning Docker Feature Engineering GCP Generative AI Jupyter Keras Machine Learning MLflow MLOps NumPy Pandas Plotly Python PyTorch Scikit-learn SQL TensorFlow XGBoost
1 day, 18 hours ago

Machine Learning Engineer Databricks - Mexico

MUTT DATA 51-200 Internet Software & Services

Muttdata is seeking a Machine Learning Engineer to operate and support Databricks-based ML platforms and production pipelines for a multinational beverage company in Mexico City.

AWS Azure CI/CD Databricks dbt Deep Learning GCP Generative AI GitFlow Machine Learning MLflow Python SQL
1 day, 18 hours ago

Machine Learning Engineer

MUTT DATA 51-200 Internet Software & Services

Mutt Data is seeking a remote Machine Learning Engineer to build and optimize scalable ML solutions for client use cases including forecasting, e-commerce, and automation.

Apache Airflow AWS Azure Databricks dbt Deep Learning Docker Feature Engineering GCP Generative AI Jupyter Keras LLM Machine Learning MLflow MLOps NumPy Pandas Plotly Python PyTorch Scikit-learn SQL TensorFlow XGBoost
1 day, 18 hours ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers