Software Engineer, Infrastructure

2 weeks ago
Senior
Software Development
fal

fal

fal provides a generative media platform that enables developers to efficiently run diffusion models through ready-to-use AI inference, training APIs, and user-friendly UI playgrounds, facilitating the creation of innovative media experiences.

Internet Software & Services
1-10
Founded 2021

Description

  • Build and maintain a Python-based fleet tracking system for the full server lifecycle, including procurement, provisioning, pricing, availability, health, and RMAs.
  • Develop server management tooling that automates provisioning, health checks, GPU diagnostics, recovery, and alerting.
  • Create and maintain metrics, dashboards, and alerts for fleet-wide hardware health issues such as GPU errors, disk failures, network problems, and thermal events.
  • Use AI extensively to automate alerting, detection, and recovery workflows.
  • Implement and enforce OS-level security controls, including hardening baselines, SELinux/AppArmor policies, SSH key management, vulnerability scanning, and compliance automation.
  • Manage and optimize storage systems for model weights, checkpoints, and scratch space across NVMe arrays, NFS, parallel file systems, and object storage.
  • Tune Linux systems for AI workloads, including kernel parameters, NUMA topology, CPU pinning, hugepages, I/O schedulers, and GPU driver stack optimization.
  • Develop automated error detection and recovery processes across the infrastructure.
  • Work with external partners to resolve technical issues that automation cannot fix.

Requirements

  • 3+ years of experience managing bare-metal and cloud-based server fleets at scale (100+ nodes).
  • Strong Python software engineering skills with experience building production tooling.
  • Deep Linux systems knowledge, including boot process, kernel tuning, networking, storage, systemd, cgroups, namespaces, and performance profiling.
  • Strong experience with configuration management and infrastructure-as-code tools such as Ansible, Terraform, and cloud-init.
  • Solid understanding of storage technologies including LVM, RAID, NVMe, NFS, Lustre or GPFS, and Linux I/O stack tuning.
  • Familiarity with hardware diagnostics and common failure modes for GPUs, NVMe, NICs, and memory.
  • Experience building internal tools or dashboards for infrastructure visibility.
  • Excellent communication skills and the ability to drive technical decisions across teams.
  • A self-starter mindset with strong ownership and a bias toward fast execution and continuous improvement.
  • Preferred experience with network diagnostics and configuration tools such as VLAN, VXLAN, ECMP, BGP, and tcpdump.
  • Preferred experience with NVIDIA GPU infrastructure, including driver management, DCGM, NVLink/NVSwitch diagnostics, RDMA, and InfiniBand/RoCEv2.
  • Preferred experience with AMD GPUs.
  • Preferred experience with bare-metal and VM provisioning tools such as PXE/iPXE, Kickstart, libvirt, and Qemu/KVM.
  • Preferred familiarity with compliance frameworks such as SOC 2 and ISO 27001.
  • Role is open to Mid, Senior, and Staff level candidates.
  • Location requirement: Turkey.

Benefits

  • Interesting and challenging work.
  • A lot of learning and growth opportunities.
  • Regular team events and offsites.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Software Engineer, Test Infrastructure

Skyline Eco-Adventures 11-50 Hotels, Restaurants & Leisure

Zipline is seeking a Senior Software Systems Validation Engineer to lead end-to-end validation of autonomous drone delivery software, from simulation through full-system flight testing, for safe deployment and operating-domain expansion.

C++ Python Rust SQL
15 hours, 13 minutes ago

Senior Software Engineer - UI/UX

Dragos 251-1K Professional Services

Dragos is hiring a Software Engineer to build the user-facing experiences that help security professionals detect, investigate, and respond to cyber threats against critical infrastructure.

Cybersecurity Design Systems React TypeScript
15 hours, 28 minutes ago

Senior Software Engineer

Xebia 1K-5K Internet Software & Services

Xebia is hiring Senior Software Engineers for an international digital payments and fintech project focused on building and evolving a scalable, reliable payment platform.

Agile Apache AWS CI/CD CSS Generative AI Git Hibernate HTML Java JavaScript Jenkins JSON JUnit Maven Microservices REST API Selenium Shell Scripting Spring Boot SQL XML
15 hours, 28 minutes ago

Software Engineer, Edge

Kargo 251-1K Media

Kargo is hiring a software engineer to develop edge-device machine vision and AI systems that improve freight dock operations through real-time perception and fleet management.

C++ Computer Vision Linux Machine Learning Python
15 hours, 28 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers