Principal Agentic Engineer

2 hours, 14 minutes ago
Full-time
Lead
Software Development
ZeroFox

ZeroFox

ZeroFox provides cutting-edge external cybersecurity solutions powered by AI and human intelligence to proactively defend organizations from threats beyond the corporate perimeter.

Internet Software & Services
251-1K
Founded 2013
$160M raised

Description

  • Design and implement production agent architectures, including state management, error handling, retry logic, graceful degradation, and human-in-the-loop escalation.
  • Build evaluation and testing frameworks for non-deterministic agent workflows, including offline tests, synthetic data generation, regression checks, and post-deploy monitoring.
  • Implement orchestration patterns such as multi-agent coordination, tool-calling chains, memory management, and context window optimization.
  • Define deployment and governance practices for agents, including versioning, rollback, behavioral telemetry, and anomaly detection.
  • Instrument agents with tracing, logging, and observability to make production behavior debuggable.
  • Establish architectural standards and best practices for agentic development through design reviews, pairing, and mentorship.
  • Work on reliability, evaluation, and operational practices for agent systems operating on adversarial data.
  • Collaborate on reusable patterns that improve how new agent projects are built and shipped to production.

Requirements

  • 7+ years of software engineering experience building production systems, with hands-on experience designing and deploying LLM-based agents.
  • Deep knowledge of agent reliability patterns, including state management, error boundaries, escalation logic, context management, and tool-calling failure modes.
  • Experience with agent orchestration frameworks and understanding of tradeoffs between existing frameworks and building custom solutions.
  • Strong backend engineering fundamentals, including testing, monitoring, deployment, debugging, and performance optimization.
  • Hands-on experience with retrieval-augmented systems and understanding of how retrieval quality affects agent behavior.
  • Experience building systems that handle adversarial or noisy input, such as in cybersecurity, fraud detection, content moderation, or similar domains.
  • Familiarity with cloud-based AI deployment, including observability, reliability, and cost considerations.
  • Preferred: Experience defining agent governance practices such as versioning, behavioral telemetry, and rollback in production.
  • Preferred: Background in cybersecurity or an adjacent domain with high-noise, adversarial data.
  • Preferred: Experience with tool-use architectures and integration patterns for connecting agents to external systems.
  • Preferred: Track record of raising engineering capability across a team through mentoring, pairing, or design review.

Benefits

  • Competitive compensation.
  • Community-driven culture with employee events.
  • Regular catered lunches for in-office work, plus snacks and drinks available daily.
  • Generous time off.
  • Comprehensive health benefits and a 401(k) plan.
  • Fun, modern workspace.
  • Respectful and nourishing work environment where every opinion is heard and employees are encouraged to participate in the culture.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Principal Software Engineer (AI) (CAD)

PointClickCare 1K-5K Health Care Providers & Services

PointClickCare is hiring a remote Principal Software Engineer (AI) to lead the design and delivery of production-grade AI systems and shared platforms that support healthcare products and operations in a regulated environment.

HIPAA Python
14 minutes ago

Principal Software Engineer (AI) ( US)

PointClickCare 1K-5K Health Care Providers & Services

PointClickCare is seeking a Principal Software Engineer (AI) to lead the design and delivery of production-grade AI systems that drive its healthcare platform’s AI-first and data-first transformation.

HIPAA
14 minutes ago

Backend Engineer - Personalization

Spotify Media

Backend Engineer for Spotify’s Personalization team, building real-time recommendation systems and content-serving infrastructure for the Now Playing View used by hundreds of millions of listeners.

Java Machine Learning Microservices System Design
29 minutes ago

Java Backend Engineer (AI/LLM Chatbot, Customer Service)

Binance 5K-10K Capital Markets

Binance is seeking a Java Backend Engineer to develop and support AI/LLM chatbot systems for customer service within its global blockchain ecosystem.

Apache Spark Blockchain Flink Java LLM Python System Design
29 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers