RemoteLeaf Logo
Log in Sign up

Senior AI Backend Engineer - Agent Evaluation & Quality

3 weeks, 1 day ago
Saudi Arabia
Full-time
Senior
AI (Artificial Intelligence)
Software Development
CI/CD LLM NLP Python System Design TypeScript
Apply Now
Salla E-Commerce Platform

Salla E-Commerce Platform

Salla E-Commerce Platform (salla.sa) offers smart and easy e-commerce solutions for merchants, enabling direct selling to customers with ease.

Internet Software & Services
251-1K
Founded 2016
$8M raised
View All Jobs 3

Description

  • Own the evaluation stack by designing and building LLM-as-judge systems, calibrating them against human labels, and making agent quality measurable by agent and failure mode.
  • Build per-PR evaluation harnesses and regression detection integrated into CI to enforce release quality automatically.
  • Develop user simulators to create test coverage and adversarial scenarios before they reach real users.
  • Feed production failures back into evaluation sets so the system improves over time.
  • Partner with product teams to turn definitions of success into concrete, measurable criteria.
  • Contribute to agent development and hardening, especially the components you understand best through evaluation.

Requirements

  • Strong software engineering fundamentals with production experience in Python, TypeScript, or similar languages.
  • Experience designing clean APIs and systems, with testing and CI/CD practices.
  • Hands-on experience building with LLMs and agents, including RAG, tool/function calling, or orchestration frameworks such as LangGraph or LangChain.
  • Ability to understand how LLM/agent systems behave and fail in practice.
  • Measurement mindset with experience reasoning about metrics, calibration, and experiments.
  • Production experience operating LLM systems with awareness of reliability, latency, cost, and observability.
  • 5+ years of software engineering experience with recent hands-on LLM/agent work.
  • Direct experience evaluating LLM/agent systems, including offline/online evals, LLM-as-judge, or systematic regression testing (preferred).
  • Experience with observability tools such as Arize or LangSmith (preferred).
  • Arabic language or NLP experience (preferred).
  • E-commerce or merchant-facing product experience (preferred).

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

ShopMy

Senior Full Stack Engineer

ShopMy 11-50 Internet Software & Services

Senior Full Stack Engineer at ShopMy, working on the platform behind creator-driven commerce to build and scale APIs, user interfaces, and internal systems.

Europe Full-time Senior Backend Engineer Full-stack Engineer
AWS JavaScript Node.js React Sass UI Design
14 hours, 4 minutes ago
Eltropy

Senior Backend Engineer

Eltropy 51-250 Communications Equipment

Senior Backend Engineer at a remote company, responsible for building and supporting backend systems and APIs that are efficient, scalable, and reliable.

India Senior Backend Engineer
AWS Docker Git Go Java Kubernetes MongoDB Node.js PHP PostgreSQL Redis SQL
14 hours, 19 minutes ago
Scale AI

SWE Fellow - Human Frontier Collective (UK)

Scale AI 251-1K Diversified Consumer Services

The Human Frontier Collective Fellowship at Scale is a fully remote contract role supporting AI research by designing, evaluating, and interpreting advanced generative AI systems with interdisciplinary experts and partner labs.

United Kingdom Contract Senior AI (Artificial Intelligence) Software Engineer
C++ Generative AI Java JavaScript Machine Learning Python Rust Swift
14 hours, 19 minutes ago
Underdog Fantasy

Senior Software Engineer - Exchange

Underdog Fantasy Media

Underdog is hiring an engineer to build and scale the core exchange infrastructure powering its prediction markets offering, with a focus on low-latency, high-reliability systems.

United States Full-time Senior Backend Engineer
$160k-$240k
C C++ Go Java REST API Rust System Design WebSockets
14 hours, 19 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers