RemoteLeaf Logo
Log in Sign up

Senior AI Backend Engineer - Agent Evaluation & Quality

20 hours, 2 minutes ago
Saudi Arabia
Full-time
Senior
AI (Artificial Intelligence)
Software Development
CI/CD LLM NLP Python System Design TypeScript
Apply Now
Salla E-Commerce Platform

Salla E-Commerce Platform

Salla E-Commerce Platform (salla.sa) offers smart and easy e-commerce solutions for merchants, enabling direct selling to customers with ease.

Internet Software & Services
251-1K
Founded 2016
$8M raised
View All Jobs 3

Description

  • Own the evaluation stack by designing and building LLM-as-judge systems, calibrating them against human labels, and making agent quality measurable by agent and failure mode.
  • Build per-PR evaluation harnesses and regression detection integrated into CI to enforce release quality automatically.
  • Develop user simulators to create test coverage and adversarial scenarios before they reach real users.
  • Feed production failures back into evaluation sets so the system improves over time.
  • Partner with product teams to turn definitions of success into concrete, measurable criteria.
  • Contribute to agent development and hardening, especially the components you understand best through evaluation.

Requirements

  • Strong software engineering fundamentals with production experience in Python, TypeScript, or similar languages.
  • Experience designing clean APIs and systems, with testing and CI/CD practices.
  • Hands-on experience building with LLMs and agents, including RAG, tool/function calling, or orchestration frameworks such as LangGraph or LangChain.
  • Ability to understand how LLM/agent systems behave and fail in practice.
  • Measurement mindset with experience reasoning about metrics, calibration, and experiments.
  • Production experience operating LLM systems with awareness of reliability, latency, cost, and observability.
  • 5+ years of software engineering experience with recent hands-on LLM/agent work.
  • Direct experience evaluating LLM/agent systems, including offline/online evals, LLM-as-judge, or systematic regression testing (preferred).
  • Experience with observability tools such as Arize or LangSmith (preferred).
  • Arabic language or NLP experience (preferred).
  • E-commerce or merchant-facing product experience (preferred).

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Peloton

Software Engineer III, Content AI

Peloton 5K-10K Hotels, Restaurants & Leisure

Peloton is hiring a Software Engineer III in Warsaw to build and operate scalable Content AI microservices that power search, voice, and subtitles across its fitness platforms.

Poland Full-time Mid Level Backend Engineer Machine Learning Engineer
Agile AWS Bottle CI/CD Datadog Docker DynamoDB Elasticsearch GitHub Actions GraphQL gRPC Java Jenkins Kotlin Kubernetes Microservices MySQL NLP PostgreSQL Python Redis REST API Spring Boot Terraform
18 hours, 47 minutes ago
JustMarkets

AI Automation Specialist

JustMarkets 1-10 Capital Markets

JustMarkets is hiring an AI Automation Specialist to transform its marketing department by replacing manual workflows with scalable AI and automation systems that accelerate campaigns, content production, and global operations.

Europe Full-time Mid Level AI (Artificial Intelligence) Prompt Engineer
CRM JSON OAuth REST API Selenium SQL
18 hours, 47 minutes ago
Mitratech

Sr. Product Manager AI

Mitratech 1K-5K Professional Services

Mitratech Legal Solutions is hiring a Senior Product Manager, AI to drive enterprise legal software AI initiatives from concept to production across a shared platform and customer-facing experiences.

United States Full-time Senior AI (Artificial Intelligence) Product Manager
Generative AI LLM Machine Learning
19 hours, 2 minutes ago
Anaplan

Level 3 Support Analyst - AI Specialist

Anaplan 1K-5K Internet Software & Services

Anaplan is hiring a Level 3 Support professional to resolve complex customer technical issues across its AI-focused planning platform while partnering with engineering, product, operations, and customer success teams.

United Kingdom Full-time Junior AI (Artificial Intelligence) Technical Support Engineer
Grafana LLM Machine Learning
19 hours, 17 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers