RemoteLeaf Logo
Log in Sign up

Freelance Agent Evaluation Engineer

1 hour, 42 minutes ago
South Africa
Part-time
Senior
AI (Artificial Intelligence)
Software Development
Docker FastAPI JavaScript Kafka PostgreSQL Python React Redis TypeScript
Apply Now
Mindrift.ai: Be the “I” in AI

Mindrift.ai: Be the “I” in AI

Join 10,000+ experts earning $15-50/hr training AI models remotely. Flexible freelance work, weekly payments. No AI experience required. Apply in 5 minutes.

Internet Software & Services
View All Jobs 295

Description

  • Build realistic simulated developer environments with codebases, infrastructure, and contextual artifacts such as tickets, docs, and conversations.
  • Design tasks from intermediate environment states, including the prompt and clear criteria for what counts as solved.
  • Write tests that validate agent solutions, accepting correct approaches and rejecting incorrect ones.
  • Review AI agent solutions, analyze failure cases, and refine tasks and tests based on QA feedback.
  • Iterate on evaluation materials until the task set is fair, robust, and solvable.
  • Help assess how well AI coding agents handle real-world developer tasks.

Requirements

  • 5+ years of software development experience.
  • Experience with the core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, and Redis.
  • Experience writing functional and integration tests.
  • English proficiency at B2 level or higher.
  • CV must be submitted in English.
  • Availability to complete project tasks within the required deadline.
  • Experience working in realistic development environments is preferred.

Benefits

  • Project-based work with flexible scheduling.
  • Up to $40/hr equivalent compensation, depending on level and pace.
  • Estimated ~20 hours per task, with work completed on your own schedule.
  • Paid project participation after passing qualification(s).

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Accenture

QA Automation Engineer

Accenture 100K+ Professional Services

Accenture Federal Services is hiring a QA Automation Engineer to build and maintain automated testing for federal software applications in a remote role with occasional DMV-area travel.

United States Full-time Mid Level SDET (Software Development Engineer in Test)
$111k-$140k
Agile AWS Azure Cypress Docker Gatling GCP Git GitHub Actions GitLab CI Jenkins Jest JMeter JUnit K6 Kubernetes Playwright Postman Pytest REST API Selenium SQL TestNG
42 minutes ago
Nimble Robotics

HIL Test / Automation Engineer

Nimble Robotics 51-250 Industrial Conglomerates

Nimble is hiring an engineer to own a core HIL regression pillar for its autonomous robotics supply chain platform, building automated validation infrastructure for the V6 push.

United States Full-time Junior SDET (Software Development Engineer in Test)
$130k-$170k
C++ CI/CD Embedded Systems Jenkins MATLAB Python Rust
1 hour, 27 minutes ago
Gramian Consultancy Group

COBOL Developer (Legacy Systems & AI Training)

Gramian Consultancy Group Professional Services

Gramian Consultancy is seeking an experienced COBOL Developer to support AI training by analyzing legacy enterprise applications, documenting business logic, and creating realistic technical scenarios.

Turkey Poland Romania India Argentina Brazil Contract Senior AI (Artificial Intelligence) Software Engineer
COBOL
1 hour, 42 minutes ago
Intetics

AI / ML Engineer.

Intetics 1K-5K Internet Software & Services

Intetics is seeking an experienced AI/ML Engineer to build and maintain production machine learning solutions for enterprise analytics, forecasting, and AI-driven decision-making.

Poland Romania Bulgaria Czech Republic Ukraine Full-time Senior AI (Artificial Intelligence)
Feature Engineering Generative AI Machine Learning MLflow MLOps NLP Python
1 hour, 42 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers