RemoteLeaf Logo
Log in Sign up

Freelance Agent Evaluation Engineer

1 hour, 19 minutes ago
South Africa
Part-time
Senior
AI (Artificial Intelligence)
Artificial Intelligence and Machine Learning
Docker FastAPI GitHub Actions JavaScript Kafka Linux PostgreSQL Pytest Python React Redis REST API TypeScript
Apply Now
Mindrift.ai: Be the “I” in AI

Mindrift.ai: Be the “I” in AI

Join 10,000+ experts earning $15-50/hr training AI models remotely. Flexible freelance work, weekly payments. No AI experience required. Apply in 5 minutes.

Internet Software & Services
View All Jobs 57

Description

  • Build virtual company environments with codebases, infrastructure, development history, and supporting context such as conversations, documentation, and tickets.
  • Assemble tasks from intermediate states of the virtual company by crafting prompts, defining evaluation criteria, and ensuring each task is solvable and fairly judged.
  • Design isolated developer-workstation environments that include Linux tools, MCP servers, and a real web application codebase.
  • Write tests that correctly accept valid solutions and reject incorrect ones without being overly strict or too permissive.
  • Iterate on tests with an AI agent to verify that they catch real issues and do not break correct solutions.
  • Review agent-written code to understand why a solution succeeded or failed and design edge cases and adversarial scenarios.
  • Incorporate feedback from expert QA reviewers and revise tasks to meet quality standards.
  • Collaborate with AI throughout the task creation and evaluation process to build challenging benchmarks for frontier coding models.

Requirements

  • Please submit your CV in English and indicate your level of English proficiency.
  • Degree in Computer Science, Software Engineering, or a related field.
  • 5+ years of software development experience, primarily in Python.
  • Experience with Python tools and patterns such as FastAPI, pytest, async/await, subprocess, and file operations.
  • Background in full-stack development, including React-based interfaces and robust back-end systems.
  • Experience writing functional and integration tests, not just executing tests.
  • Experience working with Docker containers and infrastructure tools such as Postgres, Kafka, and Redis.
  • Working familiarity with CI/CD, especially GitHub Actions triggers, labels, and reading workflow results.
  • English proficiency at B2 level.
  • Comfort reading and reasoning about code across the stack, even without expertise in every area.

Benefits

  • Project-based work rather than permanent employment.
  • Part-time, non-permanent opportunities with flexible participation.
  • Estimated 20 hours of work per project, depending on complexity.
  • You choose when and how to work, subject to submission deadlines and acceptance criteria.
  • Compensation of up to $24 per hour equivalent, depending on contribution level and pace.
  • Access to project-based AI opportunities with leading tech companies.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Toloka

Freelance Annotator (English) - AI Trainer

Toloka 251-1K Internet Software & Services

Toloka is seeking freelance annotators to support Generative AI projects by reviewing and labeling online content for accuracy, quality, and appropriateness.

Belgium Denmark Finland Ireland Norway Sweden Part-time Junior AI (Artificial Intelligence) Data Annotator
Up to $46k
Generative AI
34 minutes ago
Symmetrio

Enterprise Oncology IT & AI Strategy Leader

Symmetrio Professional Services

Symmetrio is recruiting an Enterprise Oncology IT & AI Strategy Leader for a global oncology technology and healthcare innovation company to shape enterprise technology direction and guide strategic decisions across oncology, interoperability, and AI.

United States Full-time Lead AI (Artificial Intelligence) Product Strategist
2 hours, 5 minutes ago
instacart.careers

Director of Product, Agentic Commerce

instacart.careers 1K-5K Internet Software & Services

Instacart is hiring a Director of Product for Agentic Commerce to lead the strategy and execution for AI-driven shopping products that improve consumer experiences and drive retailer outcomes at scale.

United States Full-time Lead AI (Artificial Intelligence)
$242k-$307k
E-commerce Generative AI Machine Learning SQL
2 hours, 39 minutes ago
SWORD Health

QA Analyst

SWORD Health 251-1K Health Care Providers & Services

Sword Health is hiring a QA Analyst for its Web Development Team in Porto to help validate product quality and ensure web software is functional, reliable, and user-friendly.

Portugal Full-time Junior QA Engineer
$23k-$36k
Android Git iOS Jenkins JIRA Linux Postman
2 hours, 45 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers