RemoteLeaf Logo
Log in Sign up

Freelance Agent Evaluation Engineer

3 hours, 14 minutes ago
United Kingdom
Part-time
Senior
QA Engineer
Software Development
Docker FastAPI GitHub Actions JavaScript Kafka Linux PostgreSQL Pytest Python React Redis REST API TypeScript
Apply Now
Mindrift.ai: Be the “I” in AI

Mindrift.ai: Be the “I” in AI

Join 10,000+ experts earning $15-50/hr training AI models remotely. Flexible freelance work, weekly payments. No AI experience required. Apply in 5 minutes.

Internet Software & Services
View All Jobs 57

Description

  • Build virtual company environments with codebases, infrastructure, and development history that simulate realistic work contexts.
  • Assemble tasks from intermediate states of the virtual company by crafting prompts, defining evaluation criteria, and validating solvability and fairness.
  • Design isolated workstation-based tasks using Linux development environments, CLI tools, MCP servers, and real web application codebases.
  • Write tests that correctly accept all valid solutions and reject incorrect ones.
  • Iterate on tests with an AI agent to ensure they catch real issues without blocking good approaches.
  • Review agent-written code to determine why solutions succeeded or failed and identify edge cases and adversarial scenarios.
  • Incorporate feedback from expert QA reviewers and revise work to meet quality criteria.
  • Work collaboratively with AI as part of the task creation and evaluation process.

Requirements

  • Please submit your CV in English and indicate your level of English proficiency.
  • Experience in software development, ideally 5+ years, primarily with Python.
  • Hands-on experience with FastAPI, pytest, async/await, subprocess, and file operations.
  • Background in full-stack development, including React-based interfaces and robust back-end systems.
  • Experience writing functional and integration tests, not just executing them.
  • Experience with Docker containers and infrastructure tools such as Postgres, Kafka, and Redis.
  • Understanding of CI/CD, especially using GitHub Actions to inspect triggers, labels, and results.
  • Degree in Computer Science, Software Engineering, or a related field.
  • English proficiency at B2 level.
  • Comfort reading and reasoning about code across the full stack.

Benefits

  • Project-based, non-permanent work with flexible participation.
  • Tasks are estimated at around 20 hours, depending on complexity.
  • You choose when and how to work, subject to submission deadlines and acceptance criteria.
  • Compensation up to $50 per hour equivalent, depending on level and pace of contribution.
  • Opportunity to work on leading tech companies' AI evaluation projects.
  • Paid after completing tasks that are accepted by the project criteria.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

SpaceX

Software Engineer, Flight Software (Starship)

SpaceX 10K-50K Aerospace & Defense

SpaceX is hiring a Software Engineer for Starship Flight Software to help design, develop, test, and integrate software that controls and simulates flight systems for its fully reusable launch vehicle program.

United States Full-time Junior Software Engineer
$125k-$175k
C++ Rust
1 hour, 27 minutes ago
instacart.careers

Senior Software Engineer II, (ML/AI Platform)

instacart.careers 1K-5K Internet Software & Services

Instacart is hiring a Senior Engineer for its ML/AI Platform team to build the internal platform that powers organization-wide AI model training, deployment, and advanced inference workflows.

United States Full-time Senior Machine Learning Engineer Software Engineer
$192k-$242k
1 hour, 31 minutes ago
Samsara

Staff Software Engineer

Samsara 1K-5K IT Services

Samsara is hiring a remote Staff Software Engineer to help shape the developer experience and AI-enabled engineering tooling that supports its internal teams and the broader physical operations platform.

United States Full-time Lead Software Engineer
$162k-$290k
CI/CD Go Python TypeScript
1 hour, 49 minutes ago
SWORD Health

QA Analyst

SWORD Health 251-1K Health Care Providers & Services

Sword Health is hiring a QA Analyst for its Web Development Team in Porto to help validate product quality and ensure web software is functional, reliable, and user-friendly.

Portugal Full-time Junior QA Engineer
$23k-$36k
Android Git iOS Jenkins JIRA Linux Postman
2 hours, 4 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers