RemoteLeaf Logo
Log in Sign up

Freelance Agent Evaluation Engineer

1 hour, 23 minutes ago
Spain
Part-time
Senior
AI (Artificial Intelligence)
Artificial Intelligence and Machine Learning
Docker FastAPI GitHub Actions JavaScript Kafka Linux PostgreSQL Pytest Python React Redis REST API TypeScript
Apply Now
Mindrift.ai: Be the “I” in AI

Mindrift.ai: Be the “I” in AI

Join 10,000+ experts earning $15-50/hr training AI models remotely. Flexible freelance work, weekly payments. No AI experience required. Apply in 5 minutes.

Internet Software & Services
View All Jobs 30

Description

  • Build virtual company environments with codebases, infrastructure, and supporting context such as conversations, documentation, and tickets.
  • Assemble tasks from intermediate states of a virtual company and define fair, solvable evaluation criteria.
  • Design isolated workstation-style tasks using Linux development environments, MCP servers, and real web application codebases.
  • Write tests that correctly accept valid solutions and reject incorrect ones without being too strict or too lenient.
  • Iterate on tests and task design with an AI agent to verify coverage of real problems and avoid false positives or negatives.
  • Review agent-written code to understand success and failure modes and design edge cases and adversarial scenarios.
  • Incorporate feedback from expert QA reviewers and improve task quality based on their assessments.
  • Collaborate with AI throughout the work process to create challenging evaluations for frontier coding models.

Requirements

  • Please submit your CV in English and indicate your level of English proficiency.
  • This is a project-based, non-permanent opportunity rather than a full-time employment role.
  • Degree in Computer Science, Software Engineering, or a related field is preferred.
  • 5+ years of software development experience, primarily with Python.
  • Experience with FastAPI, pytest, async/await, subprocess, and file operations.
  • Background in full-stack development, including React-based interfaces with JavaScript or TypeScript and robust back-end systems.
  • Experience writing functional and integration tests, not only running existing tests.
  • Experience with Docker containers and infrastructure tools such as Postgres, Kafka, and Redis.
  • Understanding of CI/CD workflows, especially GitHub Actions triggers, labels, and reading results.
  • English proficiency at B2 level.
  • Comfort reading and reasoning about code across the stack.
  • Experience as a software engineer or test automation specialist is a strong fit for this project.

Benefits

  • Earn up to $30 per hour equivalent, depending on contribution level and pace.
  • Flexible project-based work with no permanent employment commitment.
  • Choose when and how to work, with tasks estimated at about 20 hours depending on complexity.
  • Work on project-based AI opportunities for leading tech companies.
  • Complete tasks and get paid after passing qualification and meeting acceptance criteria.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

SecurityScorecard

Senior/Principal Product Manager, AI

SecurityScorecard 251-1K IT Services

SecurityScorecard is hiring a Principal Product Manager to lead major platform initiatives that shape strategy, execution, and the development of AI-enabled capabilities for cybersecurity products.

United States Full-time Lead AI (Artificial Intelligence) Product Manager
$175k-$285k
Agile Cybersecurity System Design
1 hour, 27 minutes ago
Wing Assistant

AI Trainer - India

Wing Assistant 51-250 Professional Services

Wing Data is hiring freelance AI Audio & Video Trainers in India for a remote, project-based study that collects human audio/video data to help train next-generation AI systems.

India Freelance Entry Level AI (Artificial Intelligence)
$4k-$10k
1 hour, 27 minutes ago
Relativity Space

Senior Build Quality Engineer

Relativity Space 251-1K Aerospace & Defense

Relativity Space is hiring a Quality Engineer to support the Terran R rocket program by building reliability and quality processes across design, manufacturing, test, and launch operations.

United States Full-time Senior QA Engineer
$122k-$167k
ERP
1 hour, 33 minutes ago
Wing Assistant

AI Trainer - Global

Wing Assistant 51-250 Professional Services

Wing Data is hiring freelance AI Audio & Video Trainers in the United Kingdom for project-based remote work supporting an AI training study through short audio and video participation tasks.

United Kingdom Freelance Entry Level AI (Artificial Intelligence)
$12k-$37k
1 hour, 34 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers