RemoteLeaf Logo
Log in Sign up

Freelance Agent Evaluation Engineer

1 hour, 6 minutes ago
India
Part-time
Senior
AI (Artificial Intelligence)
Software Development
CI/CD Docker FastAPI GitHub Actions JavaScript Kafka Linux PostgreSQL Pytest Python React Redis REST API TypeScript
Apply Now
Mindrift.ai: Be the “I” in AI

Mindrift.ai: Be the “I” in AI

Join 10,000+ experts earning $15-50/hr training AI models remotely. Flexible freelance work, weekly payments. No AI experience required. Apply in 5 minutes.

Internet Software & Services
View All Jobs 60

Description

  • Build virtual company environments with codebases, infrastructure, and realistic development context such as conversations, documentation, and tickets.
  • Assemble tasks from intermediate states of a virtual company by crafting prompts, defining evaluation criteria, and ensuring tasks are solvable and fair.
  • Design tasks in isolated developer-workstation environments using Linux, terminal tools, MCP servers, and real web application codebases.
  • Write tests that accept all correct solutions and reject incorrect solutions without being overly strict or overly lenient.
  • Iterate on tests with an AI agent to confirm they catch real issues and do not break valid approaches.
  • Review code written by agents and analyze why solutions succeeded or failed.
  • Design edge cases and adversarial scenarios to better distinguish strong and weak solutions.
  • Incorporate feedback from expert QA reviewers and revise work based on quality scoring.

Requirements

  • Please submit your CV in English and indicate your level of English proficiency.
  • This is project-based, non-permanent work rather than full-time employment.
  • Degree in Computer Science, Software Engineering, or a related field.
  • 5+ years of software development experience, primarily in Python.
  • Experience with FastAPI, pytest, async/await, subprocess, and file operations.
  • Background in full-stack development, including React-based interfaces and robust back-end systems.
  • Experience writing functional and integration tests, not just running them.
  • Experience with Docker containers and infrastructure tools such as Postgres, Kafka, and Redis.
  • Understanding of CI/CD, especially GitHub Actions as a user, including triggers, labels, and reading results.
  • English proficiency at B2 level.

Benefits

  • Project-based flexibility: choose when and how to work.
  • Estimated workload of about 20 hours per project, depending on complexity.
  • Compensation up to $12 per hour equivalent, depending on level and pace of contribution.
  • Opportunity to work with leading tech companies on AI evaluation projects.
  • Potential access to different projects with varying earning levels based on scope and expertise.
  • Work on challenging problems involving frontier AI coding agents.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Toloka

AI Trainer - Freelance Data Annotator

Toloka 251-1K Internet Software & Services

Toloka Annotators is seeking freelance annotators to support generative AI projects by reviewing and labeling online content for factual accuracy and quality when projects are available.

United States Part-time Junior AI (Artificial Intelligence) Data Annotator
Up to $48k
Generative AI
17 minutes ago
GitLab

AI Transformation Owner, Marketing

GitLab 1K-5K Internet Software & Services

GitLab is hiring an AI Transformation Owner to lead the AI strategy, adoption, and workflow redesign for a business function in partnership with an Executive Sponsor and Enterprise AI.

United States Canada United Kingdom Ireland Full-time Lead AI (Artificial Intelligence) Marketing Manager
$153k-$259k
DevSecOps
1 hour, 56 minutes ago
Nice Côte d'Azur

Automation Engineer

Nice Côte d'Azur Hotels, Restaurants & Leisure

NiCE CXone is hiring a Software Engineer in Test to build and maintain test automation for its cloud customer experience platform within Agile product teams.

United States Full-time Junior SDET (Software Development Engineer in Test)
Agile AWS C# .NET Selenium SQL
2 hours, 52 minutes ago
Resilient Co

SDET

Resilient Co 11-50 Professional Services

Resilient Co. is hiring a remote Senior Software Development Engineer in Test (SDET) for an engineering contractor role focused on improving software quality through automated testing across the SDLC.

Colombia Costa Rica Peru Argentina Brazil Contract Senior SDET (Software Development Engineer in Test)
Angular Apache Spark AWS Azure C CI/CD Docker GCP JavaScript Kubernetes MongoDB MySQL .NET Oracle PostgreSQL Python R React Redis SQL SQL Server
3 hours, 10 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers