RemoteLeaf Logo
Log in Sign up

Freelance Agent Evaluation Engineer

1 day, 3 hours ago
Mexico, Chile, Colombia, Costa Rica, Uruguay, Peru
Part-time
Senior
AI (Artificial Intelligence)
Software Development
Docker FastAPI GitHub Actions JavaScript Kafka Linux PostgreSQL Pytest Python React Redis REST API TypeScript
Apply Now
Mindrift.ai: Be the “I” in AI

Mindrift.ai: Be the “I” in AI

Join 10,000+ experts earning $15-50/hr training AI models remotely. Flexible freelance work, weekly payments. No AI experience required. Apply in 5 minutes.

Internet Software & Services
View All Jobs 30

Description

  • Build virtual company environments that include a codebase, infrastructure, and supporting context such as conversations, documentation, and tickets.
  • Assemble tasks from intermediate states of a virtual company, including crafting prompts and defining fair evaluation criteria.
  • Design tasks in isolated workstation-like environments with Linux tools, MCP servers, and real web application codebases.
  • Write tests that accept all correct solutions and reject incorrect ones without being overly strict or too lenient.
  • Iterate on tests with an AI agent to verify they catch real issues and do not break valid solutions.
  • Review agent-written code and analyze why an agent succeeded or failed on a task.
  • Design edge cases and adversarial scenarios to stress-test model behavior.
  • Incorporate feedback from expert QA reviewers and refine work based on quality criteria.
  • Submit completed tasks by the deadline while meeting the listed acceptance criteria.

Requirements

  • Please submit your CV in English and indicate your level of English proficiency.
  • Degree in Computer Science, Software Engineering, or a related field.
  • 5+ years of software development experience, primarily in Python.
  • Hands-on experience with FastAPI, pytest, async/await, subprocess, and file operations.
  • Background in full-stack development, including React-based interfaces and robust back-end systems.
  • Experience writing functional and integration tests, not just running them.
  • Experience with Docker containers and infrastructure tools such as Postgres, Kafka, and Redis.
  • Understanding of CI/CD, especially GitHub Actions triggers, labels, and reading results.
  • English proficiency at B2 level.
  • Comfort reading and reasoning about code across the stack.

Benefits

  • Project-based, non-permanent work with flexible participation.
  • Option to choose when and how to work.
  • Estimated 20 hours of tasks, depending on complexity.
  • Compensation of up to $21 per hour equivalent, depending on contribution level and pace.
  • Opportunity to work on leading tech-company AI projects.
  • Paid project work after passing qualification(s) and joining a project.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Coupa Software

Principal Agentic Architect - 11385

Coupa Software 1K-5K Internet Software & Services

Coupa is hiring a remote Principal Agentic Architect in the UK to lead strategic AI-led engagements that transform procurement and finance operations from manual workflows to agentic orchestration.

United Kingdom Lead AI (Artificial Intelligence) Product Strategist
Generative AI LLM
37 minutes ago
Wing Assistant

AI Voice Trainer

Wing Assistant 51-250 Professional Services

Wing Data is hiring freelance voice data contributors for remote, project-based AI speech training work that uses short voice sessions to help improve client speech systems.

India Freelance Entry Level AI (Artificial Intelligence)
$4k-$8k
1 hour, 34 minutes ago
Intetics

1075| QA Engineer (AI / Agent-based Systems)

Intetics 1K-5K Internet Software & Services

Intetics Inc. is seeking a QA Engineer to join a small team building AI-driven and agent-based product features, with responsibility for validating both traditional product behavior and the reliability of LLM and agent workflows.

Poland Romania Ukraine Full-time Mid Level AI (Artificial Intelligence) QA Engineer
LLM Python
1 hour, 45 minutes ago
SWORD Health

QA Analyst

SWORD Health 251-1K Health Care Providers & Services

Sword Health is hiring a QA Analyst for its Web Development Team in Porto to help validate product quality and ensure web software is functional, reliable, and user-friendly.

Portugal Full-time Junior QA Engineer
$23k-$36k
Android Git iOS Jenkins JIRA Linux Postman
1 hour, 49 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers