RemoteLeaf Logo
Log in Sign up

Freelance Agent Evaluation Engineer

1 day, 7 hours ago
Czech Republic, Estonia, Latvia, Lithuania
Part-time
Senior
AI (Artificial Intelligence)
Software Development
Docker FastAPI GitHub Actions JavaScript Kafka Linux PostgreSQL Pytest Python React Redis REST API TypeScript
Apply Now
Mindrift.ai: Be the “I” in AI

Mindrift.ai: Be the “I” in AI

Join 10,000+ experts earning $15-50/hr training AI models remotely. Flexible freelance work, weekly payments. No AI experience required. Apply in 5 minutes.

Internet Software & Services
View All Jobs 30

Description

  • Build virtual company environments with codebase, infrastructure, documentation, conversations, and ticket history.
  • Assemble tasks from intermediate states of virtual companies and define clear, fair evaluation criteria.
  • Design tasks in isolated workstation-like environments with Linux, development tools, MCP servers, and real web application codebases.
  • Write tests that accept all correct solutions and reject incorrect ones.
  • Iterate on tests with an AI agent to verify coverage, fairness, and robustness.
  • Review agent-written code to determine why solutions succeed or fail.
  • Design edge cases and adversarial scenarios that expose model weaknesses.
  • Incorporate feedback from expert QA reviewers and refine task quality accordingly.

Requirements

  • Degree in Computer Science, Software Engineering, or a related field.
  • 5+ years of software development experience, primarily in Python.
  • Experience with FastAPI, pytest, async/await, subprocess, and file operations.
  • Full-stack development experience, including React-based interfaces and JavaScript/TypeScript.
  • Experience building robust back-end systems.
  • Experience writing functional and integration tests, not just running them.
  • Experience working with Docker containers and infrastructure tools such as Postgres, Kafka, and Redis.
  • Working knowledge of CI/CD, especially GitHub Actions triggers, labels, and result interpretation.
  • English proficiency at B2 level.
  • Comfort reading and reasoning about code across the stack.
  • Preferred: experience as a test automation specialist.

Benefits

  • Project-based, non-permanent work with flexible participation.
  • Estimated 20 hours of work per project, depending on complexity.
  • Flexible scheduling: choose when and how to work.
  • Compensation of up to $30 per hour equivalent, depending on contribution level and pace.
  • Opportunity to work on leading tech company AI projects.
  • Payment for completed tasks that meet the acceptance criteria.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Coupa Software

Principal Agentic Architect - 11385

Coupa Software 1K-5K Internet Software & Services

Coupa is hiring a remote Principal Agentic Architect in the UK to lead strategic AI-led engagements that transform procurement and finance operations from manual workflows to agentic orchestration.

United Kingdom Lead AI (Artificial Intelligence) Product Strategist
Generative AI LLM
37 minutes ago
Resilient Co

SDET

Resilient Co 11-50 Professional Services

Resilient Co. is hiring a remote Senior Software Development Engineer in Test (SDET) for an engineering contractor role focused on improving software quality through automated testing across the SDLC.

Colombia Costa Rica Peru Argentina Brazil Contract Senior SDET (Software Development Engineer in Test)
Angular Apache Spark AWS Azure C CI/CD Docker GCP JavaScript Kubernetes MongoDB MySQL .NET Oracle PostgreSQL Python R React Redis SQL SQL Server
56 minutes ago
AssureSoft

QA Automation Engineer

AssureSoft 51-250 Internet Software & Services

AssureSoft is hiring a QA professional to support long-term software projects by executing manual and automated testing across web, API, and mobile platforms.

Latin America Full-time Mid Level SDET (Software Development Engineer in Test)
Agile Android Appium CI/CD Cypress iOS JavaScript JMeter Playwright Postman React REST API Scrum TypeScript
1 hour, 32 minutes ago
Wing Assistant

AI Voice Trainer

Wing Assistant 51-250 Professional Services

Wing Data is hiring freelance voice data contributors for remote, project-based AI speech training work that uses short voice sessions to help improve client speech systems.

India Freelance Entry Level AI (Artificial Intelligence)
$4k-$8k
1 hour, 35 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers