RemoteLeaf Logo
Log in Sign up

Freelance Agent Evaluation Engineer

15 hours, 31 minutes ago
India
Part-time
Senior
AI (Artificial Intelligence)
Software Development
Docker FastAPI JavaScript Kafka PostgreSQL Python React Redis TypeScript
Apply Now
Mindrift.ai: Be the “I” in AI

Mindrift.ai: Be the “I” in AI

Join 10,000+ experts earning $15-50/hr training AI models remotely. Flexible freelance work, weekly payments. No AI experience required. Apply in 5 minutes.

Internet Software & Services
View All Jobs 192

Description

  • Build realistic simulated developer environments with codebases, infrastructure, tickets, documentation, and conversation history.
  • Design coding tasks from intermediate states of those environments, including the prompt and definition of successful completion.
  • Create test cases that validate AI agent solutions and distinguish correct approaches from incorrect ones.
  • Review agent outputs, analyze failures, and refine tasks and tests based on QA feedback.
  • Ensure tasks are solvable by an AI agent while remaining challenging for frontier models.
  • Iterate on evaluation criteria until the task set is fair, robust, and reliable.
  • Submit completed tasks by the deadline and meet acceptance criteria.
  • Work independently on a project-based basis rather than in a permanent employment role.

Requirements

  • 5+ years of experience in software development.
  • Experience with the core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, and Redis.
  • Experience writing functional and integration tests.
  • English proficiency at B2 level or higher.
  • Submit a CV in English.
  • Ability to complete tasks estimated at around 20 hours each, with flexibility to work on your own schedule.
  • Understanding of how to evaluate code quality and identify where AI models fail on real-world coding tasks.

Benefits

  • Up to $30/hr equivalent compensation, depending on level and pace.
  • Project-based work with flexible scheduling.
  • Tasks estimated at approximately 20 hours each.
  • Opportunity to work on leading tech companies' AI evaluation projects.
  • Payment after completing tasks and passing acceptance criteria.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Helprise

Transformation Coach (AI & Agile)

Helprise

Helprise is seeking an experienced Transformation Coach (AI & Agile) to help a product-based organization evolve toward AI-driven ways of working by guiding teams, leaders, and Agile Release Trains through transformation and continuous improvement.

Poland Full-time Senior AI (Artificial Intelligence) Product Manager
Agile CI/CD Confluence JIRA
15 hours, 16 minutes ago
Keywords Studios

Expert Subtitle Translator/QCer English to Tagalog | Filipino

Keywords Studios 10K-50K Internet Software & Services

Keywords Studios is seeking freelance subtitling translators to adapt TV and movie content from English into Filipino/Tagalog for media and entertainment projects.

Philippines Contract Senior QA Engineer Translator
15 hours, 31 minutes ago
Hire Premium Sales & Support Global Team - CrewBloom

Configuration & Technical Setup Specialist

Hire Premium Sales & Support Global Team - CrewBloom 51-250 Professional Services

Configuration & Technical Setup Specialist at a configuration team responsible for preparing new members' systems, managing launch operations, and ensuring smooth, quality-controlled go-live support.

Philippines Mid Level Implementation Engineer QA Engineer
XML
15 hours, 31 minutes ago
The Quality Group

Werkstudent (gn) Produktkennzeichnung

The Quality Group Health Care Providers & Services

Werkstudent (gn) in der Produktkennzeichnung bei More Nutrition bzw. The Quality Group im remote Arbeitsumfeld in Deutschland, der die Pflege und Prüfung von Produktetikettierungen im Product Development Management Team unterstützt.

Germany Part-time Entry Level QA Engineer
15 hours, 31 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers