RemoteLeaf Logo
Log in Sign up

Freelance Agent Evaluation Engineer

1 day, 5 hours ago
Poland
Part-time
Senior
AI (Artificial Intelligence)
Software Development
Docker FastAPI JavaScript Kafka PostgreSQL Python React Redis TypeScript
Apply Now
Mindrift.ai: Be the “I” in AI

Mindrift.ai: Be the “I” in AI

Join 10,000+ experts earning $15-50/hr training AI models remotely. Flexible freelance work, weekly payments. No AI experience required. Apply in 5 minutes.

Internet Software & Services
View All Jobs 376

Description

  • Build realistic virtual developer environments with codebases, infrastructure, and supporting context such as tickets, documentation, and conversations.
  • Design tasks from intermediate environment states, including crafting prompts and defining what counts as a solved solution.
  • Ensure each task is solvable by an AI agent and reflects realistic development work.
  • Write tests that validate agent solutions, accepting all correct approaches and rejecting incorrect ones.
  • Review agent outputs, analyze failure cases, and refine tasks and tests based on QA feedback.
  • Iterate on evaluation criteria to make assessments fair, robust, and challenging for frontier models.
  • Work within project-based tasks rather than permanent employment.
  • Submit completed tasks by the deadline and meet acceptance criteria.

Requirements

  • 5+ years of software development experience.
  • Experience with the core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, and Redis.
  • Experience writing functional and integration tests.
  • English proficiency at B2 level or higher.
  • Ability to work on project-based assignments and complete tasks independently.
  • Willingness to submit a CV in English and indicate English proficiency level.

Benefits

  • Up to $40/hr equivalent compensation, depending on level and pace.
  • Flexible schedule with the ability to choose when and how to work.
  • Project-based work with estimated tasks of about 20 hours each.
  • Payment for completed tasks after project participation and acceptance.
  • Opportunity to work on cutting-edge AI evaluation projects for leading tech companies.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Volga Partners

Contract Compliance Subject Matter Expert | AI Evaluation Specialist (On-Call)

Volga Partners 51-250 Internet Software & Services

Volga Partners is hiring remote on-call AI Evaluation Specialists in finance, audit, insurance, compliance, and legal domains to review and improve AI-generated outputs for ongoing model training and evaluation projects.

India Pakistan Contract Mid Level AI (Artificial Intelligence) Compliance Manager
$0k-$0k
Machine Learning
5 hours ago
Volga Partners

Claims Auditing Subject Matter Expert | AI Evaluation Specialist(On-Call) Remote

Volga Partners 51-250 Internet Software & Services

Volga Partners is hiring remote claims auditing professionals in India or Pakistan for a temporary contract supporting the development and evaluation of AI models for finance and insurance reasoning.

India Pakistan Contract Mid Level AI (Artificial Intelligence)
$0k-$0k
Machine Learning Statistics
5 hours ago
Volga Partners

Forensic Auditing Subject Matter Expert | AI Evaluation Specialist (On-Call)

Volga Partners 51-250 Internet Software & Services

Volga Partners is hiring remote claims auditing and finance professionals in India/Pakistan for a temporary contract supporting the development and evaluation of AI models for financial and insurance reasoning.

India Pakistan Contract Mid Level AI (Artificial Intelligence)
$0k-$0k
Machine Learning
5 hours ago
Volga Partners

Insurance Risk Subject Matter Expert | AI Evaluation Specialist (On-Call) Remote

Volga Partners 51-250 Internet Software & Services

Volga Partners is hiring remote claims auditing and finance subject matter experts in India or Pakistan to help develop, train, and evaluate AI models for financial and insurance reasoning.

India Pakistan Contract Mid Level AI (Artificial Intelligence)
$0k-$0k
Machine Learning Statistics
5 hours ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers