RemoteLeaf Logo
Log in Sign up

Freelance Agent Evaluation Engineer

12 hours, 24 minutes ago
Sweden, Belgium, Denmark, Finland, Ireland, Norway
Part-time
Senior
AI (Artificial Intelligence)
Software Development
Docker FastAPI JavaScript Kafka PostgreSQL Python React Redis TypeScript
Apply Now
Mindrift.ai: Be the “I” in AI

Mindrift.ai: Be the “I” in AI

Join 10,000+ experts earning $15-50/hr training AI models remotely. Flexible freelance work, weekly payments. No AI experience required. Apply in 5 minutes.

Internet Software & Services
View All Jobs 376

Description

  • Build realistic developer environments that include a virtual company, codebase, infrastructure, and supporting context such as tickets, documentation, and conversations.
  • Design tasks from intermediate states of those environments, including the prompt and the definition of what counts as a solved task.
  • Write tests that verify agent solutions by accepting valid approaches and rejecting incorrect ones.
  • Iterate on tasks and tests based on QA feedback, reviewing agent outputs and refining the evaluation.
  • Analyze agent failures to improve task quality and make evaluations fair and robust.
  • Ensure tasks are solvable by an AI agent while still being challenging for advanced models.
  • Evaluate solutions across multiple valid approaches rather than forcing a single implementation path.

Requirements

  • 5+ years of experience in software development.
  • Experience with Python, specifically FastAPI.
  • Experience with JavaScript or TypeScript, specifically React.
  • Experience with Docker, Postgres, Kafka, and Redis.
  • Experience writing functional and integration tests.
  • English proficiency at B2 level or higher.
  • Submit a CV in English and indicate your English proficiency level.
  • Understanding of how to assess model weaknesses in coding tasks is preferred.

Benefits

  • Up to $50/hr equivalent compensation, depending on level and pace.
  • Project-based work with no permanent employment commitment.
  • Flexible schedule: choose when and how to work.
  • Tasks estimated at about 20 hours each, with workload depending on complexity.
  • Payment after completing tasks and passing acceptance criteria.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Leading Educators

Director of AI-Enabled Instruction, ELA

Leading Educators 51-250 Diversified Consumer Services

Leading Educators is hiring a Director of AI-Enabled Instruction, ELA to lead a multi-school literacy initiative and advance its broader AI-enabled ELA strategy through curriculum-aligned instructional design, coaching, and systems support.

United States Full-time Lead AI (Artificial Intelligence) Instructional Designer
$104k-$140k
Prototyping
11 hours, 54 minutes ago
Remote Recruitment

AI Integration Specialist - Italian Speaking

Remote Recruitment 11-50 Professional Services

AI Integration Specialist role at an Italian dental business coaching and consulting organisation, focused on implementing AI and automation solutions that improve efficiency, data use, and business performance across the company.

South Africa Italy Full-time Mid Level AI (Artificial Intelligence) AI Engineer
LLM Machine Learning Python SQL Tableau
12 hours, 24 minutes ago
Toloka

AI Trainer - Freelance Data Annotator

Toloka 251-1K Internet Software & Services

Toloka is hiring freelance annotators to support Generative AI projects by reviewing and labeling online content for accuracy and quality when project work is available.

Norway Belgium Sweden Denmark Finland Ireland Part-time Junior AI (Artificial Intelligence) Data Annotator
Up to $46k
Generative AI
12 hours, 24 minutes ago
Toloka

Creador(a) de Contenido POV para IA - Remoto

Toloka 251-1K Internet Software & Services

Proyecto remoto y flexible con una empresa de entrenamiento de IA que paga por grabaciones de actividades cotidianas realizadas desde casa para ayudar a entrenar futuros sistemas de IA y robótica.

United States Part-time Entry Level AI (Artificial Intelligence)
$21k-$21k
12 hours, 24 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers