Senior AI Agent Evaluation Engineer

3 months, 1 week ago
Part-time
Senior
Artificial Intelligence and Machine Learning
Mindrift.ai: Be the “I” in AI

Mindrift.ai: Be the “I” in AI

Join 10,000+ experts earning $15-50/hr training AI models remotely. Flexible freelance work, weekly payments. No AI experience required. Apply in 5 minutes.

Internet Software & Services

Description

  • Build realistic virtual developer environments with codebases, infrastructure, and supporting context such as tickets, documentation, and conversations.
  • Design tasks from intermediate states of the simulated environments and define what counts as a solved solution.
  • Create prompts that are challenging but solvable by an AI agent.
  • Write tests that validate agent solutions, accepting all valid approaches and rejecting incorrect ones.
  • Review agent solutions, analyze failures, and iterate on tasks and tests based on QA feedback.
  • Refine evaluation criteria until the task design is fair and robust.
  • Identify scenarios that reveal the strengths and weaknesses of frontier coding models.
  • Submit completed tasks by the deadline and meet the required acceptance criteria.

Requirements

  • 5+ years of software development experience.
  • Experience with Python and FastAPI.
  • Experience with JavaScript or TypeScript and React.
  • Experience with Docker, Postgres, Kafka, and Redis.
  • Experience writing functional and integration tests.
  • English proficiency at B2+ level.
  • Ability to work on a project-based, non-permanent basis.
  • Willingness to complete qualification steps before joining a project.

Benefits

  • Up to $50/hr equivalent compensation, depending on level and pace.
  • Flexible schedule with the ability to choose when and how to work.
  • Project-based work with estimated tasks of around 20 hours each.
  • Payment for completed tasks upon acceptance.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

SMB AI Power User — Competitive Evaluations

Meridial Marketplace, by Invisible 501-1000 information technology & services

Meridial is seeking a remote independent-contractor SMB AI Power User to evaluate a major consumer technology company’s bundled AI assistant against paid competitors using real small-business operations and marketing workflows.

E-commerce Generative AI Instagram API Shopify Social Media Marketing
1 hour, 33 minutes ago

AI Response Labeler / Annotator

Blueprint Technologies 251-1K Internet Software & Services

Blueprint, a technology solutions firm, is hiring a remote English-language AI Response Labeler/Annotator in Latin America to evaluate and compare AI-generated content for accuracy, relevance, safety, and overall usefulness.

2 hours, 3 minutes ago

Director of AI

Ludia Consulting 11-50 Professional Services

Ludia Consulting is seeking a Director of AI to lead its AI engineering team, governance, strategy, customer implementations, and Ludia Labs accelerator portfolio across Dynamics 365 and the Microsoft AI ecosystem.

Azure C# CI/CD Generative AI Git GitHub Microsoft Dynamics 365 .NET Python TypeScript
2 hours, 3 minutes ago

AI Generalist Intern

HexCoded is hiring a remote, full-time product-focused AI video studio intern for a six-month role, working directly with the small Gurugram-based team to improve Creative Studio and ship user-driven fixes.

2 hours, 18 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers