Senior AI Agent Evaluation Engineer

2 months, 4 weeks ago
Part-time
Senior
Artificial Intelligence and Machine Learning
Mindrift.ai: Be the “I” in AI

Mindrift.ai: Be the “I” in AI

Join 10,000+ experts earning $15-50/hr training AI models remotely. Flexible freelance work, weekly payments. No AI experience required. Apply in 5 minutes.

Internet Software & Services

Description

  • Build realistic virtual developer environments with codebases, infrastructure, and supporting context such as tickets, documentation, and conversations.
  • Design tasks from intermediate states of the simulated environments and define what counts as a solved solution.
  • Create prompts that are challenging but solvable by an AI agent.
  • Write tests that validate agent solutions, accepting all valid approaches and rejecting incorrect ones.
  • Review agent solutions, analyze failures, and iterate on tasks and tests based on QA feedback.
  • Refine evaluation criteria until the task design is fair and robust.
  • Identify scenarios that reveal the strengths and weaknesses of frontier coding models.
  • Submit completed tasks by the deadline and meet the required acceptance criteria.

Requirements

  • 5+ years of software development experience.
  • Experience with Python and FastAPI.
  • Experience with JavaScript or TypeScript and React.
  • Experience with Docker, Postgres, Kafka, and Redis.
  • Experience writing functional and integration tests.
  • English proficiency at B2+ level.
  • Ability to work on a project-based, non-permanent basis.
  • Willingness to complete qualification steps before joining a project.

Benefits

  • Up to $50/hr equivalent compensation, depending on level and pace.
  • Flexible schedule with the ability to choose when and how to work.
  • Project-based work with estimated tasks of around 20 hours each.
  • Payment for completed tasks upon acceptance.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

QA Engineer II

Tactacam 51-250 Household Durables

Tactacam is hiring a QA Engineer to ensure the quality of its outdoor and action camera products by building thorough test coverage and supporting quality throughout the software development lifecycle.

JIRA
14 hours, 14 minutes ago

Director, Advisory - Anthropic & FinOps

phData 251-1K IT Services

phData is hiring a remote-first Consulting Director to lead its Anthropic-focused advisory practice, helping enterprise customers develop AI strategy, governance, organizational change, and FinOps approaches while growing the company’s related services business.

14 hours, 14 minutes ago

Principal AI/ML Engineering Lead

Vida Health 251-1K Health Care Providers & Services

Vida is hiring a senior AI/ML platform engineer to help rebuild its secure healthcare data platform and enable reliable, privacy-preserving machine learning capabilities.

AWS Azure CI/CD dbt Docker Feature Engineering GCP GraphQL HIPAA Kubernetes LLM Machine Learning MLOps Python REST API Secrets Management SQL Terraform Vertex AI
14 hours, 29 minutes ago

Strategic Operations Manager, AI-Enabled Systems (AI-Fluent) Consultant

Right Side Up 51-250 Professional Services

Right Side Up is seeking freelance operations-minded builders for its talent collective to help client teams replace manual workflows and ad hoc tools with scalable processes, AI-enabled systems, and internal applications.

Supabase Vercel
14 hours, 44 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers