RemoteLeaf Logo
Log in Sign up

Freelance Agent Evaluation Engineer

1 hour, 40 minutes ago
South Africa
Part-time
Senior
AI (Artificial Intelligence)
Software Development
Docker FastAPI JavaScript Kafka PostgreSQL Python React Redis TypeScript
Apply Now
Mindrift.ai: Be the “I” in AI

Mindrift.ai: Be the “I” in AI

Join 10,000+ experts earning $15-50/hr training AI models remotely. Flexible freelance work, weekly payments. No AI experience required. Apply in 5 minutes.

Internet Software & Services
View All Jobs 324

Description

  • Build realistic virtual developer environments with codebases, infrastructure, and supporting context such as tickets, documentation, and conversations.
  • Design coding tasks from intermediate states of those environments and define clear success criteria.
  • Write tests that verify AI agent solutions and accurately accept valid approaches while rejecting incorrect ones.
  • Review agent solutions, analyze failures, and refine tasks and tests based on QA feedback.
  • Iterate on evaluation criteria to ensure tasks are fair, robust, and solvable by AI agents.
  • Assess where AI models fail and create scenarios that meaningfully distinguish strong solutions from weak ones.
  • Submit completed tasks by the deadline while meeting the required acceptance criteria.

Requirements

  • 5+ years of experience in software development.
  • Experience with Python, including FastAPI.
  • Experience with JavaScript/TypeScript, including React.
  • Experience working with Docker, Postgres, Kafka, and Redis.
  • Experience writing functional and integration tests.
  • English proficiency at B2 level or higher.
  • Submit a CV in English and indicate your English proficiency level.
  • Ability to work on a project-based, non-permanent engagement.
  • Estimated capacity to complete tasks that take around 20 hours each, with flexible scheduling.

Benefits

  • Up to $40/hr equivalent compensation, depending on level and pace.
  • Flexible schedule with the ability to choose when and how to work.
  • Project-based work rather than permanent employment.
  • Estimated 20-hour tasks with scheduling flexibility.
  • Payment upon completing tasks that meet acceptance criteria.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

LightFeather

Senior ServiceNow Developer - Public Trust Clearance

LightFeather 51-250 Internet Software & Services

LightFeather is hiring a Senior ServiceNow Developer to design, build, and deploy secure, scalable ServiceNow solutions for federal government projects in a fully remote, full-time role.

United States Full-time Lead Software Engineer
$90k-$115k
Agile Angular AWS Azure CI/CD CSS GCP HTML JavaScript JIRA Kanban REST API Scrum
1 hour, 25 minutes ago
Prolific

AI Trainer - Kannada - Kolkata, India

Prolific 51-250 Professional Services

Prolific is hiring a remote, task-based AI Trainer for Kannada to evaluate text and voice data so AI models better understand natural Kannada speech and nuance.

India Contract Entry Level AI (Artificial Intelligence)
$7k-$7k
1 hour, 25 minutes ago
Prolific

AI Trainer - Kannada - San Diego, US

Prolific 51-250 Professional Services

Prolific is hiring a remote, task-based AI Trainer for Kannada to evaluate audio and text snippets so its AI models better understand the nuance and authenticity of the Kannada language.

United States Contract Entry Level AI (Artificial Intelligence)
$7k-$7k
1 hour, 25 minutes ago
Prolific

AI Trainer - Kannada - Glasgow, UK

Prolific 51-250 Professional Services

Prolific is hiring remote task-based AI Trainers for Kannada to evaluate text and audio snippets and help improve how AI understands the nuance and naturalness of the Kannada language.

United Kingdom Contract Entry Level AI (Artificial Intelligence)
$7k-$7k
1 hour, 40 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers