RemoteLeaf Logo
Log in Sign up

Freelance Agent Evaluation Engineer

1 hour, 24 minutes ago
Canada
Part-time
Senior
AI (Artificial Intelligence)
Artificial Intelligence and Machine Learning
CI/CD Docker FastAPI GitHub Actions JavaScript Kafka Linux PostgreSQL Pytest Python React Redis TypeScript
Apply Now
Mindrift.ai: Be the “I” in AI

Mindrift.ai: Be the “I” in AI

Join 10,000+ experts earning $15-50/hr training AI models remotely. Flexible freelance work, weekly payments. No AI experience required. Apply in 5 minutes.

Internet Software & Services
View All Jobs 30

Description

  • Build virtual company environments that include a codebase, infrastructure, and supporting context such as conversations, documentation, and tickets.
  • Assemble tasks from intermediate states of a virtual company by crafting prompts, defining evaluation criteria, and ensuring solvability and fairness.
  • Design tasks in isolated developer-workstation environments with Linux tools, MCP servers, and a real web application codebase.
  • Write tests that accept all correct solutions and reject incorrect ones without being overly strict or too lenient.
  • Iterate on tests with an AI agent to verify they catch real issues, do not miss bad solutions, and do not break valid approaches.
  • Review agent-written code, analyze why the agent succeeded or failed, and create edge cases and adversarial scenarios.
  • Incorporate feedback from expert QA reviewers and revise work based on quality scoring.
  • Collaborate with AI throughout the task creation and evaluation process.
  • Complete project-based tasks within the stated deadline and acceptance criteria.

Requirements

  • Please submit your CV in English.
  • Indicate your level of English proficiency.
  • 5+ years of software development experience, primarily in Python.
  • Experience with FastAPI, pytest, async/await, subprocess, and file operations.
  • Background in full-stack development, including React, JavaScript, or TypeScript.
  • Experience building robust back-end systems.
  • Experience writing functional and integration tests, not just running tests.
  • Experience with Docker containers and infrastructure tools such as Postgres, Kafka, or Redis.
  • CI/CD understanding, including GitHub Actions triggers, labels, and reading results.
  • English proficiency at B2 level.
  • Degree in Computer Science, Software Engineering, or a related field is preferred.

Benefits

  • Project-based, non-permanent work arrangement.
  • Part-time opportunities with flexible scheduling.
  • Tasks estimated to take about 20 hours, depending on complexity.
  • Work on leading tech company AI projects.
  • Compensation of up to $45 per hour equivalent, depending on contribution level and pace.
  • Other projects on the platform may offer different earning levels depending on scope and expertise.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Toloka

Freelance Annotator (English) - AI Trainer

Toloka 251-1K Internet Software & Services

Toloka is seeking freelance annotators to support AI training projects by reviewing and labeling content that helps improve how artificial intelligence understands information.

France Part-time Junior AI (Artificial Intelligence) Data Annotator
Up to $46k
Generative AI
59 minutes ago
Wing Assistant

AI Video Trainer - Hebrew

Wing Assistant 51-250 Professional Services

Wing Data is hiring a freelance, project-based AI Audio & Video Trainer in Haifa to complete recorded participation tasks that help train next-generation AI systems.

Israel Freelance Entry Level AI (Artificial Intelligence)
$21k-$42k
1 hour, 2 minutes ago
SecurityScorecard

Senior/Principal Product Manager, AI

SecurityScorecard 251-1K IT Services

SecurityScorecard is hiring a Principal Product Manager to lead high-impact platform initiatives at the intersection of product, engineering, data, and AI, with responsibility for shaping strategy and delivery that reduce cyber risk for customers.

United States Full-time Lead AI (Artificial Intelligence) Product Manager
$175k-$285k
Cybersecurity System Design
1 hour, 7 minutes ago
mindsquare

Werkstudent Content Marketing & KI (m/w/d)

mindsquare 251-1K Internet Software & Services

mindsquare sucht eine*n Werkstudent*in im Marketing für Remote-Arbeit, der oder die Content- und digitale Marketingformate rund um Künstliche Intelligenz mitentwickelt.

Anywhere Internship Entry Level AI (Artificial Intelligence) Content Marketing Manager
Content Marketing
1 hour, 44 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers