RemoteLeaf Logo
Log in Sign up

Freelance Agent Evaluation Engineer

5 hours, 11 minutes ago
Qatar, Saudi Arabia, Turkey, Israel, Jordan, Kuwait
Part-time
Senior
SDET (Software Development Engineer in Test)
Software Development
Docker FastAPI GitHub Actions JavaScript Kafka Linux PostgreSQL Pytest Python React Redis REST API TypeScript
Apply Now
Mindrift.ai: Be the “I” in AI

Mindrift.ai: Be the “I” in AI

Join 10,000+ experts earning $15-50/hr training AI models remotely. Flexible freelance work, weekly payments. No AI experience required. Apply in 5 minutes.

Internet Software & Services
View All Jobs 57

Description

  • Build virtual company environments with codebases, infrastructure, and development context that simulate realistic engineering work.
  • Assemble tasks from intermediate states of the virtual company, including prompts, evaluation criteria, and solvable scenarios.
  • Design isolated workstation-based tasks using Linux development tools, MCP servers, and real web application codebases.
  • Write tests that correctly accept valid solutions and reject incorrect ones without being too strict or too lenient.
  • Iterate on tests with an AI agent to verify they catch real issues and do not break correct approaches.
  • Review agent-written code to determine why solutions succeed or fail.
  • Design edge cases and adversarial scenarios that expose differences between strong and weak solutions.
  • Incorporate feedback from expert QA reviewers and improve task quality accordingly.

Requirements

  • Please submit your CV in English and indicate your level of English proficiency.
  • Degree in Computer Science, Software Engineering, or a related field.
  • 5+ years of software development experience, primarily with Python.
  • Experience with FastAPI, pytest, async/await, subprocess, and file operations.
  • Background in full-stack development, including React-based interfaces with JavaScript or TypeScript and robust back-end systems.
  • Experience writing functional and integration tests, not just running them.
  • Experience with Docker containers and infrastructure tools such as Postgres, Kafka, and Redis.
  • Understanding of CI/CD, including GitHub Actions triggers, labels, and reading results.
  • English proficiency at B2 level.
  • Comfort reading and reasoning about code across the stack, even if not expert in every area.

Benefits

  • Project-based work with leading tech companies through Mindrift.
  • Part-time, non-permanent engagement with flexible participation.
  • Estimated 20 hours of work per project, with scheduling at your discretion.
  • Tasks must be completed by the deadline, but you choose when and how to work.
  • Compensation of up to $40 per hour equivalent, depending on level and pace of contribution.
  • Opportunity to work alongside AI on challenging technical evaluation tasks.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Airbnb

Senior Software Engineer, Cloud Infrastructure AI

Airbnb 5K-10K Hotels, Restaurants & Leisure

Airbnb is hiring a Senior Software Engineer to lead the development and operation of its Kubernetes-based GPU platform powering AI and machine learning infrastructure.

United States Full-time Senior Software Engineer
$191k-$225k
AWS Azure EC2 GCP Kubernetes LLM Machine Learning
11 minutes ago
Mozilla

Senior Manager, Firefox Mobile Test Engineering

Mozilla 251-1K Internet Software & Services

Mozilla is hiring a senior Test Engineering leader to set and own Firefox Mobile’s quality strategy and manage the team responsible for test automation, observability, and exploratory testing across Android and iOS.

Spain Full-time Lead Mobile Engineer SDET (Software Development Engineer in Test)
$136k-$181k
Android Espresso iOS
1 hour, 18 minutes ago
Promenade

Sr. Software Engineer, Internal Apps

Promenade 51-250 Internet Software & Services

Promenade is hiring a Senior Software Engineer for its Internal Tools team to build and improve software that helps internal teams work more efficiently and reliably.

Anywhere Full-time Senior Software Engineer
$135k-$150k
JavaScript MySQL Python Snowflake SQL System Design
1 hour, 32 minutes ago
Amwell

Sr Software Engineer

Amwell 1K-5K Diversified Telecommunication Services

Amwell is hiring a Senior Software Engineer – Full Stack to help build and support its cloud-based healthcare platform that connects patients and providers across the care continuum.

Colombia Full-time Senior Software Engineer
Angular AWS DynamoDB Java JavaScript Microservices MongoDB NestJS Node.js PostgreSQL Redis Spring Boot TypeScript
1 hour, 40 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers