RemoteLeaf Logo
Log in Sign up

Freelance Agent Evaluation Engineer

5 hours, 10 minutes ago
Germany
Part-time
Senior
SDET (Software Development Engineer in Test)
Software Development
Docker FastAPI GitHub Actions JavaScript Kafka Linux PostgreSQL Pytest Python React Redis TypeScript
Apply Now
Mindrift.ai: Be the “I” in AI

Mindrift.ai: Be the “I” in AI

Join 10,000+ experts earning $15-50/hr training AI models remotely. Flexible freelance work, weekly payments. No AI experience required. Apply in 5 minutes.

Internet Software & Services
View All Jobs 57

Description

  • Build virtual company environments with codebases, infrastructure, and realistic context such as conversations, documentation, and tickets.
  • Assemble tasks from intermediate states of the virtual company by crafting prompts and defining fair evaluation criteria.
  • Design tasks in isolated workstation-style environments that include developer tools, MCP servers, and a real web application codebase.
  • Write tests that correctly accept valid solutions and reject incorrect ones without being overly strict or overly lenient.
  • Iterate on tests with an AI agent to verify they catch real failures and do not break good solutions.
  • Review code produced by agents and analyze why tasks succeeded or failed.
  • Design edge cases and adversarial scenarios that reveal weaknesses in model-generated solutions.
  • Revise work based on feedback from expert QA reviewers to meet quality criteria.
  • Submit completed tasks by the deadline and ensure they meet acceptance criteria.

Requirements

  • Please submit your CV in English and indicate your level of English proficiency.
  • Degree in Computer Science, Software Engineering, or a related field.
  • 5+ years of software development experience, primarily in Python.
  • Experience with FastAPI, pytest, async/await, subprocess, and file operations.
  • Background in full-stack development with React, JavaScript/TypeScript, and robust back-end systems.
  • Experience writing functional and integration tests, not just running them.
  • Experience with Docker containers and infrastructure tools such as Postgres, Kafka, and Redis.
  • Understanding of CI/CD, especially using GitHub Actions to read triggers, labels, and results.
  • English proficiency at B2 level.
  • Comfort reading and reasoning about code across the stack; expertise in every listed item is not required.

Benefits

  • Project-based, non-permanent work with Mindrift.
  • Part-time flexibility with control over when and how you work.
  • Estimated 20 hours of task work per project, depending on complexity.
  • Up to $50 per hour equivalent compensation, depending on contribution level and pace.
  • Opportunities to work on leading tech-company AI projects.
  • Pay varies by project scope, complexity, and required expertise.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Airbnb

Senior Software Engineer, Cloud Infrastructure AI

Airbnb 5K-10K Hotels, Restaurants & Leisure

Airbnb is hiring a Senior Software Engineer to lead the development and operation of its Kubernetes-based GPU platform powering AI and machine learning infrastructure.

United States Full-time Senior Software Engineer
$191k-$225k
AWS Azure EC2 GCP Kubernetes LLM Machine Learning
10 minutes ago
Mozilla

Senior Manager, Firefox Mobile Test Engineering

Mozilla 251-1K Internet Software & Services

Mozilla is hiring a senior Test Engineering leader to set and own Firefox Mobile’s quality strategy and manage the team responsible for test automation, observability, and exploratory testing across Android and iOS.

Spain Full-time Lead Mobile Engineer SDET (Software Development Engineer in Test)
$136k-$181k
Android Espresso iOS
1 hour, 17 minutes ago
Promenade

Sr. Software Engineer, Internal Apps

Promenade 51-250 Internet Software & Services

Promenade is hiring a Senior Software Engineer for its Internal Tools team to build and improve software that helps internal teams work more efficiently and reliably.

Anywhere Full-time Senior Software Engineer
$135k-$150k
JavaScript MySQL Python Snowflake SQL System Design
1 hour, 31 minutes ago
Amwell

Sr Software Engineer

Amwell 1K-5K Diversified Telecommunication Services

Amwell is hiring a Senior Software Engineer – Full Stack to help build and support its cloud-based healthcare platform that connects patients and providers across the care continuum.

Colombia Full-time Senior Software Engineer
Angular AWS DynamoDB Java JavaScript Microservices MongoDB NestJS Node.js PostgreSQL Redis Spring Boot TypeScript
1 hour, 39 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers