RemoteLeaf Logo
Log in Sign up

Freelance Agent Evaluation Engineer

6 hours, 31 minutes ago
Brazil
Part-time
Senior
QA Engineer
Software Development
Docker FastAPI GitHub Actions JavaScript Kafka Linux PostgreSQL Pytest Python React Redis TypeScript
Apply Now
Mindrift.ai: Be the “I” in AI

Mindrift.ai: Be the “I” in AI

Join 10,000+ experts earning $15-50/hr training AI models remotely. Flexible freelance work, weekly payments. No AI experience required. Apply in 5 minutes.

Internet Software & Services
View All Jobs 30

Description

  • Build virtual company environments with codebases, infrastructure, documentation, tickets, and development history.
  • Assemble tasks from intermediate states of a virtual company by crafting prompts, defining evaluation criteria, and ensuring solvability and fairness.
  • Design tasks in isolated developer-workstation environments using Linux, terminal tools, MCP servers, and real web application codebases.
  • Write tests that accept all valid solutions and reject incorrect ones without being too strict or too lenient.
  • Iterate on tests with an AI agent to verify they catch real issues and do not break valid approaches.
  • Review agent-written code, analyze successes and failures, and design edge cases and adversarial scenarios.
  • Incorporate feedback from expert QA reviewers and revise work to meet quality criteria.
  • Collaborate with AI tools throughout the task creation and evaluation process.

Requirements

  • Degree in Computer Science, Software Engineering, or a related field.
  • 5+ years of software development experience, primarily in Python.
  • Experience with FastAPI, pytest, async/await, subprocesses, and file operations.
  • Background in full-stack development, including React-based interfaces and JavaScript/TypeScript.
  • Experience building robust back-end systems.
  • Experience writing functional and integration tests, not just executing them.
  • Experience with Docker containers and infrastructure tools such as Postgres, Kafka, and Redis.
  • Understanding of CI/CD, especially GitHub Actions triggers, labels, and result interpretation.
  • English proficiency at B2 level.
  • Comfort reading and reasoning about code across the stack; expertise in every listed area is not required.

Benefits

  • Project-based, non-permanent work with Mindrift.
  • Part-time work with flexible timing; you choose when and how to work.
  • Tasks are estimated at about 20 hours, depending on complexity.
  • Potential compensation of up to $17 per hour equivalent, depending on contribution level and pace.
  • Work on leading tech-company AI evaluation projects.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Pure Storage

Member of Technical Staff, FlashBlade

Pure Storage 1K-5K IT Services

EverPure is hiring a software engineer for the FlashBlade product team to help build a scale-out all-flash file and object storage system for modern data workloads.

India Full-time Lead Software Engineer
Apache Spark C C++ Elasticsearch Go Java Kafka Linux Python Splunk
19 minutes ago
K

Beam Planning Software Engineer

K2 Space Corporation 51-200 Defense and Space Manufacturing

K2 Space is hiring a software-focused satellite planning engineer to design simulation and production systems for large spacecraft constellations supporting communications and sensing missions.

United States Full-time Mid Level Software Engineer
$123k-$200k
C++ CI/CD Julia MATLAB NumPy Python Rust SciPy Statistics
53 minutes ago
Speechify

Senior Software Engineer, Core Experiences - Ljubljana, Slovenia

Speechify 51-250 Internet Software & Services

Speechify is hiring a Senior Software Engineer for its Core Experiences Team to build and maintain the foundational services and SDKs that power its cross-platform product experience.

Slovenia Full-time Senior Software Engineer
GCP Java Kotlin
1 hour, 16 minutes ago
Intetics

1075| QA Engineer (AI / Agent-based Systems)

Intetics 1K-5K Internet Software & Services

Intetics Inc. is seeking a QA Engineer to join a small team building AI-driven and agent-based product features, with responsibility for validating both traditional product behavior and the reliability of LLM and agent workflows.

Poland Romania Ukraine Full-time Mid Level AI (Artificial Intelligence) QA Engineer
LLM Python
1 hour, 45 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers