RemoteLeaf Logo
Log in Sign up

Freelance Agent Evaluation Engineer

1 day, 21 hours ago
Philippines
Part-time
Senior
SDET (Software Development Engineer in Test)
Software Development
Docker FastAPI GitHub Actions JavaScript Kafka Linux PostgreSQL Pytest Python React Redis REST API TypeScript
Apply Now
Mindrift.ai: Be the “I” in AI

Mindrift.ai: Be the “I” in AI

Join 10,000+ experts earning $15-50/hr training AI models remotely. Flexible freelance work, weekly payments. No AI experience required. Apply in 5 minutes.

Internet Software & Services
View All Jobs 30

Description

  • Build virtual company environments with codebases, infrastructure, documentation, conversations, and ticket history that simulate realistic development contexts.
  • Assemble tasks from intermediate states of virtual companies by crafting prompts, defining evaluation criteria, and verifying that tasks are solvable and fairly assessed.
  • Design tasks in isolated workstation emulations that include a Linux machine, development tools, MCP servers, and a real web application codebase.
  • Write tests that accept all correct solutions and reject incorrect ones without being overly strict or overly lenient.
  • Iterate on tests with an AI agent to confirm they catch real failures and do not break valid solutions.
  • Review code produced by agents and analyze why a solution succeeded or failed.
  • Design edge cases and adversarial scenarios that reveal weaknesses in AI coding agents.
  • Refine tasks based on feedback from expert QA reviewers to meet quality criteria.

Requirements

  • Degree in Computer Science, Software Engineering, or a related field.
  • 5+ years of software development experience, primarily in Python.
  • Experience with FastAPI, pytest, async/await, subprocess, and file operations.
  • Background in full-stack development, including React-based interfaces using JavaScript or TypeScript.
  • Experience building robust back-end systems.
  • Experience writing functional and integration tests, not just executing them.
  • Experience working with Docker containers.
  • Familiarity with infrastructure tools such as Postgres, Kafka, and Redis.
  • Understanding of CI/CD, especially GitHub Actions triggers, labels, and result interpretation.
  • English proficiency at B2 level.
  • Comfort reading and reasoning about code across the stack.
  • Experience with test automation or similar evaluation-focused work is a plus.

Benefits

  • Project-based, non-permanent work with flexible participation.
  • Tasks estimated at about 20 hours, depending on complexity.
  • You choose when and how to work.
  • Opportunities to work with leading tech companies on AI-related projects.
  • Compensation of up to $12 per hour equivalent, depending on contribution level and pace.
  • Payment varies by project scope, complexity, and required expertise.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Pure Storage

Member of Technical Staff, FlashBlade

Pure Storage 1K-5K IT Services

EverPure is hiring a software engineer for the FlashBlade product team to help build a scale-out all-flash file and object storage system for modern data workloads.

India Full-time Lead Software Engineer
Apache Spark C C++ Elasticsearch Go Java Kafka Linux Python Splunk
18 minutes ago
K

Beam Planning Software Engineer

K2 Space Corporation 51-200 Defense and Space Manufacturing

K2 Space is hiring a software-focused satellite planning engineer to design simulation and production systems for large spacecraft constellations supporting communications and sensing missions.

United States Full-time Mid Level Software Engineer
$123k-$200k
C++ CI/CD Julia MATLAB NumPy Python Rust SciPy Statistics
52 minutes ago
Resilient Co

SDET

Resilient Co 11-50 Professional Services

Resilient Co. is hiring a remote Senior Software Development Engineer in Test (SDET) for an engineering contractor role focused on improving software quality through automated testing across the SDLC.

Colombia Costa Rica Peru Argentina Brazil Contract Senior SDET (Software Development Engineer in Test)
Angular Apache Spark AWS Azure C CI/CD Docker GCP JavaScript Kubernetes MongoDB MySQL .NET Oracle PostgreSQL Python R React Redis SQL SQL Server
56 minutes ago
Speechify

Senior Software Engineer, Core Experiences - Ljubljana, Slovenia

Speechify 51-250 Internet Software & Services

Speechify is hiring a Senior Software Engineer for its Core Experiences Team to build and maintain the foundational services and SDKs that power its cross-platform product experience.

Slovenia Full-time Senior Software Engineer
GCP Java Kotlin
1 hour, 16 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers