RemoteLeaf Logo
Log in Sign up

Freelance Agent Evaluation Engineer

1 day, 21 hours ago
Poland
Part-time
Senior
AI (Artificial Intelligence)
Software Development
Docker FastAPI GitHub Actions JavaScript Kafka Linux PostgreSQL Pytest Python React Redis REST API TypeScript
Apply Now
Mindrift.ai: Be the “I” in AI

Mindrift.ai: Be the “I” in AI

Join 10,000+ experts earning $15-50/hr training AI models remotely. Flexible freelance work, weekly payments. No AI experience required. Apply in 5 minutes.

Internet Software & Services
View All Jobs 30

Description

  • Build virtual company environments with codebase, infrastructure, development history, and supporting context such as conversations, documentation, and tickets.
  • Assemble tasks from intermediate states of a virtual company by crafting prompts, defining evaluation criteria, and ensuring the task is solvable and fair.
  • Design tasks in isolated developer workstation environments that include a Linux machine, development tools, MCP servers, and a real web application codebase.
  • Write tests that accept all correct solutions and reject incorrect ones without being too strict or too lenient.
  • Iterate on tests with an AI agent to verify they catch real issues and do not break valid solutions.
  • Review agent-written code, analyze agent failures or successes, and design edge cases and adversarial scenarios.
  • Incorporate feedback from expert QA reviewers and revise work to meet quality criteria.
  • Work collaboratively with AI throughout the task creation and evaluation process.

Requirements

  • Please submit your CV in English and indicate your level of English proficiency.
  • 5+ years of experience in software development, primarily Python.
  • Experience with Python frameworks and tools such as FastAPI, pytest, async/await, subprocess, and file operations.
  • Background in full-stack development, including React-based interfaces with JavaScript or TypeScript and robust back-end systems.
  • Experience writing functional and integration tests, not just executing them.
  • Experience working with Docker containers and infrastructure tools such as Postgres, Kafka, and Redis.
  • Understanding of CI/CD, especially GitHub Actions triggers, labels, and reading results.
  • Degree in Computer Science, Software Engineering, or a related field.
  • English proficiency at B2 level.
  • Comfort reading and reasoning about code across the stack.
  • Experience as a test automation specialist or similar is a plus.
  • Availability for part-time, non-permanent project-based work is preferred.

Benefits

  • Work on project-based AI opportunities for leading tech companies.
  • Flexible, non-permanent engagement with tasks completed on your own schedule.
  • Estimated workload of about 20 hours per project, depending on complexity.
  • Ability to choose when and how to work, with no fixed schedule requirement.
  • Compensation of up to $30 per hour equivalent, depending on contribution level and pace.
  • Opportunity to get paid after completing tasks that meet acceptance criteria.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Coupa Software

Principal Agentic Architect - 11385

Coupa Software 1K-5K Internet Software & Services

Coupa is hiring a remote Principal Agentic Architect in the UK to lead strategic AI-led engagements that transform procurement and finance operations from manual workflows to agentic orchestration.

United Kingdom Lead AI (Artificial Intelligence) Product Strategist
Generative AI LLM
37 minutes ago
Wing Assistant

AI Voice Trainer

Wing Assistant 51-250 Professional Services

Wing Data is hiring freelance voice data contributors for remote, project-based AI speech training work that uses short voice sessions to help improve client speech systems.

India Freelance Entry Level AI (Artificial Intelligence)
$4k-$8k
1 hour, 34 minutes ago
Intetics

1075| QA Engineer (AI / Agent-based Systems)

Intetics 1K-5K Internet Software & Services

Intetics Inc. is seeking a QA Engineer to join a small team building AI-driven and agent-based product features, with responsibility for validating both traditional product behavior and the reliability of LLM and agent workflows.

Poland Romania Ukraine Full-time Mid Level AI (Artificial Intelligence) QA Engineer
LLM Python
1 hour, 45 minutes ago
SWORD Health

QA Analyst

SWORD Health 251-1K Health Care Providers & Services

Sword Health is hiring a QA Analyst for its Web Development Team in Porto to help validate product quality and ensure web software is functional, reliable, and user-friendly.

Portugal Full-time Junior QA Engineer
$23k-$36k
Android Git iOS Jenkins JIRA Linux Postman
1 hour, 49 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers