RemoteLeaf Logo
Log in Sign up

Freelance Agent Evaluation Engineer

1 hour, 41 minutes ago
Portugal
Part-time
Senior
AI (Artificial Intelligence)
Software Development
CI/CD Docker FastAPI GitHub Actions JavaScript Kafka Linux PostgreSQL Pytest Python React Redis REST API TypeScript
Apply Now
Mindrift.ai: Be the “I” in AI

Mindrift.ai: Be the “I” in AI

Join 10,000+ experts earning $15-50/hr training AI models remotely. Flexible freelance work, weekly payments. No AI experience required. Apply in 5 minutes.

Internet Software & Services
View All Jobs 60

Description

  • Build virtual company environments with codebases, infrastructure, and supporting context such as conversations, documentation, and tickets.
  • Assemble tasks from intermediate states of a virtual company by crafting prompts, defining evaluation criteria, and ensuring fairness and solvability.
  • Design tasks in isolated developer-workstation environments using Linux tools, MCP servers, and a real web application codebase.
  • Write tests that correctly accept valid solutions and reject incorrect ones without being overly strict or overly lenient.
  • Iterate on tests with an AI agent to verify they catch real issues and do not fail on correct approaches.
  • Review agent-written code to understand why solutions succeed or fail.
  • Design edge cases and adversarial scenarios that expose model weaknesses.
  • Incorporate feedback from expert QA reviewers and refine work based on quality criteria.
  • Submit completed tasks that meet acceptance criteria by the deadline.

Requirements

  • Please submit your CV in English and indicate your level of English proficiency.
  • 5+ years of experience in software development, primarily Python.
  • Experience with FastAPI, pytest, async/await, subprocess, and file operations.
  • Background in full-stack development, including React-based interfaces and JavaScript/TypeScript.
  • Experience writing functional and integration tests, not just running them.
  • Experience with Docker containers and infrastructure tools such as Postgres, Kafka, and Redis.
  • Working familiarity with CI/CD, especially GitHub Actions as a user for triggers, labels, and reading results.
  • Degree in Computer Science, Software Engineering, or a related field.
  • English proficiency at B2 level.
  • Comfort reading and reasoning about code across the stack.
  • Open to part-time, non-permanent project-based work.
  • Preferred experience as a software engineer or test automation specialist.

Benefits

  • Project-based work with leading tech companies through Mindrift.
  • Flexible, non-permanent participation rather than full-time employment.
  • Tasks estimated to take about 20 hours, depending on complexity.
  • You choose when and how to work.
  • Compensation of up to $30 per hour equivalent, depending on contribution level and pace.
  • Payment for completed tasks that meet acceptance criteria.
  • Opportunity to work on technically challenging AI evaluation projects.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Welo Global

Generative AI Analyst (Chinese zn-CH)

Welo Global Professional Services

Welo Data is hiring a remote Generative AI Analyst to evaluate and quality-check AI-generated videos against written prompts, ensuring accurate annotations and consistent review outcomes.

United States Contract Junior AI (Artificial Intelligence)
$22k-$22k
Generative AI LLM
1 minute ago
Bluelight Consulting

Agentic AI Solutions Architect - Remote, Latin America

Bluelight Consulting 11-50 Internet Software & Services

Bluelight is hiring a remote Agentic AI Solutions Architect in Latin America to lead client-facing AI strategy and design end-to-end agentic workflows that move from discovery through production deployment.

Latin America Argentina Full-time Senior AI (Artificial Intelligence) Solutions Architect
Apache Airflow AWS Azure GCP Generative AI LLM MLOps Prefect Python Snowflake SQL
1 minute ago
Dijital Team

Automation Engineer (RPA/RMM)

Dijital Team 11-50 Internet Software & Services

Automation Engineer (RPA/RMM) at a technology services company in Colombo, working remotely to design and improve automation solutions that reduce manual work and strengthen service delivery.

Sri Lanka Full-time Senior SDET (Software Development Engineer in Test)
Azure
1 minute ago
Inter

Software Developer Specialist – AI Agents & Generative AI

Inter 51-250 Banks

Inter is seeking a software engineer to build and evolve its generative AI agent platform for Seven, the company’s virtual assistant, as part of its AI-first transformation.

Brazil Full-time Mid Level AI (Artificial Intelligence)
AWS Docker FastAPI Grafana Java Kanban Kubernetes Mobile Development MongoDB .NET New Relic PostgreSQL Python Redis REST API Scrum
16 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers