RemoteLeaf Logo
Log in Sign up

Freelance Agent Evaluation Engineer

4 hours, 19 minutes ago
Sweden, Belgium, Denmark, Finland, Ireland, Norway
Part-time
Senior
SDET (Software Development Engineer in Test)
Software Development
Docker FastAPI GitHub Actions JavaScript Kafka Linux PostgreSQL Pytest Python React Redis REST API TypeScript
Apply Now
Mindrift.ai: Be the “I” in AI

Mindrift.ai: Be the “I” in AI

Join 10,000+ experts earning $15-50/hr training AI models remotely. Flexible freelance work, weekly payments. No AI experience required. Apply in 5 minutes.

Internet Software & Services
View All Jobs 57

Description

  • Build virtual companies with codebases, infrastructure, and supporting context such as conversations, documentation, and tickets.
  • Assemble and calibrate tasks from intermediate states of a virtual company by crafting prompts and defining fair, solvable evaluation criteria.
  • Design tasks in isolated developer-workstation environments, including Linux machines, terminal and CLI tools, MCP servers, and real web application codebases.
  • Write tests that accept all correct solutions and reject incorrect ones without being too strict or too lenient.
  • Iterate on tests with an AI agent to verify that they catch real issues and do not break valid solutions.
  • Review code written by agents, analyze success and failure cases, and design edge cases and adversarial scenarios.
  • Iterate based on feedback from expert QA reviewers and improve work quality accordingly.
  • Collaborate with AI throughout the task creation and evaluation process.
  • Submit completed tasks by the deadline and meet acceptance criteria.
  • Work on project-based assignments rather than permanent employment.

Requirements

  • Please submit your CV in English and indicate your level of English proficiency.
  • Degree in Computer Science, Software Engineering, or a related field.
  • 5+ years of software development experience, primarily with Python.
  • Experience with FastAPI, pytest, async/await, subprocess, and file operations.
  • Background in full-stack development, including React-based interfaces and robust back-end systems.
  • Experience writing functional and integration tests, not just running them.
  • Experience with Docker containers and familiarity with infrastructure tools such as Postgres, Kafka, and Redis.
  • Understanding of CI/CD, especially GitHub Actions triggers, labels, and result interpretation.
  • English proficiency at B2 level.
  • Comfort reading and reasoning across the full stack, even without being an expert in every area.

Benefits

  • Project-based, non-permanent work with flexibility in when and how you work.
  • Tasks are estimated to take about 20 hours total, depending on complexity.
  • Compensation of up to $50 per hour equivalent, depending on contribution level and pace.
  • Opportunity to work on challenging AI evaluation projects for leading tech companies.
  • Paid project work after passing qualification(s) and joining a project.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Promenade

Sr. Software Engineer, Internal Apps

Promenade 51-250 Internet Software & Services

Promenade is hiring a Senior Software Engineer for its Internal Tools team to build and improve software that helps internal teams work more efficiently and reliably.

Anywhere Full-time Senior Software Engineer
$135k-$150k
JavaScript MySQL Python Snowflake SQL System Design
41 minutes ago
Amwell

Sr Software Engineer

Amwell 1K-5K Diversified Telecommunication Services

Amwell is hiring a Senior Software Engineer – Full Stack to help build and support its cloud-based healthcare platform that connects patients and providers across the care continuum.

Colombia Full-time Senior Software Engineer
Angular AWS DynamoDB Java JavaScript Microservices MongoDB NestJS Node.js PostgreSQL Redis Spring Boot TypeScript
48 minutes ago
ZoomInfo

Senior Software Engineer

ZoomInfo 1K-5K Professional Services

ZoomInfo is hiring a Senior Full Stack Engineer for its Chorus.ai Conversation Intelligence platform to build customer-facing product experiences that turn large-scale sales call data into actionable insights for sales teams.

Canada Full-time Senior Software Engineer
$161k-$242k
Angular DynamoDB Elasticsearch MongoDB MySQL Node.js Python React REST API Solr
1 hour, 23 minutes ago
Workato

Senior Software Engineer (Ruby/RoR, Platform)

Workato 251-1K IT Services

Workato is hiring a Senior Software Engineer to build API interfaces and backend infrastructure for its user interface while collaborating across product, connectivity, and DevOps teams.

Bulgaria Full-time Senior Software Engineer
Memcached PostgreSQL Redis Ruby Ruby on Rails
1 hour, 41 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers