RemoteLeaf Logo
Log in Sign up

Freelance Agent Evaluation Engineer

3 hours, 48 minutes ago
Italy
Part-time
Senior
QA Engineer
Software Development
Docker FastAPI GitHub Actions JavaScript Kafka Linux PostgreSQL Pytest Python React Redis REST API TypeScript
Apply Now
Mindrift.ai: Be the “I” in AI

Mindrift.ai: Be the “I” in AI

Join 10,000+ experts earning $15-50/hr training AI models remotely. Flexible freelance work, weekly payments. No AI experience required. Apply in 5 minutes.

Internet Software & Services
View All Jobs 30

Description

  • Build virtual company environments with codebases, infrastructure, and supporting context such as conversations, documentation, and tickets.
  • Assemble and calibrate tasks from intermediate states of a virtual company by crafting prompts and defining fair evaluation criteria.
  • Design tasks in isolated workstation environments with Linux tools, MCP servers, and real web application codebases.
  • Write tests that correctly accept valid solutions and reject incorrect ones without being overly strict or overly lenient.
  • Iterate on tests with an AI agent to verify they catch real issues and do not break on good solutions.
  • Review code produced by agents and analyze why solutions succeeded or failed.
  • Design edge cases and adversarial scenarios to better distinguish strong and weak solutions.
  • Incorporate feedback from expert QA reviewers and revise work to meet quality criteria.

Requirements

  • Please submit your CV in English and indicate your level of English proficiency.
  • Experience in software development, software engineering, and/or test automation, ideally for 5+ years.
  • Degree in Computer Science, Software Engineering, or a related field.
  • Strong Python experience, including FastAPI, pytest, async/await, subprocess, and file operations.
  • Background in full-stack development with React-based interfaces using JavaScript or TypeScript.
  • Experience building robust back-end systems.
  • Experience writing functional and integration tests, not only running them.
  • Experience with Docker containers and infrastructure tools such as Postgres, Kafka, and Redis.
  • Basic CI/CD understanding, including GitHub Actions triggers, labels, and reading results.
  • English proficiency at B2 level.
  • Comfort reading and reasoning across the stack, even without expertise in every technology listed.

Benefits

  • Project-based, non-permanent work with flexible participation.
  • Estimated workload of about 20 hours per project, with scheduling left to the contributor.
  • Ability to choose when and how to work.
  • Compensation of up to $30 per hour equivalent, depending on contribution level and pace.
  • Opportunities to work on different projects with varying earning levels based on scope and expertise.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Anduril Industries

Senior Software Engineer

Anduril Industries 1K-5K Aerospace & Defense

Anduril Industries is hiring a Senior Software Engineer in Irvine, CA to develop and support mission software that integrates customer systems, sensor data, and autonomous capabilities for force protection deployments.

United States Full-time Senior Software Engineer
$191k-$253k
C++ Go JavaScript Python Rust
52 minutes ago
Anduril Industries

Software Engineer

Anduril Industries 1K-5K Aerospace & Defense

Anduril Industries is hiring a software engineer to develop and deploy advanced algorithms and software for mission-critical defense systems and customer applications.

United States Full-time Senior Software Engineer
$165k-$218k
Agile C C++ Computer Vision Machine Learning MATLAB Python Statistics
1 hour, 8 minutes ago
Tinybird

Software Engineer- C++/ClickHouse

Tinybird 11-50 IT Services

Tinybird is hiring a Software Engineer focused on C++ and ClickHouse to contribute full time to open-source database development and improve the performance and capabilities of the platform powering its real-time data products.

Europe Germany Netherlands Portugal Spain Full-time Senior Software Engineer
$67k-$238k
C++ ClickHouse SQL
1 hour, 11 minutes ago
instacart.careers

Senior Software Engineer, Ads

instacart.careers 1K-5K Internet Software & Services

Instacart is hiring a Senior Software Engineer for its Ads team to help build and modernize multi-platform reporting and data systems that deliver accurate insights for advertisers and retailers.

Canada Full-time Senior Software Engineer
$168k-$178k
Apache Airflow Apache Spark Databricks dbt Flink Go Hadoop Hive Kafka Machine Learning Python Ruby on Rails Scala Snowflake SQL Terraform
1 hour, 37 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers