RemoteLeaf Logo
Log in Sign up

Senior Python Engineer - AI Coding Agent Evaluation (Freelance)

3 hours, 38 minutes ago
France
Part-time
Lead
AI (Artificial Intelligence)
Software Development
Docker FastAPI JavaScript Kafka PostgreSQL Python React Redis TypeScript
Apply Now
Mindrift.ai: Be the “I” in AI

Mindrift.ai: Be the “I” in AI

Join 10,000+ experts earning $15-50/hr training AI models remotely. Flexible freelance work, weekly payments. No AI experience required. Apply in 5 minutes.

Internet Software & Services
View All Jobs 313

Description

  • Build realistic virtual developer environments with codebases, infrastructure, tickets, docs, conversations, and development history.
  • Design tasks from intermediate states of those environments, including prompts and clear success criteria.
  • Define what counts as a solved task and ensure each task is solvable by an AI agent.
  • Write tests that validate agent solutions, accepting correct approaches and rejecting incorrect ones.
  • Review agent solutions, analyze failures, and refine tasks and tests based on QA feedback.
  • Iterate on evaluation criteria to make the assessment fair and robust.
  • Work within simulated environments to assess how well AI coding agents handle real-world developer tasks.

Requirements

  • 8+ years of experience in software development.
  • Experience with a core stack including Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, and Redis.
  • Experience writing functional and integration tests.
  • English proficiency at B2+ level.
  • Submit CV in English.
  • Availability for project-based, non-permanent work.
  • Ability to complete tasks within an estimated 30 hours per task and meet deadlines.
  • Experience understanding where AI models fail and how to design challenging evaluation scenarios (preferred by role context).

Benefits

  • Up to $200/hr equivalent compensation, depending on level and pace.
  • Flexible schedule: choose when and how to work.
  • Project-based work with tasks estimated at around 30 hours each.
  • Paid tasks upon acceptance after meeting criteria.
  • Opportunity to work on leading tech company AI projects.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Epic Games

Senior Backend Engineer, Developer Platform (Horde)

Epic Games 5K-10K Internet Software & Services

Epic Games is hiring a senior programmer for its Unreal Engine developer platform team to work on Horde, the internal platform that supports CI/CD, remote execution, build health, storage, and developer workflows.

United States Canada Full-time Senior Backend Engineer
$173k-$279k
AWS Azure C# CI/CD gRPC MongoDB .NET Redis
2 hours, 53 minutes ago
Coupang

Senior Staff Back-end Engineer (eCommerce Engineering)

Coupang 1K-5K Internet Software & Services

Coupang is hiring a Senior Staff Backend Engineer to lead the architecture and technical strategy for its core e-commerce platform services across global markets.

South Korea Full-time Lead Backend Engineer
Apache Airflow Apache Spark AWS Cassandra Grafana Java Kafka Kotlin Kubernetes Microservices OpenTelemetry Prometheus Spring
3 hours, 8 minutes ago
Toloka

Record Your Daily Routine & Get Paid - AI Training (Remote)

Toloka 251-1K Internet Software & Services

Project-based freelance work for an AI training platform where contributors record first-person videos of everyday household activities to help train AI systems and robots.

Costa Rica Dominican Republic Bolivia Uruguay Freelance Entry Level AI (Artificial Intelligence)
3 hours, 38 minutes ago
Toloka

Freelance: grabación de tareas cotidianas para proyecto de IA

Toloka 251-1K Internet Software & Services

The company is looking for people to record everyday activities from a first-person perspective using a mobile app to help train AI and robotics systems.

Costa Rica Dominican Republic Bolivia Uruguay Part-time Entry Level AI (Artificial Intelligence)
$12k-$12k
3 hours, 38 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers