RemoteLeaf Logo
Log in Sign up

Senior Python Engineer - AI Agent Evaluation

6 hours, 48 minutes ago
United States
Part-time
Lead
Backend Engineer
Software Development
Docker FastAPI JavaScript Kafka PostgreSQL Python React Redis TypeScript
Apply Now
Mindrift.ai: Be the “I” in AI

Mindrift.ai: Be the “I” in AI

Join 10,000+ experts earning $15-50/hr training AI models remotely. Flexible freelance work, weekly payments. No AI experience required. Apply in 5 minutes.

Internet Software & Services
View All Jobs 276

Description

  • Build realistic simulated developer environments with codebases, infrastructure, and supporting context such as tickets, documentation, and conversations.
  • Design evaluation tasks from intermediate states of those environments, including prompts and clear definitions of what counts as a solved task.
  • Write tests that validate agent solutions, accepting all valid approaches and rejecting incorrect ones.
  • Review agent outputs, analyze failures, and refine tasks and tests based on QA feedback.
  • Iterate on tasks to ensure the evaluation is fair, robust, and representative of real developer work.
  • Assess whether tasks are solvable by an AI agent and appropriately challenging for frontier models.

Requirements

  • 8+ years of experience in software development.
  • Experience with Python and FastAPI.
  • Experience with JavaScript or TypeScript and React.
  • Experience with Docker, Postgres, Kafka, and Redis.
  • Experience writing functional and integration tests.
  • English proficiency at B2 level or higher.
  • Submit a CV in English.
  • Availability to complete project-based tasks and meet submission deadlines.
  • Ability to choose your own working schedule while completing approximately 30 hours of work per task.

Benefits

  • Up to $200/hr equivalent compensation, depending on level and pace.
  • Flexible project-based work with control over when and how you work.
  • Estimated 30 hours per task, allowing for self-paced completion.
  • Opportunity to work on leading-edge AI evaluation projects for major tech companies.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Xebia

Senior .NET Developer

Xebia 1K-5K Internet Software & Services

Xebia is hiring a .NET web application engineer to collaborate on architecture, microservices, and cloud-based solutions for international clients across multiple industries.

Bulgaria Romania Europe Full-time Senior Backend Engineer
Agile AWS Azure Confluence Docker DynamoDB EC2 Elasticsearch Git JIRA Kubernetes MariaDB Microservices MySQL .NET Oracle PostgreSQL Redis REST API Serverless Solid.js SQL Server
5 hours, 48 minutes ago
TWO95 International

Lead Java Developer

TWO95 International 51-250 Internet Software & Services

Two95 International is hiring a 100% remote Lead Java Developer for a 6-month+ contract to guide a team through a large implementation and oversee Java development work.

United States Contract Lead Backend Engineer
Angular Java SQL
6 hours, 18 minutes ago
PlayStation

QA Sr Analyst (Contract)

PlayStation 100K+ Household Durables

Sony Santa Monica Studio, part of Sony Interactive Entertainment, is hiring a QA Sr Analyst to support development testing for internal PS5 projects and help ensure high-quality release readiness.

United States Contract Junior QA Engineer
$57k-$85k
JIRA
6 hours, 18 minutes ago
TLATech

Staff QA Engineer

TLATech Specialized Consumer Services

Syllo is seeking a Lead QA Engineer to own and evolve quality assurance for its AI-powered litigation platform, ensuring reliable releases and a strong customer experience as the company scales.

United States Full-time Lead QA Engineer
$170k-$210k
CI/CD Cypress GitHub Actions GitLab CI Jenkins LLM Playwright Selenium
6 hours, 18 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers