RemoteLeaf Logo
Log in Sign up

Senior Software Engineer - Agent Evaluation

2 months, 1 week ago
United States
Part-time
Senior
QA Engineer
Software Development
Docker FastAPI JavaScript Kafka PostgreSQL Python React Redis TypeScript
Apply Now
Mindrift.ai: Be the “I” in AI

Mindrift.ai: Be the “I” in AI

Join 10,000+ experts earning $15-50/hr training AI models remotely. Flexible freelance work, weekly payments. No AI experience required. Apply in 5 minutes.

Internet Software & Services
View All Jobs 21

Description

  • Build realistic virtual developer environments with codebases, infrastructure, and supporting context such as tickets, documentation, and conversations.
  • Design evaluation tasks from intermediate states of those environments and define what counts as a solved solution.
  • Write functional and integration tests that verify agent solutions and distinguish correct from incorrect approaches.
  • Review agent solutions, analyze failure cases, and iterate on tasks and tests based on QA feedback.
  • Ensure tasks are solvable by an AI agent and challenging enough to evaluate strong coding models.
  • Refine evaluation criteria to make the assessment fair, robust, and neither too strict nor too lenient.

Requirements

  • 5+ years of experience in software development.
  • Experience with Python (FastAPI).
  • Experience with JavaScript/TypeScript (React).
  • Experience with Docker, Postgres, Kafka, and Redis.
  • Experience writing functional and integration tests.
  • English proficiency at B2+ level.
  • Submit CV in English and indicate your English proficiency.
  • Ability to work on a project-based, non-permanent engagement.
  • Preferred: understanding of how frontier AI coding models fail in realistic development scenarios.

Benefits

  • Up to $60/hr equivalent compensation, depending on level and pace.
  • Project-based work with flexible scheduling; you choose when and how to work.
  • Tasks are estimated at about 20 hours each.
  • Opportunity to work on leading tech company AI evaluation projects.
  • Payment is tied to completed tasks after qualification and project assignment.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Kapitus

Software Engineer II - Engineering

Kapitus 251-1K Diversified Financial Services

Kapitus is seeking a Software Engineer to build modern interfaces and backend services that replace legacy systems and support small-business lending operations.

United States Full-time Senior Software Engineer
$96k-$154k
Agile AWS CI/CD Flask Python Salesforce Scrum Terraform
8 hours, 41 minutes ago
AcuityMD

Senior Software Engineer, Applications

AcuityMD 51-250 Health Care Providers & Services

AcuityMD is hiring a Senior Software Engineer to build end-to-end web and mobile application features that help MedTech companies improve sales strategy and expand access to medical technologies.

United States Full-time Senior Software Engineer
$155k-$250k
GCP GraphQL JavaScript Node.js PostgreSQL Python React SQL TypeScript
8 hours, 56 minutes ago
GR8 Tech

Middle QA Engineer for Client Centric Team

GR8 Tech 251-1K IT Services

GR8_TECH is seeking a Manual QA Engineer to ensure reliable quality across its B2B iGaming Web, native Mobile, Backend, and integration products from feature analysis through release support.

Poland Full-time Mid Level QA Engineer
Android CI/CD Docker Git HTTP iOS Kafka Kubernetes Microservices OpenAPI Postman SQL TLS
8 hours, 56 minutes ago
CompTech Computer Technologies

Software Systems Engineer

CompTech Computer Technologies 51-250 Internet Software & Services

CompTech is seeking a remote Software Systems Engineer to support the Defense Logistics Agency’s ALMS enterprise system by sustaining, enhancing, and optimizing its DocuSign CLM platform and related automation.

United States Full-time Lead Software Engineer
$115k-$125k
Agile
9 hours, 26 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers