Senior Software Engineer - Agent Evaluation

3 months ago
Part-time
Senior
Software Development
Mindrift.ai: Be the “I” in AI

Mindrift.ai: Be the “I” in AI

Join 10,000+ experts earning $15-50/hr training AI models remotely. Flexible freelance work, weekly payments. No AI experience required. Apply in 5 minutes.

Internet Software & Services

Description

  • Build realistic virtual developer environments with codebases, infrastructure, and supporting context such as tickets, documentation, and conversations.
  • Design evaluation tasks from intermediate states of those environments and define what counts as a solved solution.
  • Write functional and integration tests that verify agent solutions and distinguish correct from incorrect approaches.
  • Review agent solutions, analyze failure cases, and iterate on tasks and tests based on QA feedback.
  • Ensure tasks are solvable by an AI agent and challenging enough to evaluate strong coding models.
  • Refine evaluation criteria to make the assessment fair, robust, and neither too strict nor too lenient.

Requirements

  • 5+ years of experience in software development.
  • Experience with Python (FastAPI).
  • Experience with JavaScript/TypeScript (React).
  • Experience with Docker, Postgres, Kafka, and Redis.
  • Experience writing functional and integration tests.
  • English proficiency at B2+ level.
  • Submit CV in English and indicate your English proficiency.
  • Ability to work on a project-based, non-permanent engagement.
  • Preferred: understanding of how frontier AI coding models fail in realistic development scenarios.

Benefits

  • Up to $60/hr equivalent compensation, depending on level and pace.
  • Project-based work with flexible scheduling; you choose when and how to work.
  • Tasks are estimated at about 20 hours each.
  • Opportunity to work on leading tech company AI evaluation projects.
  • Payment is tied to completed tasks after qualification and project assignment.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Senior Software Engineer, Control Plane

Starburst 251-1K IT Services

Starburst is hiring a Senior Software Engineer for its globally distributed Control Plane team to unify the platform supporting Starburst Enterprise, Starburst Galaxy, and Trino across SaaS, self-managed, and air-gapped environments.

AWS Azure GitHub Java React REST API Trino TypeScript
2 hours, 18 minutes ago

Staff Software Engineer, Privileged Access Management (PAM) / Password Manager (PWM) - India

JumpCloud 251-1K Internet Software & Services

JumpCloud is seeking a remote Staff Software Engineer to architect and guide secure, scalable core-platform microservices while aligning engineering teams around long-term technical strategy and AI-enabled delivery.

AWS C# CI/CD GCP Go gRPC HashiCorp Vault Kubernetes Microservices .NET Playwright PostgreSQL SQL Server TypeScript
2 hours, 18 minutes ago

Senior Software Engineer - Privileged Access Management (PAM / IAM)- India

JumpCloud 251-1K Internet Software & Services

JumpCloud is hiring a remote Senior/Lead Software Engineer to evolve its core AI-powered IT management platform through secure, scalable architecture, enterprise PAM integration, reliable delivery, and AI-assisted engineering practices.

Agile AWS C# Docker Go gRPC HashiCorp Vault Kubernetes Linux macOS Microservices .NET Playwright PostgreSQL REST API SQL Server TypeScript
2 hours, 33 minutes ago

Senior Software Engineer, Add-Ons

Mozilla 251-1K Internet Software & Services

Mozilla is hiring a Senior Software Engineer – Operations to modernize and operate addons.mozilla.org, building reliable systems and tools that support reviewers, developers, and millions of Firefox add-on users.

Django GCP MySQL Python React TypeScript
2 hours, 48 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers