RemoteLeaf Logo
Log in Sign up

Senior Software Engineer - AI Coding Benchmarks

4 weeks, 2 days ago
Poland
Part-time
Senior
QA Engineer
Software Development
Docker FastAPI JavaScript Kafka PostgreSQL Python React Redis TypeScript
Apply Now
Mindrift.ai: Be the “I” in AI

Mindrift.ai: Be the “I” in AI

Join 10,000+ experts earning $15-50/hr training AI models remotely. Flexible freelance work, weekly payments. No AI experience required. Apply in 5 minutes.

Internet Software & Services
View All Jobs 84

Description

  • Build realistic developer environments with codebases, infrastructure, and supporting context such as tickets, docs, and conversations.
  • Design tasks from intermediate states of those environments, including prompts and clear definitions of what counts as a solved task.
  • Write tests that validate agent solutions by accepting all correct approaches and rejecting incorrect ones.
  • Review agent outputs, analyze failures, and refine tasks and tests based on QA feedback.
  • Iterate on evaluation criteria to make tasks fair, robust, and solvable by AI agents.
  • Assess whether tasks are appropriately challenging for frontier coding models.
  • Guide and evaluate the agent’s work rather than writing the full code solution from scratch.

Requirements

  • 5+ years of experience in software development.
  • Experience with Python and FastAPI.
  • Experience with JavaScript/TypeScript and React.
  • Experience with Docker, Postgres, Kafka, and Redis.
  • Experience writing functional and integration tests.
  • English proficiency at B2 level or higher.
  • CV must be submitted in English.
  • Project-based participation with the ability to complete tasks by the deadline.
  • Ability to estimate work on tasks that take around 20 hours each, depending on complexity.

Benefits

  • Up to $40/hr equivalent compensation, depending on level and pace.
  • Flexible schedule with work completed on your own time.
  • Project-based work with estimated 20-hour task assignments.
  • Opportunity to contribute to AI evaluation work for leading tech companies.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Litify

QA Engineer

Litify 51-250 Specialized Consumer Services

Litify is seeking a QA Engineer to own end-to-end quality for its Salesforce-based legal software within an agile Product & Engineering squad.

United States Full-time Mid Level QA Engineer
$100k-$110k
Agile CI/CD Cypress JIRA Playwright Salesforce Scrum Selenium
15 hours ago
Pinterest

Senior Software Engineer

Pinterest 1K-5K Internet Software & Services

Pinterest is hiring a backend engineer to build and evolve large-scale, distributed systems that support user-facing applications and improve the end-to-end product experience.

United States Full-time Senior Software Engineer
$252k-$320k
Apache Airflow Apache Spark AWS C++ CI/CD Docker Elasticsearch Flink Grafana Java Kafka Kubernetes Memcached MySQL Node.js PostgreSQL Python Redis
15 hours ago
G

Senior Software Development Engineer

GoDaddy 5001-10000 Technology, Information and Internet

GoDaddy is hiring a remote Growth team engineer in Colombia to build and operate production AI agent systems and full-stack applications that support customers at scale.

Colombia Full-time Senior Software Engineer
AWS Azure CI/CD Docker Elasticsearch Go Kubernetes LLM Microservices MongoDB Node.js PostgreSQL Python React Redis SQL Vue.js
15 hours, 15 minutes ago
Scale AI

SWE Fellow - Human Frontier Collective (UK)

Scale AI 251-1K Diversified Consumer Services

The Human Frontier Collective Fellowship at Scale is a fully remote contract role supporting AI research by designing, evaluating, and interpreting advanced generative AI systems with interdisciplinary experts and partner labs.

United Kingdom Contract Senior AI (Artificial Intelligence) Software Engineer
C++ Generative AI Java JavaScript Machine Learning Python Rust Swift
15 hours, 15 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers