RemoteLeaf Logo
Log in Sign up

Software Engineering Evaluation Specialist

53 minutes ago
Jamaica, Costa Rica, Ecuador
Part-time
Mid Level
AI (Artificial Intelligence)
Software Development
Bash Docker Git Go Hypothesis Java JIRA Linux Nginx Node.js NumPy Pytest Python PyTorch Rust SciPy
Apply Now
Mindrift.ai: Be the “I” in AI

Mindrift.ai: Be the “I” in AI

Join 10,000+ experts earning $15-50/hr training AI models remotely. Flexible freelance work, weekly payments. No AI experience required. Apply in 5 minutes.

Internet Software & Services
View All Jobs 51

Description

  • Create realistic software-development scenarios involving bugs, broken ETL processes, or missing features.
  • Build reproducible Docker environments with pinned dependencies.
  • Write deterministic pytest tests that verify outcomes without exposing the solution.
  • Create Jira-style task instructions and a reference solve.sh demonstrating solvability.
  • Calibrate task difficulty so state-of-the-art AI agents succeed 20–60% of the time.
  • Iterate on tasks based on expert QA feedback.
  • Review other authors’ tasks as a QA reviewer when needed.

Requirements

  • 3+ years of production software development experience in Python, Go, Node.js, Java, or Rust, with depth in one backend stack.
  • Fluency with Python and pytest, including fixtures, parametrization, monkeypatching, timeouts, and conftest.py.
  • Experience authoring reproducible Dockerfiles, pinned dependencies, multi-stage builds, and non-root containers.
  • Comfort debugging Linux and Bash environments, including tools such as strace, lsof, and journalctl.
  • Hands-on experience using AI coding agents such as Claude Code, Cursor, or Roo Code on non-trivial work.
  • B2+ written English proficiency.
  • Preferred: expertise in security, system administration, scientific computing, DevOps, or Git internals.
  • Preferred: experience with uv, Poetry, pyproject.toml, coverage tools, fuzzing, or property-based testing with Hypothesis.
  • Preferred: contributions to agent-evaluation benchmarks or related frameworks.
  • Submit an English CV, indicate English proficiency, and include a GitHub profile link if available.

Benefits

  • Paid project-based contributions at rates up to $35 per hour.
  • Task-based compensation with rates varying by expertise, location, project needs, performance, and volume.
  • Some projects offer incentive payments.
  • Flexible contribution schedule, with a typical commitment of 8–20 hours per week.
  • Higher-volume opportunities may be available for top performers.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Superior Executive And Legal Recruiting

AI Solutions Engineer (Contractor) - Fully Remote

Superior Executive And Legal Recruiting 1-10 Professional Services

Superior Executive & Legal Recruiting is seeking a fully remote AI Solutions Engineer Consultant to help a high-volume legal business assess workflows and implement a firm-wide AI and automation strategy.

United States Contract Lead AI (Artificial Intelligence)
7 minutes ago
kake

Electrical Engineer with Python Proficiency - AI Trainer

kake 1-10 Internet Software & Services

Electrical Engineering professionals with Python expertise contribute to project-based AI development initiatives by designing and evaluating simulation-driven tasks that test and improve frontier AI models.

United States Contract Junior AI (Artificial Intelligence)
Linux Python
8 minutes ago
Nimble Gravity

AI Enablement & Adoption Manager

Nimble Gravity 51-250 IT Services

Nimble Gravity is seeking a remote, US-based AI Enablement Lead to help clients—particularly financial institutions—adopt AI responsibly and achieve measurable improvements in how work gets done.

United States Senior AI (Artificial Intelligence) Training Manager
Generative AI System Design
8 minutes ago
EvolutionIQ

Senior AI / ML Engineer (LLMs)

EvolutionIQ 51-250 Internet Software & Services

EvolutionIQ is hiring an AI/ML Engineer to build and scale reliable AI-powered medical claim synthesis solutions that improve accuracy, efficiency, and outcomes for insurance claims teams.

United States Full-time Senior AI (Artificial Intelligence)
$200k-$235k
Microservices Python
23 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers