RemoteLeaf Logo
Log in Sign up

Software Engineering Evaluation Specialist

6 days ago
Indonesia, Malaysia, South Africa
Part-time
Senior
QA Engineer
Software Development
Bash Docker Go Hypothesis Java JIRA Linux Nginx Node.js NumPy Pytest Python PyTorch Rust SciPy
Apply Now
Mindrift.ai: Be the “I” in AI

Mindrift.ai: Be the “I” in AI

Join 10,000+ experts earning $15-50/hr training AI models remotely. Flexible freelance work, weekly payments. No AI experience required. Apply in 5 minutes.

Internet Software & Services
View All Jobs 51

Description

  • Invent realistic developer scenarios based on real bugs, broken ETL jobs, or missing features.
  • Build reproducible Docker environments with pinned dependencies and appropriate container setup.
  • Write deterministic pytest suites that verify outcomes without leaking the fix.
  • Create clear instruction.md files that read like Jira tickets for developers.
  • Develop reference solve.sh scripts that prove each task is solvable.
  • Tune task difficulty so state-of-the-art agents solve them roughly 20–60% of the time.
  • Revise tasks based on feedback from expert QA reviewers.
  • Review other authors’ tasks as a QA reviewer in later stages of the project.

Requirements

  • 3+ years of production software development experience in one backend stack: Python, Go, Node.js, Java, or Rust.
  • Strong Python and pytest fluency, including fixtures, parametrize, monkeypatch, timeouts, and conftest.py.
  • Experience authoring reproducible Dockerfiles with pinned dependencies, multi-stage builds when needed, and non-root users.
  • Comfort debugging in Linux and Bash, including tools like strace, lsof, and journalctl.
  • Experience using AI coding agents such as Claude Code, Cursor, or Roo Code on non-trivial work.
  • Ability to describe a case where an AI coding agent was confidently wrong and how you caught it.
  • Written English proficiency at B2+ level.
  • Preferred domain depth in Security, System Administration, Scientific Computing, DevOps, or Git internals.
  • Familiarity with modern Python tooling such as uv, poetry, and pyproject.toml.
  • Experience with coverage tooling such as pytest-cov, coverage.py, gcov, llvm-cov, or kcov.
  • Experience with fuzzing or property-based testing such as Hypothesis.
  • Prior contribution to agent-evaluation benchmarks or related frameworks.

Benefits

  • Paid contributions with rates up to $35/hour.
  • Task-based compensation tied to performance and volume.
  • Some projects include incentive payments.
  • Flexible schedule: you choose when and how to contribute.
  • Realistic weekly load of 8–20 hours, with more volume for top performers.
  • Project-based work rather than permanent employment.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Parachute Health

Staff Software Engineer

Parachute Health 51-250 Health Care Providers & Services

Parachute Health is hiring a Staff-level software engineer to build and operate its digital post-acute-care ordering platform, helping connect healthcare providers, suppliers, and patients with essential medical equipment.

United States Full-time Lead Software Engineer
$200k-$260k
Agile AWS Go JavaScript Python Ruby on Rails
7 minutes ago
Celonis

Engineering Intern

Celonis 1K-5K IT Services

Celonis is hiring an Engineering Intern to contribute to real features for its cloud-based process intelligence platform while gaining experience with large-scale enterprise software engineering.

United States Internship Entry Level Software Engineer
$63k-$63k
Agile Angular Java Kubernetes Python Spring Boot TypeScript
7 minutes ago
Parachute Health

Staff Software Engineer, Applied AI

Parachute Health 51-250 Health Care Providers & Services

Parachute Health is hiring a Staff AI Engineer to build and scale agentic LLM capabilities across its digital post-acute care product ecosystem.

United States Full-time Lead Software Engineer
$200k-$260k
FastAPI Python React Ruby on Rails TypeScript
7 minutes ago
Pinterest

Staff Software Engineer, Storage Services

Pinterest 5K-10K Internet Software & Services

Pinterest is hiring a Staff Software Engineer to lead the strategy, design, and operation of scalable storage infrastructure supporting SQL and graph workloads across business-critical products.

United States Full-time Lead Software Engineer
$177k-$365k
C C++ Java Python SQL
7 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers