RemoteLeaf Logo
Log in Sign up

Software Engineering Evaluation Specialist

4 days ago
Jordan, Lebanon, Turkey, Armenia
Part-time
Senior
QA Engineer
Software Development
Bash Docker Git Go Hypothesis Java JIRA Linux Nginx Node.js NumPy Pytest Python PyTorch Rust SciPy
Apply Now
Mindrift.ai: Be the “I” in AI

Mindrift.ai: Be the “I” in AI

Join 10,000+ experts earning $15-50/hr training AI models remotely. Flexible freelance work, weekly payments. No AI experience required. Apply in 5 minutes.

Internet Software & Services
View All Jobs 51

Description

  • Invent realistic developer scenarios such as real bugs, broken ETL jobs, or missing features.
  • Build reproducible Docker environments with pinned dependencies and non-root execution where needed.
  • Write deterministic pytest-based tests that verify outcomes without leaking the fix.
  • Draft instruction files that read like developer tickets.
  • Create reference solve scripts that prove each task is solvable.
  • Calibrate task difficulty so current state-of-the-art agents solve them 20–60% of the time.
  • Iterate on tasks based on feedback from expert QA reviewers.
  • Review other authors’ tasks later as part of QA work.

Requirements

  • 3+ years of production software development in one backend stack such as Python, Go, Node.js, Java, or Rust.
  • Strong Python and pytest fluency, including fixtures, parametrize, monkeypatch, timeouts, and conftest.py.
  • Experience authoring reproducible Dockerfiles with pinned dependencies, multi-stage builds when needed, and non-root users.
  • Comfort debugging inside Linux containers using tools like strace, lsof, and journalctl, plus broader Bash experience.
  • Experience using AI coding agents such as Claude Code, Cursor, or Roo Code on non-trivial work.
  • B2+ written English proficiency.
  • Domain depth in Security, System Administration, Scientific Computing, DevOps, or Git internals is preferred.
  • Familiarity with modern Python tooling such as uv, poetry, or pyproject.toml is preferred.
  • Experience with coverage tooling such as pytest-cov, coverage.py, gcov, llvm-cov, or kcov is preferred.
  • Experience with fuzzing or property-based testing, such as Hypothesis, is preferred.
  • Prior contribution to agent-evaluation benchmarks or related frameworks is preferred.

Benefits

  • Paid project-based contributions with rates up to $35/hour.
  • Task-based compensation tied to performance and volume.
  • Some projects include incentive payments.
  • Flexible time commitment, with work scheduled by the contributor.
  • Realistic weekly load of 8–20 hours, with higher volume available for top performers.
  • Onboarding and task workload are clearly defined, including estimated hours per task.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Parachute Health

Staff Software Engineer

Parachute Health 51-250 Health Care Providers & Services

Parachute Health is hiring a Staff-level software engineer to build and operate its digital post-acute-care ordering platform, helping connect healthcare providers, suppliers, and patients with essential medical equipment.

United States Full-time Lead Software Engineer
$200k-$260k
Agile AWS Go JavaScript Python Ruby on Rails
6 minutes ago
Celonis

Engineering Intern

Celonis 1K-5K IT Services

Celonis is hiring an Engineering Intern to contribute to real features for its cloud-based process intelligence platform while gaining experience with large-scale enterprise software engineering.

United States Internship Entry Level Software Engineer
$63k-$63k
Agile Angular Java Kubernetes Python Spring Boot TypeScript
6 minutes ago
Parachute Health

Staff Software Engineer, Applied AI

Parachute Health 51-250 Health Care Providers & Services

Parachute Health is hiring a Staff AI Engineer to build and scale agentic LLM capabilities across its digital post-acute care product ecosystem.

United States Full-time Lead Software Engineer
$200k-$260k
FastAPI Python React Ruby on Rails TypeScript
7 minutes ago
Pinterest

Staff Software Engineer, Storage Services

Pinterest 5K-10K Internet Software & Services

Pinterest is hiring a Staff Software Engineer to lead the strategy, design, and operation of scalable storage infrastructure supporting SQL and graph workloads across business-critical products.

United States Full-time Lead Software Engineer
$177k-$365k
C C++ Java Python SQL
7 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers