RemoteLeaf Logo
Log in Sign up

Technical AI Evaluator - Python, Automation or Security

20 hours, 40 minutes ago
United States
Part-time
Senior
AI (Artificial Intelligence)
Software Development
Docker FastAPI JavaScript Kafka PostgreSQL Python React Redis TypeScript
Apply Now
Mindrift.ai: Be the “I” in AI

Mindrift.ai: Be the “I” in AI

Join 10,000+ experts earning $15-50/hr training AI models remotely. Flexible freelance work, weekly payments. No AI experience required. Apply in 5 minutes.

Internet Software & Services
View All Jobs 87

Description

  • Build realistic developer environments with codebases, infrastructure, and supporting context such as tickets, documentation, and conversations.
  • Design coding tasks from intermediate environment states, including prompts and clear definitions of what counts as solved.
  • Write functional and integration tests that validate correct solutions and reject incorrect ones.
  • Review AI agent solutions, analyze failure cases, and refine tasks and tests based on QA feedback.
  • Iterate on evaluation criteria to ensure tasks are fair, robust, and solvable by AI agents.
  • Assess how well AI coding agents handle real-world developer tasks.

Requirements

  • 5+ years of software development experience.
  • Experience with Python, including FastAPI.
  • Experience with JavaScript/TypeScript, including React.
  • Experience with Docker, Postgres, Kafka, and Redis.
  • Experience writing functional and integration tests.
  • English proficiency at B2 level or higher.
  • Submit your CV in English and indicate your English proficiency level.
  • Able to work in a project-based, non-permanent role.
  • Preferred candidates are comfortable evaluating complex coding tasks and solution validity.

Benefits

  • Up to $50/hr equivalent compensation, depending on level and pace.
  • Tasks are estimated at approximately 20 hours each.
  • Flexible schedule with self-directed working hours.
  • Project-based opportunities with leading tech companies.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

PandaDoc

Senior Python Engineer, Document App (Content Engineering Team)

PandaDoc 251-1K Internet Software & Services

PandaDoc is hiring a Senior Python Engineer for its Content engineering team to build and operate the backend systems that power real-time collaborative document editing at global scale.

Ukraine Contract Senior Backend Engineer
Agile AWS Docker Grafana Kafka Kubernetes Microservices NATS PostgreSQL Python RabbitMQ Redis SQL
19 hours, 25 minutes ago
PandaDoc

Senior Python Engineer, Document App (Content Engineering Team)

PandaDoc 251-1K Internet Software & Services

PandaDoc is hiring a Senior Python Engineer to help build and operate the backend systems that power real-time collaborative document editing and document workflow automation.

Germany Contract Senior Backend Engineer
AWS Docker Grafana Kafka Kubernetes Microservices NATS PostgreSQL Python RabbitMQ Redis SQL
19 hours, 25 minutes ago
PandaDoc

Senior Python Engineer, Document App (Content Engineering Team)

PandaDoc 251-1K Internet Software & Services

PandaDoc is hiring a Senior Python Engineer to help build and operate the backend systems that power real-time collaborative document editing across its global SaaS platform.

Poland Contract Senior Backend Engineer
$3k-$5k
Agile AWS Docker Grafana Kafka Kubernetes LLM Microservices NATS PostgreSQL Python RabbitMQ Redis SQL
19 hours, 25 minutes ago
PandaDoc

Senior Python Engineer, Document App (Content Engineering Team)

PandaDoc 251-1K Internet Software & Services

PandaDoc is hiring a Senior Python Engineer to help build and operate the backend systems that power real-time collaborative document editing across its remote, global SaaS platform.

Spain Contract Senior Backend Engineer
Agile AWS Docker Grafana Kafka Kubernetes LLM Microservices NATS PostgreSQL Python RabbitMQ Redis SQL
19 hours, 40 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers