Researcher, Evaluations

1 month, 1 week ago
Full-time
Mid Level
Data Science and Analytics
Epoch AI

Epoch AI

Epoch is a research institute focused on exploring critical trends and questions that influence the development and governance of artificial intelligence, providing valuable insights into its societal implications.

Professional Services
1-10
Founded 2022

Description

  • Create and curate an evaluation suite of realistic real-world tasks and update it as AI capabilities evolve.
  • Design grading rubrics for evaluating AI performance on open-ended tasks.
  • Regularly evaluate new frontier AI models and products using the task suite.
  • Analyze evaluation results and compare model performance across tasks.
  • Create public-facing reports, blog posts, and data visualizations based on research findings.
  • Ensure evaluation work informs other research topics and keeps the team updated.
  • Automate parts of the workflow where useful and help build standalone benchmarks from the evaluation process.

Requirements

  • Experience conducting research and data analysis with enough comfort in light coding to analyze results.
  • Strong analytical thinking and the ability to run rigorous, evidence-based experiments.
  • A grounded, skeptical perspective on AI capabilities and limitations.
  • Comfort working with AI agents and tools, including delegating tasks to AI systems.
  • Familiarity with AI benchmarks and evaluations, with opinions on what they can and cannot show.
  • Strong written communication skills for clear, precise explanations of nuanced observations.
  • Experience testing frontier models and writing assessments of their capabilities, preferred.
  • Coding skills, including Python proficiency, preferred.
  • Professional-level English proficiency and the ability to submit application materials in English.
  • Preference for candidates who can overlap with UTC-8 and UTC time zones and who can travel for three annual retreats.

Benefits

  • Annual salary of $115,000–$200,000 USD, depending on location and experience.
  • Fully remote work with flexible hours.
  • Comprehensive global benefits, including health insurance, supplemental local benefits where available, life insurance, and a pension plan if applicable.
  • Generous PTO, including 30 protected days per year, unlimited personal and sick leave, and 4 months of paid parental leave for eligible permanent staff.
  • Flexible expense budget for equipment, productivity tools, learning and development, and unlimited AI tools spending subject to approval.
  • Paid work trips, including three staff retreats per year and relevant conferences.
  • Access to well-equipped Berkeley offices with paid meals, snacks, gym access, and at least 20 in-office days per year for all staff.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

AI Evaluation Engineer (Python, QA or Security)

Mindrift.ai: Be the “I” in AI Internet Software & Services

Mindrift is seeking experienced software developers for project-based work creating realistic coding evaluation tasks and tests for AI agents.

Docker FastAPI JavaScript Kafka PostgreSQL Python React Redis TypeScript
13 hours, 12 minutes ago

Social Media Content Evaluator (Hindi)

RWS Group 5K-10K Internet Software & Services

RWS is hiring a freelance part-time media analyst in India to review short-form videos and produce language classification and locale-validation data for AI training.

Machine Learning
1 day, 12 hours ago

Forward Deployed Engineer, Data & AI

Warner Music Group is hiring a Forward Deployed Engineer to partner with business stakeholders across labels and global functions to rapidly build data- and AI-driven solutions within its Global Tech Data and Insights organization.

Databricks Python React Snowflake
1 day, 12 hours ago

PointClickCare - (Canada) Senior Clinical AI Nurse SME- 6 months contract

PointClickCare 1K-5K Health Care Providers & Services

PointClickCare is hiring a part-time clinical subject matter expert to support AI and data labeling work in healthcare with a focus on clinical accuracy and provider workflows.

LLM Machine Learning NLP
1 day, 12 hours ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers