RemoteLeaf Logo
Log in Sign up

Senior Data Scientist AI Evaluation

2 hours, 9 minutes ago
Remote
Full-time
Senior
AI (Artificial Intelligence)
Data Science and Analytics
CI/CD Machine Learning Python SQL
Apply Now
Alpaca

Alpaca

Alpaca is a developer-first API for stock and crypto trading, offering easy-to-use APIs for building apps and trading algorithms.

Capital Markets
51-250
Founded 2015
$87M raised
View All Jobs 61

Description

  • Define ground truth, metrics, and scoring methods for AI models and agents.
  • Build repeatable evaluation loops to monitor quality and detect regressions before release.
  • Partner with engineering and analytics engineering to operationalize evaluation harnesses.
  • Translate evaluation results into actionable system-improvement recommendations.
  • Establish evaluation guidelines, documentation, quality standards, and review practices.
  • Mentor teams and promote measurable AI-quality practices across the organization.
  • Collaborate with Product, Engineering, Analytics Engineering, and business stakeholders to define quality criteria.

Requirements

  • 6–10 years of experience in quantitative data science or machine learning, with focused experience in measurement or evaluation.
  • Strong statistical and machine-learning foundation, including sample sizing, confidence intervals, significance testing, and non-determinism handling.
  • Experience validating automated graders against human ground truth.
  • Proficiency in Python and SQL.
  • Experience evaluating models in production environments.
  • Strong judgment in defining quality metrics and ground truth for ambiguous outputs.
  • Excellent communication and cross-functional collaboration skills.
  • Strong problem-solving ability in fast-paced, greenfield environments.
  • Quantitative degree or equivalent industry experience.
  • Experience with production LLM or agent evaluation, eval harnesses, LLM-as-judge calibration, and CI regression gates preferred.
  • Experience evaluating text-to-SQL, analytics agents, or systems with data-verifiable correctness preferred.
  • Fintech, brokerage, or other high-consequence domain experience preferred.

Benefits

  • Competitive salary and stock options.
  • Health benefits.
  • One-time $500 new-hire home-office setup allowance.
  • $150 monthly stipend via a Brex Card.
  • Globally distributed work environment.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

MongoDB

Senior Manager, AI and People Transformation

MongoDB 1K-5K Internet Software & Services

MongoDB is seeking a Senior Manager, AI & People Transformation to redesign People operations through AI, automation, workflow architecture, and scalable cross-functional solutions.

United States Full-time Lead AI (Artificial Intelligence) Operations Manager
$115k-$198k
AWS Azure GCP MongoDB
2 hours, 9 minutes ago
phData

AI Services Lead - Client Services

phData 251-1K IT Services

phData is seeking an AI Services Lead to build and grow Anthropic-focused client accounts, overseeing delivery, shaping AI transformation roadmaps, and driving long-term expansion opportunities.

United States Full-time Senior Account Manager AI (Artificial Intelligence)
Generative AI
2 hours, 9 minutes ago
Phoenix Ecommerce Technologies

AI-Native Chief of Staff

Phoenix Ecommerce Technologies 11-50 information technology & services

Phoenix is hiring an AI-native Chief of Staff to run the CEO’s operating system, drive cross-company execution, and build agentic workflows that help its DTC commerce platform scale.

United States Full-time Mid Level AI (Artificial Intelligence) Executive Assistant
CRM E-commerce
3 hours, 9 minutes ago
Weekday

Music & Lyrics Expert - Swedish

Weekday 11-50 Construction & Engineering

A client is seeking a Swedish Music & Lyrics Expert to evaluate generative AI music outputs for quality, originality, linguistic authenticity, cultural relevance, and prompt adherence in collaboration with an AI research organization.

United States Contract Mid Level AI (Artificial Intelligence)
$87k-$162k
3 hours, 9 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers