RemoteLeaf Logo
Log in Sign up

Applied Research Scientist, LLM Evaluation & Post-Training

2 months, 3 weeks ago
United States
Full-time
Senior
Research Scientist
Software Development
Hugging Face LLM Machine Learning Python PyTorch Statistics TensorFlow
Apply Now
Innodata

Innodata

Innodata Inc. is a global leader in data engineering, offering end-to-end AI solutions and platforms for businesses worldwide, combining AI and human expertise to solve complex data challenges.

IT Services
1K-5K
Founded 1988
View All Jobs 23

Description

  • Define and execute a research agenda focused on LLM evaluation and evaluation-driven model improvement.
  • Design rigorous experiments to study how evaluation methods affect fine-tuning and post-training outcomes.
  • Develop and validate evaluation frameworks for LLM and multimodal systems, including benchmarks, scoring methods, judge-assisted evaluation, human protocols, and stress testing.
  • Lead research on advanced evaluation areas such as long-context, cross-modal, and dynamic multi-turn evaluation.
  • Analyze model behavior and failure patterns to identify actionable model improvement opportunities.
  • Compare existing evaluation techniques and propose improved methods with clear validity and scalability tradeoffs.
  • Collaborate with AI/ML Research Engineers to translate research methods into scalable pipelines.
  • Partner with Language Data Scientists to integrate human-in-the-loop and synthetic evaluation strategies.
  • Engage customer technical stakeholders to review methodology and provide expert recommendations.
  • Produce technical documentation, internal research reports, and client-facing materials explaining methods, results, assumptions, and limitations.

Requirements

  • MS or PhD in Computer Science, Machine Learning, Statistics, Applied Mathematics, AI, or a related quantitative scientific field; PhD strongly preferred.
  • 5+ years of relevant applied research or research science experience in ML/AI, with substantial work in LLMs or foundation models.
  • Demonstrated experience with LLM evaluation, benchmarking, alignment, post-training, or model quality research.
  • Strong foundation in experimental design, statistical analysis, and scientific reasoning for ML systems.
  • Strong Python coding skills for research experimentation and analysis, including data processing, evaluation pipelines, statistical analysis, and visualization.
  • Experience with modern ML frameworks and tooling such as PyTorch, Hugging Face, and JAX/TensorFlow as applicable.
  • Ability to evaluate human and automated evaluation methods, including tradeoffs in cost, reliability, validity, and scalability.
  • Experience designing reproducible evaluation studies and protocols across datasets, model versions, and evaluation runs.
  • Ability to collaborate directly with research scientists, ML engineers, data scientists, and customer technical stakeholders.
  • Strong communication skills with the ability to present nuanced technical conclusions, assumptions, and limitations clearly.

Benefits

  • Competitive salary range of $175,000 to $225,000 USD per year, based on experience, skills, and qualifications.
  • Opportunity to work on cutting-edge GenAI research at a global data engineering company.
  • Work on research with direct impact on customer solutions and internal platform innovation.
  • Collaborative environment with researchers, engineers, and language/data operations teams.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Precision Medicine Group

Senior Research Scientist, Health Economic Modelling

Precision Medicine Group 251-1K Pharmaceuticals

Precision Medicine Group is seeking a Senior Research Scientist to develop R-based health economic models and evidence synthesis analyses for pharmaceutical and biotech clients.

United Kingdom Senior Research Scientist
C++ Git GitHub Python R Statistics
4 hours ago
Parallel Wireless

Senior/Principal RAN Digital Twin & AI Simulation Engineer

Parallel Wireless 251-1K Wireless Telecommunication Services

Parallel Wireless is seeking a senior wireless systems engineer to lead development of a scalable multi-RAT digital twin for Open RAN, enabling feature development, testing, optimization, and pre-validation across LTE, 5G NR, and 2G.

United States Full-time Senior Machine Learning Engineer Research Scientist
CI/CD Git Linux MATLAB Neural Networks NumPy Pandas Python PyTorch Reinforcement Learning SciPy TensorFlow
4 hours, 30 minutes ago
P

Senior Research Scientist, Health Economic Modelling

Precision AQ 1001-5000 Business Consulting and Services

Precision Medicine Group is seeking a Senior Research Scientist to develop health economic models and evidence synthesis analyses in R for pharmaceutical and biotechnology clients within its Evidence Synthesis and Decision Modelling team.

United Kingdom Full-time Senior Research Scientist
C++ Git GitHub Python R
2 days, 4 hours ago
T

Research Engineer

Talentpluto, Inc. 1-10 Recruiting

Research Engineer at a YC-backed AI training data infrastructure company, building automated systems that verify and improve data quality across a decentralized marketplace.

United States Full-time Mid Level Research Scientist
$140k-$250k
3 days, 4 hours ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers