RemoteLeaf Logo
Log in Sign up

Applied Research Scientist, LLM Evaluation & Post-Training

1 month, 3 weeks ago
Canada
Full-time
Senior
Research Scientist
Software Development
Generative AI Hugging Face LLM Machine Learning Python PyTorch Statistics TensorFlow
Apply Now
Innodata

Innodata

Innodata Inc. is a global leader in data engineering, offering end-to-end AI solutions and platforms for businesses worldwide, combining AI and human expertise to solve complex data challenges.

IT Services
1K-5K
Founded 1988
View All Jobs 23

Description

  • Define and execute a research agenda focused on LLM evaluation and post-training.
  • Design rigorous experiments to study how evaluation methodologies impact fine-tuning and post-training outcomes.
  • Develop and validate evaluation frameworks for LLM and multimodal systems.
  • Design benchmark and task datasets, scoring methods, judge/model-assisted evaluation, human evaluation protocols, and robustness tests.
  • Lead research on advanced evaluation areas including long-context, cross-modal, and dynamic multi-turn evaluations.
  • Analyze model behavior and failure patterns and recommend improvements to models and evaluation design.
  • Collaborate with AI/ML Research Engineers to translate research methods into scalable evaluation and post-training pipelines.
  • Collaborate with Language Data Scientists to integrate human-in-the-loop and synthetic data/evaluation strategies.
  • Engage customer technical stakeholders to review methodologies and provide expert recommendations.
  • Produce technical documentation, research reports, and client-facing materials explaining methods, results, assumptions, and limitations.

Requirements

  • MS/PhD in Computer Science, Machine Learning, Statistics, Applied Mathematics, AI, or a related quantitative scientific field; PhD strongly preferred.
  • 5+ years of relevant applied research or research science experience in ML/AI, with substantial work in LLMs or foundation models.
  • Demonstrated experience with LLM evaluation, benchmarking, alignment, post-training, or model quality research.
  • Strong foundation in experimental design, statistical analysis, and scientific reasoning for ML systems.
  • Strong coding skills in Python for research experimentation and analysis.
  • Experience with modern ML tooling/frameworks such as PyTorch, Hugging Face, JAX, or TensorFlow.
  • Ability to evaluate human and automated evaluation methods and their tradeoffs in cost, reliability, validity, and scalability.
  • Experience designing reproducible evaluation studies and protocols across datasets, model versions, and runs.
  • Ability to collaborate with research scientists, ML engineers, data scientists, and customer technical stakeholders.
  • Strong communication skills with the ability to present nuanced technical conclusions clearly.

Benefits

  • Expected salary range of $245,000–$315,000 CAD per year.
  • Opportunity to work on cutting-edge GenAI research and evaluation methods.
  • Collaborative environment across research, engineering, and data operations.
  • Chance to contribute to internal benchmark datasets, evaluation frameworks, and reusable research assets.
  • Opportunity to influence customer solutions and internal platform innovation.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Precision Medicine Group

Senior Research Scientist, Health Economic Modelling

Precision Medicine Group 251-1K Pharmaceuticals

Precision Medicine Group is seeking a Senior Research Scientist to develop R-based health economic models and evidence synthesis analyses for pharmaceutical and biotech clients.

United Kingdom Senior Research Scientist
C++ Git GitHub Python R Statistics
4 hours ago
Parallel Wireless

Senior/Principal RAN Digital Twin & AI Simulation Engineer

Parallel Wireless 251-1K Wireless Telecommunication Services

Parallel Wireless is seeking a senior wireless systems engineer to lead development of a scalable multi-RAT digital twin for Open RAN, enabling feature development, testing, optimization, and pre-validation across LTE, 5G NR, and 2G.

United States Full-time Senior Machine Learning Engineer Research Scientist
CI/CD Git Linux MATLAB Neural Networks NumPy Pandas Python PyTorch Reinforcement Learning SciPy TensorFlow
4 hours, 30 minutes ago
P

Senior Research Scientist, Health Economic Modelling

Precision AQ 1001-5000 Business Consulting and Services

Precision Medicine Group is seeking a Senior Research Scientist to develop health economic models and evidence synthesis analyses in R for pharmaceutical and biotechnology clients within its Evidence Synthesis and Decision Modelling team.

United Kingdom Full-time Senior Research Scientist
C++ Git GitHub Python R
2 days, 3 hours ago
T

Research Engineer

Talentpluto, Inc. 1-10 Recruiting

Research Engineer at a YC-backed AI training data infrastructure company, building automated systems that verify and improve data quality across a decentralized marketplace.

United States Full-time Mid Level Research Scientist
$140k-$250k
3 days, 4 hours ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers