RemoteLeaf Logo
Log in Sign up

Applied Research Scientist, LLM Evaluation & Post-Training

3 weeks, 5 days ago
Canada
Full-time
Senior
Research Scientist
Software Development
Generative AI Hugging Face LLM Machine Learning Python PyTorch Statistics TensorFlow
Apply Now
Innodata

Innodata

Innodata Inc. is a global leader in data engineering, offering end-to-end AI solutions and platforms for businesses worldwide, combining AI and human expertise to solve complex data challenges.

IT Services
1K-5K
Founded 1988
View All Jobs 24

Description

  • Define and execute a research agenda focused on LLM evaluation and post-training.
  • Design rigorous experiments to study how evaluation methodologies impact fine-tuning and post-training outcomes.
  • Develop and validate evaluation frameworks for LLM and multimodal systems.
  • Design benchmark and task datasets, scoring methods, judge/model-assisted evaluation, human evaluation protocols, and robustness tests.
  • Lead research on advanced evaluation areas including long-context, cross-modal, and dynamic multi-turn evaluations.
  • Analyze model behavior and failure patterns and recommend improvements to models and evaluation design.
  • Collaborate with AI/ML Research Engineers to translate research methods into scalable evaluation and post-training pipelines.
  • Collaborate with Language Data Scientists to integrate human-in-the-loop and synthetic data/evaluation strategies.
  • Engage customer technical stakeholders to review methodologies and provide expert recommendations.
  • Produce technical documentation, research reports, and client-facing materials explaining methods, results, assumptions, and limitations.

Requirements

  • MS/PhD in Computer Science, Machine Learning, Statistics, Applied Mathematics, AI, or a related quantitative scientific field; PhD strongly preferred.
  • 5+ years of relevant applied research or research science experience in ML/AI, with substantial work in LLMs or foundation models.
  • Demonstrated experience with LLM evaluation, benchmarking, alignment, post-training, or model quality research.
  • Strong foundation in experimental design, statistical analysis, and scientific reasoning for ML systems.
  • Strong coding skills in Python for research experimentation and analysis.
  • Experience with modern ML tooling/frameworks such as PyTorch, Hugging Face, JAX, or TensorFlow.
  • Ability to evaluate human and automated evaluation methods and their tradeoffs in cost, reliability, validity, and scalability.
  • Experience designing reproducible evaluation studies and protocols across datasets, model versions, and runs.
  • Ability to collaborate with research scientists, ML engineers, data scientists, and customer technical stakeholders.
  • Strong communication skills with the ability to present nuanced technical conclusions clearly.

Benefits

  • Expected salary range of $245,000–$315,000 CAD per year.
  • Opportunity to work on cutting-edge GenAI research and evaluation methods.
  • Collaborative environment across research, engineering, and data operations.
  • Chance to contribute to internal benchmark datasets, evaluation frameworks, and reusable research assets.
  • Opportunity to influence customer solutions and internal platform innovation.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

P

Associate Research Scientist, Real World Evidence

Precision AQ 1001-5000 Business Consulting and Services

Precision AQ is hiring a fully remote Associate Research Scientist on its Real-World Evidence team to support real-world studies in the pharmaceutical and biotech space.

United States Full-time Entry Level Research Scientist
$66k-$94k
Python R SQL
1 day, 6 hours ago
Pinterest

Sr. Staff Quantitative Product Researcher

Pinterest 5K-10K Internet Software & Services

Pinterest is hiring a Sr. Staff Quantitative Product Researcher to lead user-centered measurement across the consumer experience and help shape how the company evaluates the Pinner experience.

United States Full-time Lead Data Scientist Research Scientist
$164k-$337k
Machine Learning Python R SQL
3 days, 6 hours ago
Natera

Lead Bioinformatician (cfDNA Algorithms and Pipelines)

Natera 1K-5K Pharmaceuticals

Natera is hiring a Lead Bioinformatician to develop and support the computational methods behind diagnostic sequencing assays for women’s and organ health.

United States Full-time Senior Data Scientist Research Scientist
$131k-$164k
Prototyping Python
1 week ago
Natera

Associate Director of Bioinformatics (Women's Health and Organ Health)

Natera 1K-5K Pharmaceuticals

Natera is hiring an Associate Director to lead bioinformatics research and assay-development work for women’s health and organ health diagnostics, with responsibility for guiding a team and transitioning new methods into production.

United States Full-time Lead Data Scientist Research Scientist
$167k-$202k
AWS CI/CD Docker GCP Git Java Kubernetes Machine Learning Python R Shell Scripting
1 week ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers