RemoteLeaf Logo
Log in Sign up

Applied Research Scientist, LLM Evaluation & Post-Training

1 month, 3 weeks ago
United States
Full-time
Senior
Research Scientist
Software Development
Hugging Face LLM Machine Learning Python PyTorch Statistics TensorFlow
Apply Now
Innodata

Innodata

Innodata Inc. is a global leader in data engineering, offering end-to-end AI solutions and platforms for businesses worldwide, combining AI and human expertise to solve complex data challenges.

IT Services
1K-5K
Founded 1988
View All Jobs 24

Description

  • Define and execute a research agenda focused on LLM evaluation and evaluation-driven model improvement.
  • Design rigorous experiments to study how evaluation methods affect fine-tuning and post-training outcomes.
  • Develop and validate evaluation frameworks for LLM and multimodal systems, including benchmarks, scoring methods, judge-assisted evaluation, human protocols, and stress testing.
  • Lead research on advanced evaluation areas such as long-context, cross-modal, and dynamic multi-turn evaluation.
  • Analyze model behavior and failure patterns to identify actionable model improvement opportunities.
  • Compare existing evaluation techniques and propose improved methods with clear validity and scalability tradeoffs.
  • Collaborate with AI/ML Research Engineers to translate research methods into scalable pipelines.
  • Partner with Language Data Scientists to integrate human-in-the-loop and synthetic evaluation strategies.
  • Engage customer technical stakeholders to review methodology and provide expert recommendations.
  • Produce technical documentation, internal research reports, and client-facing materials explaining methods, results, assumptions, and limitations.

Requirements

  • MS or PhD in Computer Science, Machine Learning, Statistics, Applied Mathematics, AI, or a related quantitative scientific field; PhD strongly preferred.
  • 5+ years of relevant applied research or research science experience in ML/AI, with substantial work in LLMs or foundation models.
  • Demonstrated experience with LLM evaluation, benchmarking, alignment, post-training, or model quality research.
  • Strong foundation in experimental design, statistical analysis, and scientific reasoning for ML systems.
  • Strong Python coding skills for research experimentation and analysis, including data processing, evaluation pipelines, statistical analysis, and visualization.
  • Experience with modern ML frameworks and tooling such as PyTorch, Hugging Face, and JAX/TensorFlow as applicable.
  • Ability to evaluate human and automated evaluation methods, including tradeoffs in cost, reliability, validity, and scalability.
  • Experience designing reproducible evaluation studies and protocols across datasets, model versions, and evaluation runs.
  • Ability to collaborate directly with research scientists, ML engineers, data scientists, and customer technical stakeholders.
  • Strong communication skills with the ability to present nuanced technical conclusions, assumptions, and limitations clearly.

Benefits

  • Competitive salary range of $175,000 to $225,000 USD per year, based on experience, skills, and qualifications.
  • Opportunity to work on cutting-edge GenAI research at a global data engineering company.
  • Work on research with direct impact on customer solutions and internal platform innovation.
  • Collaborative environment with researchers, engineers, and language/data operations teams.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

P

Associate Research Scientist, Real World Evidence

Precision AQ 1001-5000 Business Consulting and Services

Precision AQ is hiring a fully remote Associate Research Scientist on its Real-World Evidence team to support real-world studies in the pharmaceutical and biotech space.

United States Full-time Entry Level Research Scientist
$66k-$94k
Python R SQL
1 day, 6 hours ago
Pinterest

Sr. Staff Quantitative Product Researcher

Pinterest 5K-10K Internet Software & Services

Pinterest is hiring a Sr. Staff Quantitative Product Researcher to lead user-centered measurement across the consumer experience and help shape how the company evaluates the Pinner experience.

United States Full-time Lead Data Scientist Research Scientist
$164k-$337k
Machine Learning Python R SQL
3 days, 6 hours ago
Natera

Lead Bioinformatician (cfDNA Algorithms and Pipelines)

Natera 1K-5K Pharmaceuticals

Natera is hiring a Lead Bioinformatician to develop and support the computational methods behind diagnostic sequencing assays for women’s and organ health.

United States Full-time Senior Data Scientist Research Scientist
$131k-$164k
Prototyping Python
1 week ago
Natera

Associate Director of Bioinformatics (Women's Health and Organ Health)

Natera 1K-5K Pharmaceuticals

Natera is hiring an Associate Director to lead bioinformatics research and assay-development work for women’s health and organ health diagnostics, with responsibility for guiding a team and transitioning new methods into production.

United States Full-time Lead Data Scientist Research Scientist
$167k-$202k
AWS CI/CD Docker GCP Git Java Kubernetes Machine Learning Python R Shell Scripting
1 week ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers