RemoteLeaf Logo
Log in Sign up

Principal Data Scientist - Agent Builder

6 hours, 7 minutes ago
Spain
Full-time
Lead
Data Scientist
Data Science and Analytics
Elasticsearch LLM Machine Learning NLP Pandas Python PyTorch Transformers
Apply Now
Elastic

Elastic

Elastic is a leading platform for search-powered solutions, providing real-time insights and making data usable for developers and enterprises worldwide.

Internet Software & Services
1K-5K
Founded 2010
View All Jobs 108

Description

  • Define the evaluation strategy for conversational and agentic search, including offline and online evaluation, golden datasets, rubrics, LLM-as-judge calibration, groundedness checks, citation checks, and A/B testing.
  • Lead the design of quality metrics and decision frameworks for RAG, agents, tools, model selection, agent routing, prompt behavior, and cost/latency trade-offs.
  • Build, compare, and improve retrieval and re-ranking approaches, including sparse and dense retrieval, vector search, query understanding, semantic rewrites, and context enrichment.
  • Translate experimental results into product and business decisions about model choice, request routing, tool exposure, and agent customization for different Elastic use cases.
  • Partner with engineering to productionize evaluation pipelines, telemetry, dashboards, CI guardrails, and regression detection for chat quality, helpfulness, latency, and cost.
  • Influence roadmap direction by identifying high-leverage quality gaps, proposing practical solutions, and communicating trade-offs to product, engineering, and leadership.
  • Mentor data scientists and engineers in experiment design, evaluation methodology, statistical rigor, and approaches to improving LLM-powered systems.
  • Share findings through documentation, notebooks, pull requests, dashboards, technical proposals, and cross-functional reviews.

Requirements

  • 8+ years of applied data science or machine learning experience in IR, NLP, ranking, semantic search, RAG, or LLM-powered product experiences.
  • Strong track record leading evaluation for production AI/ML systems, including offline metrics, online experimentation, LLM-as-judge methods, groundedness, citation quality, and model comparison.
  • Experience influencing product and technical strategy through data in ambiguous or emerging domains.
  • Hands-on ability with Python, PyTorch/Transformers, Pandas, notebooks, reproducible experiments, versioned datasets, and clean, reviewable code.
  • Strong understanding of retrieval systems, including dense and sparse retrieval, re-ranking, vector search, query understanding, and evaluation metrics such as nDCG, MRR, Recall@k, precision, and latency/cost trade-offs.
  • Experience collaborating with engineering teams to move from prototype to production, including telemetry design, dashboards, CI guardrails, and quality regression tracking.
  • Practical Elasticsearch experience, or experience with similar search and distributed data systems; ES|QL familiarity is a plus.
  • Excellent written and verbal communication skills, with the ability to explain complex scientific and technical trade-offs to engineering, product, design, and leadership audiences.
  • A collaborative, low-ego style with a strong ability to mentor, raise standards, and improve transparency in a distributed team.

Benefits

  • Starting salary range of €80,400 to €127,200 EUR.
  • Base salary compensation with no variable compensation component.
  • Competitive pay based on the work you do rather than your previous salary.
  • Health coverage for you and your family in many locations.
  • Flexible locations and schedules for many roles.
  • Generous number of vacation days each year.
  • Up to 40 hours each year for volunteer projects you love.
  • Minimum of 16 weeks of parental leave.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Adswerve

Data Scientist 1

Adswerve 251-1K Media

Adswerve is hiring a Data Scientist 1 for its Tech Services team to work on client-facing data engineering and analytics solutions that improve marketing performance and business outcomes.

United States Full-time Entry Level Data Scientist
$95k-$115k
AWS Azure GCP Google Ads Google Analytics HTML JavaScript Machine Learning Python SQL
6 hours, 7 minutes ago
Adswerve

Sr. Data Scientist 3 (South Africa)

Adswerve 251-1K Media

Adswerve is seeking a Senior Data Scientist 3 to join its Technical Services team, leading client-facing data science and data engineering work that turns complex marketing data into actionable business solutions.

South Africa Full-time Lead Data Scientist
AWS Azure GCP Generative AI Google Analytics HTML JavaScript Machine Learning Python SQL Statistics
6 hours, 52 minutes ago
American Institutes for Research

Senior Economist, Healthcare Innovations

American Institutes for Research 1K-5K Professional Services

AIR is seeking a Senior Economist to join its Healthcare Innovations team, conducting policy-related research and evaluation studies on U.S. health care programs and public health policy.

United States Full-time Senior Data Scientist Program Manager
$130k-$160k
1 day, 6 hours ago
Pinterest

Staff Data Scientist, Ads Product

Pinterest 5K-10K Internet Software & Services

Pinterest is hiring a senior individual contributor to lead data-driven strategy and decision support for its Ads organization, connecting product, engineering, business, and finance teams around reliable insights and analytical excellence.

United States Full-time Lead Data Scientist
$165k-$339k
Looker Machine Learning Python SQL Statistics Tableau
1 day, 6 hours ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers