Machine Learning Engineer, Evaluation & Reliability

4 days, 19 hours ago
Full-time
Mid Level
Software Development
Bask Health

Bask Health

Bask Health is the compliant, accessible, and no-code platform for telehealth, providing EMR, payment processing, pharmacy fulfillment, website builder, and patient management system. It's the Shopify for E-Prescribing.

Internet Software & Services
1-10

Description

  • Build and maintain evaluation systems for AI-powered applications across Bask Health.
  • Develop automated testing frameworks, benchmarks, and regression tests for LLMs and AI workflows.
  • Create and maintain representative evaluation datasets, test scenarios, and model-response review tools.
  • Measure model quality, including accuracy, relevance, completeness, consistency, hallucinations, and unsupported claims.
  • Evaluate retrieval-augmented generation, structured outputs, tool calling, agent actions, and schema compliance.
  • Test AI workflows for unsafe behavior, unauthorized actions, incorrect permissions, and exposure of sensitive information.
  • Build quality dashboards, reports, monitoring, and alerting for production AI performance.
  • Analyze evaluation results and production incidents to identify failure patterns and drive engineering improvements.
  • Collaborate with engineering and product teams to define measurable AI acceptance criteria and improve prompts, retrieval logic, and application behavior.
  • Own evaluation and reliability projects from implementation through deployment and continuous improvement.

Requirements

  • 3+ years of professional experience in machine learning engineering, software engineering, applied AI, or a related field.
  • Experience building and maintaining production software or machine learning applications.
  • Strong programming skills in Python, TypeScript, or comparable languages.
  • Experience testing or evaluating machine learning models or AI-powered software.
  • Understanding of machine learning evaluation metrics, experimental methodologies, and regression testing.
  • Experience developing automated tests, validation pipelines, and data-processing scripts or services.
  • Familiarity with large language models, prompt engineering, model-output analysis, and LLM failure modes.
  • Experience analyzing structured and unstructured datasets, relational databases, or analytical data systems.
  • Familiarity with APIs, backend development, cloud deployment environments, logging, monitoring, and production troubleshooting.
  • Experience designing representative test cases, identifying model-failure patterns, communicating results, and independently owning cross-functional technical projects.
  • Experience with LLM evaluation frameworks such as LangSmith, DeepEval, or Ragas; LLM-as-a-judge techniques; or human-labeled datasets (preferred).
  • Experience evaluating RAG systems, embeddings, vector databases, AI agents, tool-calling workflows, prompt injection, or AI systems in healthcare and other sensitive environments (preferred).

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Senior Machine Learning Systems Engineer (US)

PointClickCare 1001-5000 Health Care Providers & Services

PointClickCare is seeking a Senior Machine Learning Systems Engineer to build and operate the secure, scalable ML platform powering AI products across its healthcare technology organization.

AWS Azure CI/CD Databricks Docker GCP Java Kubeflow Kubernetes LLM Machine Learning MLflow MLOps Network Security Python Serverless
1 day, 19 hours ago

Staff Machine Learning Engineer, Home Surfaces

Spotify Media

As a Staff Machine Learning Engineer at Spotify, you will lead the development of production-scale podcast recommendation systems and AI-powered discovery experiences for listeners worldwide.

Apache Airflow GCP LLM Machine Learning Python PyTorch
2 days, 18 hours ago

Senior AI/ML Engineer (GenAI, AWS)

Provectus 501-1000 Professional Services

Provectus is hiring a Senior AI Engineer to partner with forward-deployed teams and deliver production-grade agentic AI systems for enterprise clients in financial services, insurance, healthcare, and life sciences.

Apache Airflow Apache Spark AWS Azure CI/CD GCP Generative AI GitHub Actions GitLab CI Go Kafka Kubernetes LLM Machine Learning NLP Python REST API Rust TypeScript
2 days, 18 hours ago

Senior AI/ML Engineer (GenAI, AWS)

Provectus 501-1000 Professional Services

Provectus is seeking a Senior AI Engineer to partner with forward-deployed teams in delivering production-grade agentic AI, LLM, and RAG solutions for enterprise clients in financial services, insurance, healthcare, and life sciences.

Apache Airflow Apache Spark AWS Azure CI/CD GCP Generative AI GitHub Actions GitLab CI Go Kafka Kubernetes LLM Machine Learning NLP Python REST API Rust TypeScript
2 days, 18 hours ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers