Staff Applied AI Engineer, Product & Agent Performance

21 hours, 55 minutes ago
Full-time
Lead
Software Development
Arcadia

Arcadia

Arcadia provides a healthcare data platform that enables organizations to unify diverse data sources, derive actionable insights through analytics, and enhance patient outcomes by delivering high-quality care experiences.

IT Services
501-1000
Founded 2002
$154M raised

Description

  • Design and iterate on agent behavior for live, long-horizon, multi-turn workflows.
  • Design retrieval, context, memory, and state architectures that ground agents in appropriate source data.
  • Create prompt and context templates using examples, structured formatting, and reasoning scaffolding.
  • Improve agent accuracy, reliability, latency, steerability, and cost through experimentation with prompting, tool use, and context construction.
  • Build and run production-representative evaluations to measure performance, regressions, failure modes, and edge cases.
  • Develop severity- and cost-weighted evaluation rubrics, quality heuristics, thresholds, and production monitoring.
  • Design and validate confidence-based human-review and escalation paths, including adversarial and edge-case testing.
  • Baseline, comparatively evaluate, and approve or reject model changes before customer release.
  • Maintain model cards, intended-use guidance, limitations, and known-failure documentation.
  • Partner with Product and Engineering to translate production evidence into scalable AI improvements.

Requirements

  • Bachelor’s degree in Engineering or related field and/or a Master’s degree in Data Science or related field.
  • 4+ years of production software engineering experience.
  • 3+ years of hands-on ownership of production ML, LLM, or agentic systems.
  • Direct experience in healthcare, finance, or another regulated industry.
  • Ability to diagnose agent failures and determine whether instruction, retrieval, context, or memory design is the appropriate fix.
  • Experience prioritizing failures by severity and cost, not frequency alone.
  • Hands-on experience with RAG architecture, production-grounded evaluation frameworks, and fallback or human-in-the-loop logic.
  • Working familiarity with AWS AI/ML services, including Bedrock and SageMaker.
  • Evidence-led judgment to challenge launch decisions and run experiments from production failure through resolution.
  • Experience applying AI to healthcare workflows, multi-turn or multi-agent systems, or product-level AI documentation such as model cards (preferred).

Benefits

  • Meaningful ownership of agent performance, safety, and readiness for production healthcare workflows.
  • Cross-functional role influencing AI capabilities across Arcadia’s platform.
  • Flexible, remote-friendly work culture.
  • Employee-driven personal and professional development programs.
  • Mission-driven work focused on improving patient care.
  • Membership in a diverse, purpose-driven employee community.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Microsoft Copilot Studio Subject Matter Expert (SME)

TrueTandem 51-200 Internet Software & Services

TrueTandem is seeking a Microsoft Copilot Studio Subject Matter Expert to support Government-led initiatives by developing, deploying, and transferring knowledge on Copilot Studio and Power Platform solutions.

Agile Azure Microsoft Dynamics 365 REST API SQL Server
21 hours, 25 minutes ago

Senior AI Engineer

Egen.ai IT Services

Egen is seeking a Senior AI Engineer to design, deploy, and optimize production-grade generative AI, document intelligence, and RAG solutions for client needs on Google Cloud.

GCP Generative AI LLM Machine Learning MLOps Neural Networks Python Salesforce Shell Scripting SQL Vertex AI
21 hours, 40 minutes ago

Principal AI Engineer

Egen.ai IT Services

Egen is seeking a hands-on Principal AI Engineer to lead the architecture and delivery of scalable, production-grade generative AI and agentic systems for client engagements.

GCP Generative AI GPT Machine Learning MLOps Neural Networks Python Salesforce Shell Scripting SQL Vertex AI
21 hours, 40 minutes ago

Lead AI Engineer

Egen.ai IT Services

Egen is seeking a Lead AI Engineer to lead the design, deployment, and continuous improvement of production-grade generative AI, document intelligence, and RAG solutions for clients.

GCP Generative AI LLM Machine Learning MLOps Neural Networks Python Salesforce Shell Scripting SQL Vertex AI
21 hours, 40 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers