RemoteLeaf Logo
Log in Sign up

Staff Applied AI Engineer, Product & Agent Performance

1 month, 2 weeks ago
United States
Full-time
Lead
AI Engineer
Artificial Intelligence and Machine Learning
AWS LLM Machine Learning
Apply Now
Arcadia

Arcadia

Arcadia provides a healthcare data platform that enables organizations to unify diverse data sources, derive actionable insights through analytics, and enhance patient outcomes by delivering high-quality care experiences.

IT Services
251-1K
Founded 2002
$154M raised
View All Jobs 14

Description

  • Design and iterate on agent behavior across real, live workflows, including long-horizon and multi-turn tasks.
  • Design retrieval and context architecture so models receive the right source data in the right structure.
  • Design memory and state handling across multi-turn and multi-agent flows.
  • Create prompt and context templates using few-shot examples, structured formatting, and reasoning scaffolding.
  • Improve performance through prompting, tool-use strategy, and context construction using direct experimentation.
  • Build and run production-grounded evaluations to measure performance, regressions, failure modes, and edge cases.
  • Write evaluation rubrics, quality heuristics, and severity-weighted thresholds.
  • Design and validate escalation paths to human review based on confidence and uncertainty.
  • Evaluate and sign off on model changes by comparing baselines and making go/no-go release decisions.
  • Maintain product-level AI documentation, including model cards, intended use, limitations, and known failure modes.
  • Partner with Product and Engineering to ensure agents are steerable, trustworthy, and ready to scale.

Requirements

  • 8+ years of production software engineering experience.
  • 3+ years of hands-on ownership of ML, LLM, or agentic systems in production.
  • Direct experience in healthcare, finance, or another regulated industry.
  • Equivalent practical experience that demonstrates staff-level depth.
  • Hands-on experience with RAG architecture, production-grounded evaluation frameworks, and human-in-the-loop or fallback logic.
  • Ability to diagnose agent failures and attribute fixes to instruction, retrieval, context, or memory design.
  • Working familiarity with AWS AI/ML services, including Bedrock and SageMaker.
  • Evidence-led judgment and the ability to push back on launch decisions.
  • Experience applying AI to healthcare data or workflows where safety, transparency, and calibrated uncertainty matter.
  • Experience with long-horizon, multi-turn, or multi-agent workflows and product-level AI documentation such as model cards.

Benefits

  • Flexible, remote-friendly culture.
  • Employee-driven programs and initiatives for personal and professional development.
  • Opportunity to define how agent performance, safety, and readiness are measured in production healthcare workflows.
  • Meaningful ownership across prompts, context, memory, evaluations, and escalation patterns at product scale.
  • Cross-functional impact translating production evidence into AI improvements across Arcadia’s platform.
  • Mission-driven work focused on improving patient care.
  • Membership in a talented, energized, diverse, and purpose-driven community.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Lingaro

Fullstack Data Scientist - freelancer

Lingaro 5K-10K IT Services

Lead GenAI engineering initiatives by designing, building, evaluating, and governing production-ready AI applications and architectures for enterprise use cases.

Poland Freelance Senior AI Engineer Data Scientist
Apache Spark AWS Azure CI/CD Databricks Docker Generative AI GitHub Actions GitLab Kubernetes Machine Learning Microservices Python
2 hours ago
CI&T

[Job-00023] Senior AI Engineer, Brazil

CI&T 5K-10K Internet Software & Services

CI&T is seeking a Senior AI Engineer to build reliable AI-powered information extraction solutions and enhance a web application using LLMs, document intelligence, and full-stack engineering.

Brazil Full-time Senior AI Engineer Full-stack Engineer
Next.js Python React
2 hours ago
CI&T

[Job 32221] SR. AI Engineer -Foco em Criação e Orquestração de Agentes (Agentic SDLC)

CI&T 5K-10K Internet Software & Services

A CI&T busca um(a) Software Engineer Mid-Senior para arquitetar, construir e entregar soluções de software para clientes, usando agentic SDLC e IA aplicada da arquitetura ao código em produção.

Brazil Full-time Senior AI Engineer
CI/CD Microservices NestJS Node.js TypeScript
2 hours, 15 minutes ago
CareDx

Senior Data & AI Engineer

CareDx 251-1K Pharmaceuticals

CareDx is seeking a Senior Data and AI Engineer to build production data engineering and clinical NLP solutions that convert structured and unstructured EMR data into standardized, reliable transplant-focused clinical data for patient care, research, and business operations.

United States Full-time Senior AI Engineer
$126k-$158k
CI/CD Deep Learning Git HIPAA Machine Learning NLP Pandas Python PyTorch SQL TensorFlow
2 hours, 45 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers