RemoteLeaf Logo
Log in Sign up

Staff Applied AI Engineer, Product & Agent Performance

4 weeks, 1 day ago
United States
Full-time
Lead
AI Engineer
Artificial Intelligence and Machine Learning
AWS LLM Machine Learning
Apply Now
Arcadia

Arcadia

Arcadia provides a healthcare data platform that enables organizations to unify diverse data sources, derive actionable insights through analytics, and enhance patient outcomes by delivering high-quality care experiences.

IT Services
251-1K
Founded 2002
$154M raised
View All Jobs 14

Description

  • Design and iterate on agent behavior across real, live workflows, including long-horizon and multi-turn tasks.
  • Design retrieval and context architecture so models receive the right source data in the right structure.
  • Design memory and state handling across multi-turn and multi-agent flows.
  • Create prompt and context templates using few-shot examples, structured formatting, and reasoning scaffolding.
  • Improve performance through prompting, tool-use strategy, and context construction using direct experimentation.
  • Build and run production-grounded evaluations to measure performance, regressions, failure modes, and edge cases.
  • Write evaluation rubrics, quality heuristics, and severity-weighted thresholds.
  • Design and validate escalation paths to human review based on confidence and uncertainty.
  • Evaluate and sign off on model changes by comparing baselines and making go/no-go release decisions.
  • Maintain product-level AI documentation, including model cards, intended use, limitations, and known failure modes.
  • Partner with Product and Engineering to ensure agents are steerable, trustworthy, and ready to scale.

Requirements

  • 8+ years of production software engineering experience.
  • 3+ years of hands-on ownership of ML, LLM, or agentic systems in production.
  • Direct experience in healthcare, finance, or another regulated industry.
  • Equivalent practical experience that demonstrates staff-level depth.
  • Hands-on experience with RAG architecture, production-grounded evaluation frameworks, and human-in-the-loop or fallback logic.
  • Ability to diagnose agent failures and attribute fixes to instruction, retrieval, context, or memory design.
  • Working familiarity with AWS AI/ML services, including Bedrock and SageMaker.
  • Evidence-led judgment and the ability to push back on launch decisions.
  • Experience applying AI to healthcare data or workflows where safety, transparency, and calibrated uncertainty matter.
  • Experience with long-horizon, multi-turn, or multi-agent workflows and product-level AI documentation such as model cards.

Benefits

  • Flexible, remote-friendly culture.
  • Employee-driven programs and initiatives for personal and professional development.
  • Opportunity to define how agent performance, safety, and readiness are measured in production healthcare workflows.
  • Meaningful ownership across prompts, context, memory, evaluations, and escalation patterns at product scale.
  • Cross-functional impact translating production evidence into AI improvements across Arcadia’s platform.
  • Mission-driven work focused on improving patient care.
  • Membership in a talented, energized, diverse, and purpose-driven community.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Caseware

Senior AI Software Engineer

Caseware 251-1K Internet Software & Services

Caseware is hiring a fully remote AI Platform Architect in Colombia to define, build, and operate scalable, secure, and trustworthy AI capabilities for its global accounting and audit software platform.

Colombia Full-time Lead AI Engineer
AWS AWS CDK Confluence DynamoDB GitHub GitHub Actions JIRA Microservices NestJS Python Terraform TypeScript
8 hours, 21 minutes ago
Agiloft

AI Ops Engineer GTM

Agiloft 51-250 Capital Markets

Agiloft is hiring an AI Ops Engineer, GTM to shape and deliver practical AI solutions across Sales, Marketing, Customer Success, and Revenue Operations, using customer and market insight to improve productivity, revenue outcomes, and customer experience.

United States Full-time Junior AI Engineer Sales Operations
CRM HubSpot Python Salesforce Snowflake
8 hours, 21 minutes ago
CI&T

[Job-31694] Tech Lead AI Engineer | Agentic SDLC, Brazil

CI&T 5K-10K Internet Software & Services

A CI&T busca uma pessoa Tech Lead – AI Engineer/Orchestrator para liderar tecnicamente soluções de software em uma operação estratégica do setor financeiro, conectando arquitetura, desenvolvimento e agentes de IA em um ambiente de Agentic SDLC.

Brazil Full-time Lead AI Engineer Technical Lead
API Gateway AWS CI/CD DevSecOps Generative AI Java Microservices REST API
8 hours, 21 minutes ago
Pavago

AI Systems Engineer (LLM & AI Agent Development)

Pavago IT Services

The company is seeking a remote AI Systems Engineer to build, integrate, and scale production AI applications, autonomous agents, and internal software systems in collaboration with the founder.

Philippines Full-time Senior AI Engineer
HubSpot Java Node.js Python Salesforce
8 hours, 36 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers