RemoteLeaf Logo
Log in Sign up

Staff Applied AI Engineer, Product & Agent Performance

48 minutes ago
United States
Full-time
Lead
AI Engineer
Artificial Intelligence and Machine Learning
AWS LLM Machine Learning
Apply Now
Arcadia

Arcadia

Arcadia provides a healthcare data platform that enables organizations to unify diverse data sources, derive actionable insights through analytics, and enhance patient outcomes by delivering high-quality care experiences.

IT Services
251-1K
Founded 2002
$154M raised
View All Jobs 16

Description

  • Design and iterate on agent behavior across real, live workflows, including long-horizon and multi-turn tasks.
  • Design retrieval and context architecture so models receive the right source data in the right structure.
  • Design memory and state handling across multi-turn and multi-agent flows.
  • Create prompt and context templates using few-shot examples, structured formatting, and reasoning scaffolding.
  • Improve performance through prompting, tool-use strategy, and context construction using direct experimentation.
  • Build and run production-grounded evaluations to measure performance, regressions, failure modes, and edge cases.
  • Write evaluation rubrics, quality heuristics, and severity-weighted thresholds.
  • Design and validate escalation paths to human review based on confidence and uncertainty.
  • Evaluate and sign off on model changes by comparing baselines and making go/no-go release decisions.
  • Maintain product-level AI documentation, including model cards, intended use, limitations, and known failure modes.
  • Partner with Product and Engineering to ensure agents are steerable, trustworthy, and ready to scale.

Requirements

  • 8+ years of production software engineering experience.
  • 3+ years of hands-on ownership of ML, LLM, or agentic systems in production.
  • Direct experience in healthcare, finance, or another regulated industry.
  • Equivalent practical experience that demonstrates staff-level depth.
  • Hands-on experience with RAG architecture, production-grounded evaluation frameworks, and human-in-the-loop or fallback logic.
  • Ability to diagnose agent failures and attribute fixes to instruction, retrieval, context, or memory design.
  • Working familiarity with AWS AI/ML services, including Bedrock and SageMaker.
  • Evidence-led judgment and the ability to push back on launch decisions.
  • Experience applying AI to healthcare data or workflows where safety, transparency, and calibrated uncertainty matter.
  • Experience with long-horizon, multi-turn, or multi-agent workflows and product-level AI documentation such as model cards.

Benefits

  • Flexible, remote-friendly culture.
  • Employee-driven programs and initiatives for personal and professional development.
  • Opportunity to define how agent performance, safety, and readiness are measured in production healthcare workflows.
  • Meaningful ownership across prompts, context, memory, evaluations, and escalation patterns at product scale.
  • Cross-functional impact translating production evidence into AI improvements across Arcadia’s platform.
  • Mission-driven work focused on improving patient care.
  • Membership in a talented, energized, diverse, and purpose-driven community.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

EQT

Land Data Scientist II/III/Senior

EQT 251-1K Gas Utilities

EQT is hiring a Land Data Scientist to support its Land Department by translating business needs into governed analytics and AI solutions that improve operations and decision-making.

United States Full-time Senior AI Engineer Data Scientist
Apache Spark Databricks Git Machine Learning R Salesforce SQL
18 minutes ago
Tatari

Senior Product Manager - Upstream

Tatari 251-1K Media

Tatari is hiring a Senior Product Manager to own technical supply-path and optimization products for its TV advertising platform, working with engineering, data science, and streaming publishers to improve live campaign performance.

Poland Europe Ukraine Full-time Mid Level Technical Product Manager
Agile Confluence JIRA Machine Learning
33 minutes ago
Provectus

FDE AI/ Solutions Architect (AI, Python/Data)

Provectus 251-1K Professional Services

Provectus is hiring a Forward Deployed Engineer to embed with enterprise clients and turn AI blueprints into production systems that improve business outcomes in regulated industries.

Moldova Montenegro North Macedonia Bosnia and Herzegovina Bulgaria Croatia Czech Republic Full-time Senior AI Engineer Solutions Architect
Apache Airflow Apache Spark AWS Azure CI/CD GCP Generative AI GitHub Actions GitLab CI Go Kafka Kubernetes Machine Learning Python REST API Rust TypeScript
48 minutes ago
CI&T

[Job-31203] AI Engineer Master, Brazil

CI&T 5K-10K Internet Software & Services

CI&T is seeking a hands-on Senior Technical Architect to build and own a client’s AWS enterprise cloud foundation and Agentic-SDLC delivery system while leading architecture, engineering, and client-facing technical decisions.

Brazil Full-time Lead AI Engineer Software Architect
API Gateway AWS AWS CDK DNS GitHub Actions Java Microservices OpenID Connect React REST API TypeScript
48 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers