RemoteLeaf Logo
Log in Sign up

Senior AI Engineer - Observability

10 hours, 10 minutes ago
United States
Full-time
Senior
AI Engineer
Software Development
Azure C# CI/CD Git Machine Learning Microservices .NET OpenTelemetry Python
Apply Now
OnBoard

OnBoard

OnBoard offers AI-powered board management software designed to simplify governance, enhance decision-making, and streamline the processes of board meetings for leadership teams.

Internet Software & Services
251-1K
Founded 2003
$105M raised
View All Jobs 3

Description

  • Partner with product engineers to design, build, and improve LLM, RAG, semantic search, and agentic AI features.
  • Contribute production code in Python, C#/.NET, or related technologies.
  • Improve prompts, retrieval, context assembly, ranking, grounding, and response-generation patterns.
  • Design evaluation pipelines, scoring methodologies, quality gates, and regression tests for AI features.
  • Build and maintain versioned datasets covering real-world use cases, edge cases, failure modes, and critical workflows.
  • Instrument LLM interactions, RAG pipelines, tool calls, and agent workflows using observability platforms.
  • Monitor and analyze latency, token usage, cost, retrieval quality, groundedness, safety signals, failures, and user feedback.
  • Build dashboards, alerts, runbooks, reusable libraries, SDKs, and standards for AI observability and evaluation.
  • Coach product teams on AI quality, evaluation design, observability, and reliable release practices.
  • Support secure, compliant, auditable AI delivery aligned with privacy, security, SOC 2, ISO 27001, and data-residency requirements.

Requirements

  • 5+ years of software engineering experience building production systems.
  • Hands-on experience building or operating LLM-powered features, RAG systems, AI workflows, or similar applications.
  • Strong engineering skills in Python, C#/.NET, or both.
  • Practical knowledge of prompts, embeddings, vector search, retrieval quality, orchestration, and LLM application architecture.
  • Experience designing evaluations, quality metrics, or regression frameworks and assessing whether they measure meaningful outcomes.
  • Strong observability fundamentals, including tracing, logging, metrics, alerting, and production debugging.
  • Experience with CI/CD, Git workflows, cloud environments, and production releases; Azure DevOps preferred.
  • Ability to communicate effectively with engineering, product, QA, security, and business stakeholders.
  • Proficiency with AI-assisted development tools such as Claude Code or PlayerZero.
  • Preferred experience with LLMOps and observability tools, vector databases, AI frameworks, evaluation platforms, experimentation, regulated environments, QA, ML, or data engineering.

Benefits

  • Fully remote work within the United States with company-provided equipment.
  • Medical, prescription, dental, and vision insurance options.
  • HSA employer contributions and medical and dependent-care flexible spending accounts.
  • $50,000 or one-times-salary basic life insurance, plus disability and accidental death coverage.
  • 401(k) plan with automatic enrollment and up to a 4% company match.
  • Paid time off and holidays.
  • International employee benefits vary by location, including retirement contributions, private medical coverage, insurance, and employee assistance programs.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Provectus

Python/GenAI Solutions Architect

Provectus 251-1K Professional Services

Provectus is seeking a Senior Software Engineer or Solutions Architect to embed with enterprise client teams and deliver production-ready Python, cloud, and generative AI systems from discovery through deployment.

United States Full-time Senior AI Engineer Solutions Architect
AWS CI/CD Django Docker FastAPI Flask GCP Generative AI GitHub Actions GitLab CI Go Kubernetes Machine Learning Microservices Node.js Pytest Python React REST API Rust SQL Vue.js
9 hours, 55 minutes ago
CI&T

[Job - 31015] Master Software Developer ( AI Generative)

CI&T 5K-10K Internet Software & Services

CI&T is seeking a Master Software Developer to design, build, and scale secure, production-grade Generative AI and Agentic AI solutions for enterprise clients in a global delivery environment.

Colombia Brazil Full-time Senior AI Engineer
AWS Azure CI/CD Databricks Docker Generative AI Kubernetes MLOps Python Secrets Management Vertex AI
10 hours, 10 minutes ago
Smart Working

LLM Systems / AI Agent Engineer (Remote, Full-Time) [AS311]

Smart Working Internet Software & Services

Smart Working is hiring an LLM Systems / AI Agent Engineer to build and operate production AI agents and their orchestration, evaluation, observability, and monitoring systems in a remote-first product team.

India Full-time Senior AI Engineer
AWS Node.js OpenTelemetry PostgreSQL Serverless TypeScript Vertex AI
10 hours, 10 minutes ago
CI&T

[Job-31101] Mid Level AI Developer | Fullstack (Java / Angular), Brazil

CI&T 5K-10K Internet Software & Services

A CI&T busca uma pessoa desenvolvedora Full Stack para criar e evoluir aplicações com Java/Spring Boot e Angular, incorporando IA e práticas modernas de desenvolvimento em soluções escaláveis e seguras.

Brazil Full-time Mid Level AI Engineer Full-stack Engineer
Angular GitLab Java Jenkins JIRA SonarQube Spring Boot
10 hours, 25 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers