RemoteLeaf Logo
Log in Sign up

Quality Analyst II

1 hour, 25 minutes ago
United States
Full-time
Junior
AI (Artificial Intelligence)
Artificial Intelligence and Machine Learning
LLM SQL
Apply Now
HighLevel

HighLevel

HighLevel provides an all-in-one sales and marketing platform that agencies can white label and resell, offering tools and resources designed to help businesses consolidate their marketing efforts and achieve their growth objectives.

Internet Software & Services
251-1K
Founded 2018
$60M raised
View All Jobs 78

Description

  • Evaluate AI output across the full advisor call lifecycle, including pre-call briefs, in-call guidance, post-call outputs and transcription-driven follow-ups.
  • Classify failure modes and document patterns using standardized quality rubrics.
  • Participate in calibration sessions to maintain consistent scoring across reviewers.
  • Refine prompts, update knowledge base entries and tweak existing skills and orchestrations inside ORA.
  • Test changes, validate improvements and roll out updates within the platform.
  • Maintain logs of issue types, trends and conversation behaviors.
  • Surface recurring patterns and systemic gaps to leadership and the Lead Analyst.
  • Build reporting that gives Success leadership visibility into AI quality and the improvement backlog.
  • Partner with RevOps Engineering and the Lead Analyst on platform changes, end-to-end builds and quality criteria.
  • Stay current on LLM behavior, prompt patterns and AI evaluation techniques, and share learnings with the team.

Requirements

  • 2+ years of experience in Quality Assurance, Customer Support, AI evaluation or a related role.
  • Working understanding of how LLMs behave, including prompt design, common failure modes and conversational AI patterns.
  • General understanding of databases and how data flows between systems.
  • Comfort reading code and making minor edits to support integrations and prompt debugging.
  • Strong written communication and attention to detail.
  • Ability to follow structured guidelines and scoring rubrics consistently.
  • Comfort working with evolving tools and AI-assisted workflows.
  • Self-starter mindset with the ability to manage review volume and prioritize work with minimal direction.
  • Bachelor’s degree in a related field or equivalent practical experience.
  • Hands-on experience with AI development platforms such as Claude, Cursor or similar tools (preferred).
  • Familiarity with retrieval-augmented generation (RAG), vector databases or knowledge-base architecture (preferred).
  • Familiarity with SQL or similar database querying (preferred).
  • Experience with APIs, webhooks or data integration between systems (preferred).
  • Background in customer success, support, sales enablement or call center environments (preferred).
  • Experience with the HighLevel platform or comparable SaaS products (preferred).
  • Prior experience evaluating AI-generated conversations, chatbots or automated support output (preferred).

Benefits

  • Remote-first work environment.
  • Opportunity to work on an AI-powered platform serving businesses in 150+ countries.
  • Growth path from evaluator-focused work into deeper creation work with the Lead Analyst.
  • Equal Opportunity Employer status with privacy-protected demographic data collection.
  • Use of AI tools in the hiring process with final decisions made by humans.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

INFUSE

Production AI Innovation Lead (Contract, Remote)

INFUSE 251-1K Professional Services

INFUSE Academy is seeking a design-and-AI-focused learning experience creator to develop and scale new production workflows for course and asset creation.

Ukraine Contract Senior AI (Artificial Intelligence) Design Lead
Figma Illustrator Photoshop
10 minutes ago
INFUSE

Production AI Innovation Lead (Contract, Remote)

INFUSE 251-1K Professional Services

INFUSE Academy is hiring a learning experience designer to create and scale AI-informed workflows for producing courses and learning assets.

Portugal Contract Senior AI (Artificial Intelligence) Design Lead
Figma Illustrator Photoshop
25 minutes ago
On.Energy

Director of Program Management - AI Enablement

On.Energy 51-250 Electric Utilities

ON.energy is hiring a Director of Program Management, AI Enablement to build and run its enterprise AI adoption program, helping the company turn AI use into measurable business impact across the organization.

United States Full-time Lead AI (Artificial Intelligence) Program Manager
ERP
25 minutes ago
INFUSE

Production AI Innovation Lead (Contract, Remote)

INFUSE 251-1K Professional Services

INFUSE Academy is seeking a design and AI-focused learning experience creator to build, test, and scale new workflows for producing courses and learning assets.

Moldova Contract Lead AI (Artificial Intelligence) Design Lead
Figma
40 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers