RemoteLeaf Logo
Log in Sign up

Senior AI Software Developer in Test

1 month, 3 weeks ago
Colombia
Full-time
Senior
AI (Artificial Intelligence)
Artificial Intelligence and Machine Learning
CI/CD GitHub Actions Java JavaScript Jenkins JMeter K6 LLM Machine Learning New Relic Python TypeScript
Apply Now
Caseware

Caseware

CaseWare International Inc. provides cutting-edge software solutions for accounting firms, corporations, and governments, enabling users worldwide to work smarter and transform insights into impact.

Internet Software & Services
251-1K
Founded 1988
View All Jobs 12

Description

  • Evolve an AI-first quality strategy for a fast-scaling cloud-native SaaS platform and emerging agentic systems.
  • Integrate AI-enhanced testing into CI/CD pipelines, including predictive flakiness detection, automated test generation, and self-healing scripts.
  • Design deterministic and statistical testing approaches for LLM-based and agentic systems to address hallucinations, prompt injection, bias, drift, and safety risks.
  • Build automated evaluation pipelines and harnesses for correctness, faithfulness, retrieval quality, generation accuracy, tool-calling, planning sequences, and multi-agent flows.
  • Develop and execute test frameworks across the full AI lifecycle, including prompts, datasets, embeddings, model versions, RAG pipelines, and guardrails.
  • Implement red-teaming, bias and fairness checks, compliance mechanisms, and AI quality signals for automated gating and continuous monitoring.
  • Partner with product, data science, AI engineering, and development teams to test AI features and support roadmap delivery.
  • Drive quality metrics and observability, including DORA metrics, test coverage, hallucination rate, context precision, and drift detection.
  • Build dashboards, support A/B testing of models, and monitor post-deployment AI behavior.
  • Mentor SDETs, lead workshops on AI testing best practices, and help define roadmaps and standards for sustainable AI quality assurance.

Requirements

  • 7+ years of experience in Quality Engineering or SDET roles within cloud-native SaaS environments.
  • 2+ years of hands-on experience with AI, ML, or LLM systems.
  • Strong experience with automated testing infrastructure, CI/CD tools such as Jenkins or GitHub Actions, and test pyramid strategies from unit to end-to-end.
  • Full-stack testing experience across frontend, backend, and API layers.
  • Proven experience testing LLMs, AI agents, and RAG pipelines, including risks such as hallucinations, prompt injection, bias, and drift.
  • Proficiency in JavaScript or TypeScript and working knowledge of Python or Java.
  • Experience with AI evaluation frameworks such as Ragas, DeepEval, LangChain, LangSmith, or LangFuse.
  • Knowledge of observability tools such as New Relic, statistical testing methods, red-teaming, and ethical AI practices.
  • Experience with performance, stress, and load testing tools such as k6, JMeter, or BlazeMeter is nice to have.
  • Bachelor's or Master's degree in Computer Science, AI, or a related field; ISTQB AI Testing certification is a plus.
  • Strong English communication and collaboration skills.
  • A strong portfolio, open-source contributions, or relevant case studies are highly regarded.

Benefits

  • Contrato a término indefinido with all legal benefits.
  • Prepaid medicine, life insurance, and funeral assistance.
  • Internet allowance and home office stipend.
  • Competitive compensation above the market average.
  • 100% remote work environment with excellent work-life balance.
  • Budget for training and mentorship from a highly experienced professional.
  • 5 personal time off days per year, plus sick leave top-up to 100% salary from day 3 to 90.
  • Recognition awards, additional paid time off, and vacation upgrades starting at 5 years of service.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

TEST

Ecom AI

TEST 1-10 Industrial Conglomerates

Lifely is hiring an AI-native Marketing, eCommerce & CRO Specialist to improve how the company acquires, converts, and retains customers across website, campaign, and automation work.

Malaysia Australia Full-time Senior AI (Artificial Intelligence) Growth Marketing Manager
CSS E-commerce Email Marketing Facebook Ads Figma Google Ads Hotjar HTML Notion SEO Shopify
1 hour, 22 minutes ago
FutureSight

Co-Founder & CEO - AI Reverse Logistics Orchestration Platform

FutureSight 11-50 Internet Software & Services

FutureSight is seeking a Co-Founder & CEO to lead an AI-native reverse logistics orchestration venture focused on helping commerce and supply chain organizations control returns, recover value, and close the financial loop after the sale.

Canada Lead AI (Artificial Intelligence) Operations Manager
ERP LLM Prototyping
1 hour, 25 minutes ago
CrowdGen by Appen

Text-to-Speech Voice Rater; French (Canada)

CrowdGen by Appen Internet Software & Services

Appen is seeking a remote independent contractor for Project Hogwarts to evaluate Canadian French text-to-speech audio clips and help improve audio quality and naturalness in AI voice interactions.

Canada Contract Entry Level AI (Artificial Intelligence)
$0k-$0k
1 hour, 56 minutes ago
NeuraFlash

Salesforce Technical Architect, Data & AI

NeuraFlash 251-1K IT Services

NeuraFlash, Part of Accenture is hiring a Technical Architect, Data & AI to design and deliver Salesforce Data 360 and Agentforce solutions that turn unified customer data into intelligent, AI-powered client experiences.

Canada Full-time Senior AI (Artificial Intelligence)
CI/CD GCP Generative AI Java JavaScript Python Salesforce Salesforce Apex Snowflake SQL
2 hours, 1 minute ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers