CodeRoad

CodeRoad

CodeRoad is a software development company that specializes in turning ideas into reality. With a team of over 100 experts, we offer a range of services including product management, creative design, development, quality control, and project management...

Internet Software & Services
51-250
Founded 2002

Description

  • Create question templates and Python scripts to test how AI agents and RAG systems solve complex tasks.
  • Evaluate agent performance using metrics such as success rate, tool use accuracy, planning quality, and autonomy.
  • Collaborate with AI engineers to build ground truth and agentic task datasets for benchmarking.
  • Write unit and integration tests for Python-based RESTful APIs and agent endpoints.
  • Run load and bulk tests using tools like Locust or JMeter to assess high-volume performance.
  • Perform rigorous input and output validation to catch prompt injection risks and logic failures.

Requirements

  • Basic understanding of agentic workflows, prompt engineering (ReAct prompts), and LLM orchestration.
  • Familiarity with AI observability tools such as LangSmith, Langfuse, or OpenTelemetry, or a strong desire to learn them.
  • Proficiency in Python 3.10+ for automation and data manipulation.
  • Strong experience testing and validating RESTful APIs and JSON structures.
  • Analytical, breaker mindset with the ability to identify edge cases and AI failure modes.
  • Advanced English (B2/C1) for global collaboration and technical documentation.
  • 1–3 years of experience in Software QA (manual or automation) is preferred.
  • Exposure to Pytest, Playwright, or Selenium is preferred.
  • Exposure to LangChain, LangGraph, or Pydantic AI is preferred.
  • Basic knowledge of Docker and vector databases is preferred.
  • Experience with Locust or JMeter for bulk testing is preferred.

Benefits

  • 100% remote work.
  • Holidays off.
  • Paid time off.
  • Health insurance assistance program.
  • Competitive USD compensation.
  • Strong team culture and collaborative environment.
  • Ongoing training and growth opportunities.

Interested in this position?

Apply directly on the company website

Apply Now

Similar Roles

Senior Product Quality Analyst - AI Voice

Spotify Media

Spotify’s Personalization team is hiring a Senior Product Quality Analyst in London to define and measure the quality of AI voice experiences across its products.

Generative AI
1 hour, 13 minutes ago

Director, Nuclear Project Quality Assurance

ELEMENTL POWER Renewable Electricity

Elementl Power Inc. is hiring a Director, Nuclear Project Quality Assurance to lead quality assurance across its advanced nuclear project portfolio, with primary focus on development-phase licensing, supplier oversight, and NRC readiness.

1 hour, 13 minutes ago

AI Evaluation Engineer (Python, QA or Security)

Mindrift.ai: Be the “I” in AI Internet Software & Services

Mindrift is seeking a project-based software developer to help build and evaluate realistic coding tasks for AI agents in simulated development environments.

Docker FastAPI JavaScript Kafka PostgreSQL Python React Redis TypeScript
1 hour, 13 minutes ago

Senior Product Quality Analyst - AI Voice

Spotify Media

Spotify’s Personalization team is hiring a Senior Product Quality Analyst to define and measure the quality of AI voice experiences across products, languages, and markets.

Generative AI Machine Learning
1 hour, 13 minutes ago

You're on a roll! Sign up now to keep applying.

Sign Up

Already have an account? Log in

Used by 14,729+ remote workers