Senior Software QA Engineer

The Role

What are the main responsibilities for this role?

• Design & Build: Develop, maintain, and scale robust, end-to-end automation test frameworks from scratch (UI, API, and Integration layers).
• CI/CD Integration: Seamlessly bake automated test suites into our CI/CD pipelines to ensure rapid, high-confidence deployments.
• Coverage Strategy: Define, track, and scale test coverage metrics (targeting high-purity API and critical UI path coverage) to ensure regression-free deployments.
• Performance & Scale: Conduct API performance, load, and stress testing to ensure our infrastructure keeps up with demand.
• AI/LLM Validation: Design testing strategies for non-deterministic systems. You’ll evaluate LLM outputs for accuracy, bias, safety, and hallucination rates.
• Prompt Regression Testing: Establish benchmarks to ensure that tweaking an AI prompt or upgrading a foundational model doesn't secretly break existing features.
• Data & Model Drift Monitoring: Collaborate with Data Scientists and ML Engineers to monitor model performance metrics in production.
• Edge Case Hunting: Actively simulate adversarial inputs (prompt injections, jailbreaks) to ensure our AI features remain secure and guardrailed.

What skills and experience are required?

• 4+ years of experience in Software QA Automation, with at least 1–2 years specifically focused on testing AI/ML-powered applications or LLM integrations.
• Strong programming proficiency in Python (highly preferred for AI workflows) or JavaScript/TypeScript.
• Hands-on experience with modern automation tools (e.g., Playwright, Cypress, Selenium).
• Expertise in API testing tools (e.g., Postman, RestAssured, PyTest).
• Familiarity with AI evaluation frameworks (e.g., Ragas, TruLens, Deepchecks) or a proven ability to write custom evaluation scripts for probabilistic outputs.
• Basic understanding of vector databases, embeddings, and how data pipelines feed into ML models.
• Bachelor’s degree in Computer Science, Engineering, or equivalent practical experience.

Bonus Points

• Familiarity with cloud platforms (AWS, GCP, or Azure) and containerization (Docker, Kubernetes).
• A healthy obsession with AI safety and ethics.

Why work for Crimson?

• Rapidly growing start up, with a flexible working environment where you will be empowered to structure how you work
• Limitless development and exposure - our internal promotions/role changes made up 33% of all recruitment last year
• $1000 individual training budget per year, we love to ‘Level Up’ (it’s one of our core values)!
• Psychologist on staff
• Insightful fireside chats and workshops to help support our high-performing and ambitious team
• Radical Candor is a feedback approach we live by
• We’re a global player with 28 markets (and growing) across the globe. Most roles have the option to work from one of our many offices or remotely!

If you're passionate about growing in a fast-paced, collaborative environment and want to work with cutting-edge technology, then we'd love to hear from you!