Beyond test automation. The next chapter in QA is written by AI
The age of predictable software is over – welcome to testing systems that learn, adapt, and sometimes surprise even their makers.
Back in 2009, everyone in QA circles was talking about automation as if it were the end of the profession. I still remember the nervous energy at meet-ups, with manual testers whispering that scripts would make them redundant. In truth, the opposite happened. The more we automated, the more we built, the more testing there was to do. And now, here we are again, standing on another edge, this time looking out at a landscape powered by Artificial Intelligence.
The difference is that this one is much bigger.
Predictable vs Probabilistic
Traditional software is like a recipe: follow the steps and you’ll get the same dish every time. AI systems are more like chefs with opinions. Give them the same ingredients twice and you may get two slightly different results. That unpredictability is what makes testing AI both thrilling and mildly terrifying in equal measure.
As a tester, you’re no longer just verifying whether the “Save” button works. You’re exploring how the system thinks. The challenge has shifted from asking “Does it do what it should?” to asking “Why did it do that?”
OpenAI’s GPT-4 Technical Report (2024) shows how subtle prompt tweaks can produce wildly different behaviours, and DeepMind’s research on language-model evaluation dives into the complexity of measuring reliability and safety. For testers, this is fresh territory that demands curiosity as much as it does rigour.
A bit like being Sherlock Holmes again
When I first started in QA, testing felt like detective work: a hunt for patterns, a bit of logic, a dash of intuition. Somewhere along the way, with automation and tooling and process, we made it more mechanical. AI brings the mystery back.
Testing an LLM-powered feature is like investigating a suspect who keeps changing their story. It’s not because they are lying but because context reshapes their answers. The fun part is designing tests that understand that behaviour, then deciding what “acceptable” even means.
A personal detour – From QA to PM
When I moved into product management, I discovered that my QA instincts never really left me; they simply evolved. One of my interesting recent projects was the MVP for “Generating Test Cases with AI” at TestRail. I approached it in the same way I used to approach testing, questioning assumptions, anticipating edge cases, and thinking about how users would trust and validate what the AI produced.
Working on that PRD felt like testing in disguise. Every requirement was a hypothesis to challenge. Every user scenario needed to be grounded in real QA behaviour, not just product ambition. When the feature came to life, I could not resist testing the generator myself. Watching it create different test cases from the same input reminded me that AI testing is not about pass or fail. It is about understanding patterns, behaviours and probabilities. The analytical curiosity we develop in QA, the instinct to explore how something behaves rather than simply confirm that it behaves, is exactly what this new era demands.
A new horizon, not a sunset
Some people say that AI will replace testers. In some cases it might, but not completely. Automation didn’t end QA, it expanded it, and AI will do the same - only faster. Every AI-powered product still needs people who understand uncertainty, bias, hallucination and fairness. It needs testers who can measure quality when quality isn’t binary, who can recognise that “working as intended” is no longer enough when systems behave in ways even their creators can’t always predict.
The probabilistic nature of models means we will need new kinds of acceptance criteria, new tooling and new metrics. We will also need the same curiosity that made us good at this in the first place. As testers, we remain the ones asking awkward questions, exploring the boundaries, shining torches into dark corners and making sure what we build actually works for the humans who use it.
A word to my fellow testers
If you are a tester today and you feel that itch, that sense that you want to experience something new and take your career somewhere different but still keep digging, this is your moment. Start learning about AI and how to test it. Learn how to make sense of systems that think rather than simply execute.
To truly excel at testing AI products and features, it helps to build an understanding of how these technologies work under the hood. You don’t need to become a data scientist, but getting familiar with how models are trained, how data quality affects outcomes, and what metrics like precision and recall actually mean will make you a stronger tester. A bit of data science knowledge gives you the vocabulary to collaborate effectively with engineers and the insight to ask the right questions when things behave unpredictably.
I’ll be sharing more articles and stories from my own experiences to help you set off on this path, the good, the confusing, and the downright fascinating. The goal isn’t to turn every tester into a data scientist, but to show that curiosity, critical thinking and a love for problem solving are exactly what this new world needs.
Keep your curiosity alive and your mind open, because testing is no longer just about finding bugs, it’s about discovering intelligence!