On Testing — The Reality of Testing in Web Development and Automated Testing with LLMs

Who's writing this

Speaking purely subjectively, I think of myself as a probably-average software engineer who has picked up more or less the full set of skills demanded by modern web application development. I'm a veteran, so I've been through a variety of workplaces and have a picture of how the world builds software. That said, my experience is skewed: I've only worked at startups and small-scale development projects. Teams of a few people, at most a dozen or so. The kind of place where you look after the entire development lifecycle end to end. Conversely, I only know large-scale enterprise contract development — the world of systems integrators (in Japan, "SIer"), where big IT firms build systems for client companies under formal specifications — from hearsay, so in that sense there's a bias. I've studied a mountain of application development techniques, but I've barely studied test design or quality assurance techniques themselves.

I understand nothing about testing…

What do we test for?

Before going further, I first want to confirm the very question of what it is we test for in the first place.

What we software engineers do is, first and foremost, develop software features that users can use and deliver them. Testing is an indispensable part of that goal. But testing itself is a means, not an end. In other words, if the software feature gets delivered, testing might be skippable — but the reverse doesn't hold ¹. Personally, I think the purpose of testing is to gain confidence that the software works as intended ². And the thing that gives you the greatest confidence, end to end, from a quality-assurance standpoint, is actually operating what you built by hand the same way a user would — so-called manual testing. There can't be many programmers who check in code without running a manual smoke test at least once ³. As a technique for gaining confidence at a micro level, there's also TDD, running the red→green cycle, but that's hard to apply end to end. If a test automation script is written clearly, you can read it and understand what it's meant to verify — but I think you can't be sure it's a correct test until you've run it at least once. Code is unknowable until you run it; to put it another way, there's a limit to a human's ability to grasp code.

However, no matter how much confidence manual testing gives you that the software works as intended, on its own it isn't enough to keep the development lifecycle turning — because of problems with execution cost, procedural accuracy, coverage, and so on. So various kinds of automation become necessary. And so, in addition to manual testing, we end up implementing automated, repeatable tests such as unit tests and E2E tests.

The reality of testing in ordinary web development

This may be prejudice, but the average web engineer is extremely interested in programming languages, frameworks, and design techniques, yet doesn't seem all that interested in testing itself. TDD and DI are among the popular topics. But interest in them is as development techniques; it isn't about how to design the tests themselves. In fact, even in Kent Beck's TDD book, the content of the tests is decided in a somewhat improvised, intuitive way. That's natural, because TDD's concern is gaining confidence that the feature right in front of you is working correctly. There's even a sense that the average web engineer thinks the content of the tests themselves is so obvious it doesn't warrant deep thought. As a result, in many software projects, tests aren't properly "managed." Of all the workplaces I've been involved with, there is honestly not a single one that seriously manages the tests themselves. At best, code coverage is tracked as a metric, but its content and quality are never grasped. ⁴

On the other hand, the concerns of the people called QA — whom I don't know well — seem entirely different. QA people analyze and design tests based on the application's specification, plan them, and implement and execute them. There, what percentage of the specification the tests cover becomes a major concern, and boundary values and error-time behavior are tested meticulously. As a result, the system's behavior ends up matching the behavior defined in the specification down to every corner. This is what high-quality software means. For the workplaces I've been in, the running system is everything, and there's no such thing as a specification document anywhere to begin with — so it's a completely different world.

But why does web development testing end up this way? I think the practice I described above is an "equilibrium." It's true that many applications are released without being systematically tested. The primary task imposed on developers is to deliver features (quickly), not to assure quality meticulously, so there's always an incentive toward getting testing over with sloppily if possible. So when you carefully inspect a released system down to every corner, there's some trivial bug left somewhere. The backlog is probably piled high with minor defects. Sometimes there's even a serious oversight that leads to an incident, and at those times you really do have to reflect on it.

If tests aren't managed, how is the system's quality guaranteed? Quality depends heavily on the developer's programming skill, code review, and the development process. Skilled, experienced programmers know bug patterns and best practices from past experience, so they can design and implement in a way that keeps bugs from arising in the first place. By intuition, they identify which test code is worth writing and secure coverage. Skilled, experienced programmers can also catch the fishy parts in code review. Furthermore, a development process of short iterations minimizes the damage from bugs that do surface, through triage and quick recovery. Because of these things, even without systematically managed tests, the situation doesn't become all that dire. Also, because we develop with an agile process, the project's situation is monitored. If a quality problem is clearly recognized, a corrective force operates within the process and some countermeasure gets implemented. Every event is assigned a priority and stuffed into a queue.

If unmanaged testing were clearly recognized as a fatal problem, the process's self-cleaning action should repair it. If tests going unmanaged never surfaces as a problem, then it probably isn't a problem. To put it another way — optimistically — you could say that low-value tests are adaptively dropped through craftsmanship and process, and thereby optimized. This is what "equilibrium" means. And where the equilibrium lands, of course, differs by the context and environment of the team and project.

Using LLMs (AI agents) for testing

LLMs can of course be used for testing too, and there are various ways to use them.

The most naive way is: when you have a coding agent write the application code, you also have it write the test code along the way (or the agent writes it on its own). At this point, unless you give it particular instructions or constraints, the agent does the test analysis and design on its own (implicitly). The LLM writes tests without accessing runtime information — detecting accessibility IDs and such from the code — or, if the hooks needed for testing aren't there, it adds them to the application. Once you start using AI agents, I think this sort of thing starts happening almost naturally. LLMs are generally thought to be good at coming up with test cases, and indeed they cover even fairly fine-grained cases. What probably tends to happen here is a situation where lots of test code gets added, but it isn't reviewed much — including by the person who committed it — and nobody knows whether it's appropriate. Analyzing the correctness of sloppily written test code is a hassle to begin with; and once an LLM has generated it in bulk, nobody wants to spend serious time reviewing it. As a result, the code grows, but what value it has is unclear. A practice roughly like sorcery ends up spreading here in 2026. If you're lucky, those unmanaged automated tests may later catch a regression ⁵.

Another approach that's emerged recently is to describe tests in natural language, have the LLM interpret them, and then operate the application under test through tool calls. Tools of this kind include playwright-cli/mcp and appium-mcp. There are also many test tools with a similar concept offered as SaaS. Since an LLM can also poke the UI based on a screenshot of what's currently displayed on screen, there's even the possibility of doing things that are hard to cover with ordinary tests. Tools of this sort can be divided into two kinds. One kind is driven by the LLM on the first run, but generates test code at that time, so that from the next run onward you can execute a deterministic test with the generated code. In other words, this is effectively a test-code generator that operates while interacting with the execution environment. playwright-mcp and appium-mcp fall into this. The other kind doesn't generate code; it's a pure, natural-language-based test execution engine.

That said, operating the execution environment via an LLM is extremely slow. And above all, it's expensive. Running frequent, high-volume test executions like regression testing entirely through an LLM currently has issues in terms of both speed and cost. And because it's an LLM, you can't guarantee the intended behavior 100%, so it may not be usable for mission-critical purposes. On the other hand, the nature of an AI agent — carrying out tasks asynchronously even when it runs into some unexpected situation — is undoubtedly convenient when it fits well. If this kind of LLM test execution engine were to run 24/7 on an open-weight model without worrying about token cost, I feel a different world would come into view. On the other hand, considering that an AI agent's greatest merit is asynchrony (i.e., not requiring human hands), I feel a design that simply executes test scenarios you hand it as natural-language input doesn't get you much of the sweetness of autonomy — because it increases the room in which a human has to intervene. Personally, what I'd like is: you give it the target system's URL and permissions to the GitHub repository, and it explores and analyzes the system on its own, accumulates knowledge, and goes all the way to autonomously confirming and reporting that the system has no problems — with almost no human intervention (other than answering questions the AI raises). If it went that far, I think "slowness" would stop being a problem.

In any case, speaking only in the context of test implementation and execution, I feel the first approach — having it write test code at the same time as the application code — is currently sufficient. It's also the best timing for writing tests. A coding agent has complete access to the app's internal information (the code), and can even intervene in the app itself (adding accessibility IDs, and so on) if need be. The problem of how to audit the LLM's enormous output remains, but that's something we have to deal with either way. However, practicing this approach presupposes that the person doing the testing can access the app's code itself, so in a situation where the development team and the QA team are separate and QA can't touch the code, this can't be done. In that case, you have no choice but to take a different approach.

Also, as a use beyond test implementation and execution, you could consider having the LLM do test analysis and design, and perform test management. As I said earlier, I've never seriously tackled QA before, so I think the style of a programmer writing app code and test code simultaneously and ad hoc is sufficient — but if high-quality software is demanded and you need to do QA systematically, you'll have to think about a more serious approach, and it's worth considering having the LLM analyze and design the tests. There's also an LLM-based SaaS product that supports this.

Summary

• At the average web development site, they don't do very systematic testing, but they manage somehow
• As long as tokens cost money, I think you don't need to have the LLM directly execute tests (especially if developer = QA)
• If developers and QA are separated, or you need to do systematic testing, the room for leveraging LLMs seems likely to grow