A QA lead had three weeks to prove what her test suite covered. Rebuilding it by hand exposed the bigger problem. That led to a platform that reconstructs QA documentation from the test suite, its linked tickets, and run history.
About the client
A QA lead had three weeks to prove what her test suite covered. Rebuilding it by hand exposed the bigger problem. That led to a platform that reconstructs QA documentation from the test suite, its linked tickets, and run history.
Background:
Dana runs QA on a mature B2B platform. She has been at the company for nine years and has handled release sign-off for the last four.
When a junior engineer asks whether a failed test points to a real bug or a stale expectation, Dana knows. Every release, she answers the same question: what does this touch, and what proof do we have that it still works? She finds the answer by reading tests and tickets, then talking to the two engineers who were there when the tricky parts were built.
She had raised the issue for two years, but it kept getting pushed to Q3.
Dana wanted the release process to stop depending on her. In planning meetings, she framed it as an operational problem: release sign off needed to be repeatable and something two other engineers could handle. The simpler reason was that she wanted to take a vacation during a release window. She had skipped two.
The Challenge:
Then a partner review came with a three week deadline. The client had to provide current test documentation detailing which behaviors are covered, by which tests, and when they were verified. Three weeks.
Dana built it by hand, and it took most of those three weeks. It also raised more questions than it answered. Of roughly 1,850 automated tests and 640 manual cases, 212 manual cases pointed to requirement IDs that no longer existed in Jira. Around 90 automated tests had been skipped in CI for so long that nobody knew what they were originally meant to cover. Dozens of manual cases described behavior the product had dropped two releases earlier. Some automated tests also enforced rules that appeared in no requirement. Changing one could break a test, with nobody left to explain why the rule existed.
The document shipped on time. It was accurate that day and outdated within a sprint. Producing the documentation was possible; keeping it current without spending another three weeks was the real problem.
Her tests proved results. They didn’t explain rules. An assertion shows the system returned something specific under specific conditions. The reason behind that result lived in a ticket, a commit message, or a decision made on a call in 2021. test_retry_limit_v2 tells you what the code does. It says nothing about whether the rule behind it still stands.
Writing it down had already failed twice, with both previous documentation pushes producing good pages that were stale within a quarter. The cost of a single correction was the problem: leave what you’re doing, open a second tool, find the page, learn that tool’s rules. Each interruption was small, but together they made the process unsustainable. People stopped, and the pages drifted.
More people wouldn’t have helped. Dana had considered asking for a technical writer. A writer would have needed the same answers from the same two engineers and produced a document with the same shelf life.
The knowledge already existed; what was missing was the connective tissue between the tests, tickets, and run history. The tests, tickets, and run history were already there. Dana was the only thing connecting them.
Solution:
Dana didn’t find them. Her VP of Engineering found Devox through a former colleague who had worked with the company on a legacy modernization project two years earlier. What he remembered was the system map. The team had spent its first month tracing dependencies and came back with a view his own engineers did not have.
Dana was understandably skeptical, having sat through two vendor demos in the previous eighteen months that worked beautifully on clean repositories but looked nothing like her codebase.
Devox asked whether the ticket history went back far enough to explain the code and whether the tests and commits contained identifiers that could connect the records.
The answers were yes, partially, and yes, enough to justify the engagement. If the client had only tests and a README, Devox would have started by rebuilding the missing engineering record.
Bringing in a vendor whose system reads the entire test suite, the ticket history, and production logs put the engagement in front of Dana’s InfoSec team before anything was signed. They set three conditions up front, and the team designed around them from the start.
Source code and logs stay inside the client’s environment. Client data could never be used to train an external model. Every AI-assisted change is traceable to a named person who approved it.
They started with a fixed scope assessment that lasted four weeks. Devox reviewed the existing engineering record to see what the team could reconstruct and what information was simply missing. Roughly a third of the manual case library turned out to be unrecoverable: no linked ticket, no author, no commit trail. The team found that during the assessment instead of four months into the project.
Then the work moved to a dedicated monthly team. Four Devox engineers, QA automation, AI/ML, backend, and a solution architect, working alongside Dana’s QA team rather than in place of it. As the team learned more, the scope kept changing. A dedicated team made more sense than trying to define another fixed project up front. The platform ran inside the client environment from the first build, and the knowledge base was contracted to stay with the client.
Dana had final say on what counted as correct. Devox reviewed the generated output with Dana and two QA engineers every week, and they flagged anything they would hesitate to trust. That cost her team about half a day a week. The alternative, where the vendor builds in isolation and presents at the end, is how the two previous demos had gone.
Devox engineers worked in the client’s Jira instance and repository with limited read access. Two things Dana asked for stayed out of scope. Generating test cases from requirements could wait until the documentation was reliable enough to use as input. Self-healing selectors were still experimental, and this engagement wasn’t where that got tested.
After handoff, the team moved to a lighter support cadence. Dana’s team owns the generation rules now and has added several of its own.
Dana’s test suite already contained the specification. The job was to make it readable by someone else.
The first build ran cleanly and produced a polished document that simply restated the test suite in prose.
Dana read four pages and stopped. Every sentence was true. PaymentRetrySpec covers payment retries. The document restated her test suite in prose at greater length and answered none of the questions she’d asked.
A test alone can support a sentence about behavior. A useful explanation also needs the original ticket, the commit, and the test history. The next build pulled all of that in.
The parser treated tests as structures rather than files.
AST analysis handled what plain parsing gets wrong. A parameterized test can look like a single test while validating six separate rules with different expected values. Shared helpers hide assertions a level below the test that appears to own them. Each test became a node, with every assertion stored as a child node. One test usually validates several behaviors, and a change usually touches one of them.
The build that nearly ended the project generated documentation Dana liked. It read well. Given a test named assertRejectsExpiredToken, it produced a clear paragraph about session policy.
She found one that described a rule that had been reversed two years earlier. Nothing in the writing distinguished it from the correct entries around it.
A document she could not trust was worse than spending three weeks doing the work manually, because the manual version was at least accurate when it shipped. If this thing shipped to her team, someone would sign a release on a sentence a model had assembled from a filename.
The team pulled inference completely out of extraction. The team set a strict rule: every behavior statement had to come from the test body and explicitly linked evidence. If a test has no supporting ticket, case record, or comment, the platform records only the observable behavior. It marks the intent as unresolved. Inference moved into a separate, clearly labeled step.
Every documented behavior carries a state.
Verified means a requirement, a test that checks it, and a recent passing run. Partially supported means a test and a run without a requirement, or a requirement and a test that hasn’t executed lately. Reconstructed means no current requirement exists, so the intent comes from code, history, or a retired case.
Reconstructed entries are written differently. They say what the evidence supports and what the team still can’t verify. Because most entries for the older parts of the product were initially reconstructed, they also gave the team a clear map of where further investigation was needed.
When the sources disagreed about the same behavior, the platform stored the disagreement, with each version attached and timestamped. Three patterns recurred: documented but never executed, executed but never documented, and quarantined- an expectation that was true once, is unenforced now, and has evidence in neither direction.
Conflict resolution remained a product decision. Which version of a rule is correct is a product decision. In the first month, the platform surfaced 340 conflicts, which sounds alarming until you remember that all 340 already existed and the team had been shipping around them.
The team ran the output by QA engineers who had not helped build it, and the feedback was blunt. Entries ran too long. Evidence links sat at the bottom where nobody clicked. Reviewers wanted the last run date at the top, because that was the first thing they checked anyway.
Three reviewers made the same point in different words: they trusted entries that clearly showed their gaps and questioned the ones that sounded too polished.
Each correction remains attached to the behavior, along with its author and timestamp. If the same correction keeps coming up, the team can turn it into a generation rule. Corrections also happen where the work already is, in the test file, the pull request, or the run report, in one action. Dana’s two failed documentation pushes were the design brief for that part, and anything that required opening a second tool got cut.
A CI hook triggers incremental re-analysis. The platform computes which test nodes changed from the diff, updates their behavior statements, and walks the links to mark connected documentation and coverage entries as potentially stale. Execution results update the last-verified date without regenerating text. Unchanged areas aren’t reprocessed, so a small change only triggers a small update.
A coverage percentage answers whether code executed. At sign-off Dana needed something else: which behaviors have current evidence, and how fresh it is. The platform produces a coverage view in four states: checked by an automated test with a recent run, checked by a manual case executed this cycle, documented with no execution, and quarantined. Release scope now comes from that view, and each release produces a snapshot that is exported for partner and audit review.
Test-suite migration, release coverage, and change-impact mapping don’t require a complete engineering record. Migrating a test suite from one framework to another. Generating regression coverage per release slice. Mapping which paths a change affects before the test cycle starts. Suite size changes the timeline, not whether the work can be done.
Documentation reconstruction is different: it depends on what evidence already exists. Those three questions tell Devox whether the project is viable before anyone proposes the work. A repository full of tests, a README, and no ticket history gives the platform nothing to work from, and a system that produces confident documentation out of that is inventing it. Where the answer is no, the honest first engagement is building that record. Devox Software works this out during scoping, because finding it in month two helps nobody.
Technology
Stack:
Local and Hosted LLMs • Retrieval-Augmented Generation • Test Framework Parsing • AST Analysis • Traceability Graph • Jira API • Confluence API • Test Management Integrations • CI/CD Hooks • Git • Vector Search • Embeddings • Log Ingestion • Structured Outputs • Markdown / Confluence / PDF Export • Docker
Almost everything load-bearing in this platform is deterministic code. Parsing test files into symbols and assertions, resolving definitions across modules, building and traversing the graph, computing changed nodes from a Git diff, resolving links by explicit identifier- none of that goes near a language model. It parses LSP data and graph queries, and it produces the same answer every time it runs.
The model has only two jobs: turning assembled evidence into readable QA documentation and proposing candidate links where no explicit identifier exists.
Every task handed to the model is one where a wrong answer is visible on inspection. Every task where a wrong answer would be invisible stayed in code.
The model never sees the entire test suite at once. The traceability graph holds the state between calls.
For each behavior, the harness sends the model a small evidence bundle: the test and its assertions, linked ticket text, recent run results, and any previous human corrections. That bundle is what the model receives. Each bundle stays small, usually a few thousand tokens, and generation runs one feature area at a time. Give the model ten behaviors at once, and it starts borrowing evidence across them.
| Task | Model | Why |
| Assertion extraction, classification, terminology normalization, structured field population | Small instruction-tuned models, running locally | Fixed output shape, narrow input. A reasoning model spends tokens deliberating over work that is rule application. |
| Candidate link proposals | Local embedding models | Similarity scoring, never a final decision. |
| Behavior descriptions where several pieces of evidence must be reconciled | Larger hosted model | The one task where reconciliation quality justified the call. |
Permissions. The model had read access and nothing else. It could read test files, ticket text, and run metadata for the area it was working on. It could not read application source outside test directories. It had no write access to the repository, to Jira, or to the knowledge base. Generated output landed in a staging state, and promoting an entry to reference documentation required a named human. The platform refused to promote entries with unresolved intent until someone looked at them.
Deterministic validation before storage. Before storing generated text, the harness checks every factual claim against the attached evidence. If the required evidence isn’t attached, the output is rejected and regenerated. Validation runs in deterministic code. Any failed output goes back for regeneration.
Audit trail. Every generated statement stores the model that produced it, the prompt version, the IDs of the evidence objects in its bundle, the timestamp, and the person who approved it. The audit trail made the bad entry easy to diagnose. Instead of guessing what the model had done, the team pulled the exact evidence bundle it had seen. A stale ticket had been paired with a current test. Link resolution caused the error by pairing a stale ticket with a current test.
Extraction, classification, embedding, and redaction were performed using local models within the client’s environment. Log ingestion passed through field-level redaction before anything reached a model, so payloads and customer identifiers were stripped at the boundary. Raw source code and unredacted logs never left the environment.
The hosted model received only redacted evidence bundles under an enterprise agreement that prohibited training on client inputs. The bundles contained no source code or identifiers. The client can also run synthesis entirely on local models. After testing the quality trade-off, the client chose to keep the hosted setup with redactions.
Results:
The second partner review took one afternoon. More importantly, someone else on the team ran it.
Dana now starts with the release and works back to the affected behaviors and their evidence. She focuses on the entries marked partially supported or reconstructed. The platform now gathers most of that evidence overnight.
Two other engineers can now prepare a sign-off because they have access to the context that used to live in Dana’s head.
She used to rely on both evidence and memory at sign-off, with no clear line between the two. Now the document tells her which is which. Some of what she’d been confident about turned out to be reconstructed, and she’d rather know.
She took a vacation during a release window in March. Someone else signed.
What the platform delivers:
Customer
Voice:
“Signing off a release used to mean reading the tests to figure out what they actually proved. Now I can see which behaviors a release touches, what evidence supports each one, and where the gaps are. The last part turned out to matter most. I can see where the evidence is strong and where it is thin, and that matters more than polished writing.”
— Dana, QA Lead
If release sign off depends on one person, you already have a risk.
Devox Software builds documentation platforms that stay in sync with your engineering tools as the product changes.
Tell us where your system needs help — we’ll show you how to move forward with clarity and speed. From architecture to launch — we’re your engineering partner.
Book your free consultation. We’ll help you move faster, and smarter.
Share the details of your project – like scope or business challenges. Our team will carefully study them and then we’ll figure out the next move together.
We appreciate you reaching out. Your message has been received, and a member of our team will get back to you within 24 hours.
In the meantime, feel free to follow our social.
Welcome to the Devox Software community! We're excited to have you on board. You'll now receive the latest industry insights, company news, and exclusive updates straight to your inbox.
