In 2026, approaches to corporate knowledge management shifted as teams moved from manual information retrieval to automated, multi-agent systems. Internal Research Automation (IRA) became a key tool for technology and analytics teams. This system integrates language models, a Retrieval-Augmented Generation (RAG) architecture, and specialized scripts to collect, analyze, and synthesize historical web data into analytical reports.
We integrated with the Internet Archive’s CDX API, enabling longitudinal analysis of market data. The architecture combines semantic parsing, dynamic output correction, and multi-agent orchestration to ensure reliability and consistency.
From Raw Data to Automated Workflows
Corporate content teams create deep technical materials for B2B buyers. Preparing a single analytical article can leave teams juggling dozens of browser tabs, scattered notes, and AI chats. Editors then have to piece the research trail back together just to verify every fact before publication. The publishing workflow follows a clear sequence: brief definition, targeted search, text creation, automated proofreading, and export. To scale the process without adding more manual work, we built the Internal Research Automation (IRA) system. The platform takes a technical brief from research request to publication-ready draft, maintaining a clear link between every claim and its supporting evidence. The system’s foundation combines large language models (LLMs), Retrieval-Augmented Generation (RAG) pipelines, and tools for interacting with web archives. A dedicated application layer orchestrates prompt logic, controls factual accuracy, and manages integrations with CMS platforms.
System Design and Architecture
The automated pipeline begins with a Brief Schema that defines the audience, objective, and structural requirements for the article. This central document stays with the article throughout the workflow, from the initial search through final proofreading.
The information-processing logic relies on a targeted evidence-selection principle. The platform decomposes complex research queries, fetches relevant pages, strips out visual noise, and extracts the 8 to 12 most relevant passages for analysis. During drafting, the model uses a claim map and builds each section around specific supporting evidence.
We built a multi-agent ecosystem that autonomously collects historical data, extracts the semantic core, and synthesizes publication-ready articles. Every generated statement maintains a clear, verifiable link back to its source.
Say an editor needs to establish when a competitor removed a feature from its product page. The system does not hand that question straight to an LLM. First, the Planner turns it into a search task: which URL matters, what date range matters, and what evidence would actually answer the question. The Retriever then asks the Wayback Machine for snapshots from that period, removes the menus and page chrome, and keeps the passages that contain the product language we care about. Only then does the Writer see the evidence. It drafts the claim from those passages, and the Evaluator checks the finished sentence against the archived source before the paragraph moves forward.
But the whole workflow falls apart if the system cannot reliably get the source material in the first place. And that became a problem when Google Cache disappeared.
Transitioning to Web Archive Infrastructure
Historically, developers relied on Google Cache as the primary source for restoring historical web data. Since Google discontinued its public cache, historical data collection required a different approach. We transitioned to the Internet Archive’s Wayback Machine, a widely recognized, scalable resource that has archived over 1 trillion web pages since 1996. One URL now gives us a timeline instead of a single page.
The key tool is the CDX API, which provides structured, high-speed access to archive metadata, allowing us to build precise queries for data collection.
Technical Constraints and Data Collection
At scale, the Wayback Machine forces you to work within access limits and rate constraints. Downloading full web pages adds significant navigation elements and visual noise, which distracts the language model during processing.
Developers implement specialized solutions using waybackpy and automated proxy-rotation algorithms to keep the data-collection pipeline running smoothly. The Trafilatura library cleans archived HTML and extracts the main content while preserving its structure. The CRAG evaluator assesses the relevance of retrieved fragments during the retrieval and evaluation phase, enabling the system to dynamically filter information and initiate extended searches for up-to-date knowledge, while Self-RAG helps the generation layer stay grounded. A dedicated QA module validates factual claims, verifying numbers, product names, and technical specifications against established evidence.
How does the platform collect the historical pages it needs?
How Historical Pages Are Collected
Our data collection module extracts historical snapshots from the Wayback Machine. We use HTTP requests to the CDX API to fetch structured metadata about page snapshots: timestamp, URL, status, content type, and unique digest.
Each request is configured using six main parameters:
- url — specifies the exact address or domain, with a * wildcard for large-scale scanning.
- from/to — defines the historical date range, allowing analysis of market response to specific events.
- matchType — enables data collection from subdomains, subdirectories, or the entire domain tree.
- collapse — deduplicates snapshots by digest so the system downloads unique page versions.
- filter — selects only successful text documents, excluding irrelevant files.
- limit — restricts the number of returned records to control memory usage.
The retrieval flow is straightforward: after receiving the metadata list, the system builds the full URL by combining the Wayback Machine base, timestamp, and original address. This provides
access to the archived page’s full HTML, which is then passed to the next processing stage. To keep retrieval reliable, we implemented request throttling and retry logic. waybackpy plus exponential backoff helps us stay within rate limits without bringing the pipeline to a crawl. For partner initiatives, we authenticate via S3 access keys, which provide expanded request quotas.
This approach provides a predictable stream of historical source data, a critical element in building a reliable article generation system. Every step, from query formation to HTML retrieval, keeps the pipeline moving without unnecessary retries or downtime. The result is a clean, structured stream of historical data ready for analysis, forming the foundation for all subsequent system operations.
HTML Cleaning and Semantic Chunking
The hard part starts here. Archived HTML has to become clean text without losing the structure that gives it meaning. We use Trafilatura to extract the main content from complex web pages. It analyzes the DOM structure, identifies the main content, and preserves meaningful structure such as headings, paragraphs, and lists.
For edge cases, we employ a combination of Requests and BeautifulSoup with targeted XPath rules to extract information from specific HTML tags.
Instead of fixed-size chunks, we split the text along semantic boundaries, while preserving complete sentences, paragraphs, and lists regardless of length. This ensures every chunk contains a complete unit of thought, ready for vectorization and retrieval.
In practical terms, the model does not search these pages the way a browser searches for an exact word. Each chunk is converted into a numerical representation of its meaning. The user’s research question is represented the same way, and the retrieval layer looks for the chunks that are closest to it semantically. That is what lets a question about “pricing changes,” for example, find a passage that says “we revised our plans,” even when the wording differs.
Semantic chunking improves the system’s ability to extract relationships between facts. This means the system reliably extracts complex dependencies and facts from historical data, giving the generation layer reliable evidence to work from. Each chunk is converted into a vector representation in the knowledge base that the generator can quickly retrieve and use.
This stage closes the gap between raw HTML and clear evidence the rest of the pipeline can use. It ensures downstream components, RAG, CRAG, and Self-RAG, work with clean, semantically coherent, and relevant content, directly impacting the quality of the final output.
At this point, we have clean text. But clean text is not the same thing as useful evidence. The next problem is deciding which pieces of that text actually deserve to reach the model.
Advanced RAG Architectures and Caching
This is the weak point in a conventional RAG pipeline. The source can contain the right answer and the model can still miss it, simply because retrieval returned the wrong passages. The LLM cannot reason from evidence it never received. CRAG and Self-RAG decide which evidence reaches the model in the first place.
We implement the CRAG (Corrective Retrieval Augmented Generation) architecture, an intelligent evaluator that assesses the quality of retrieved data and makes one of three decisions:
- Correct retrieval: if the data is highly relevant, the system applies a “decomposition-recomposition” algorithm to focus on key information.
- Incorrect retrieval: if the data does not match the prompt or historical coverage in the Wayback Machine is missing, the system triggers a fallback search on the live web to fill information gaps.
Ambiguity: a hybrid approach combines archived knowledge with live web searches to reach an accurate conclusion, especially when verifying recent market changes relative to historical baselines.
Suppose the first retrieval returns a pricing page from 2024, but the claim we are checking is about a change made in 2022. The system should not simply give the model the closest-looking result and hope for the best. The evaluator marks the evidence as insufficient, narrows the date range, and sends the query back through retrieval. Drafting continues only when the evidence matches the claim.
For harder research tasks, we use Self-RAG. This transforms article drafting into an iterative loop: the model analyzes the prompt, generates candidates, evaluates them for factual grounding, and integrates the best option into the text. This approach helps keep generated content grounded in relevant context.
To optimize compute costs, we utilize semantic caching: the system stores previously generated responses and vectors in a low-latency cache. If a new query is close enough to one we have already answered, the system serves the cached result instead of doing the same work twice.
Each step is designed to keep weak evidence out of the final draft. The system extracts, verifies, and structures facts before they reach the draft.
The end product is an article that directly addresses core audience questions, backed by concrete data and citations, maintaining the layout, tone, and positioning appropriate for the target market. The goal is simple: give editors a draft they can review without retracing the entire research process.
Up to this point, you can think of the system as a pipeline: step A hands work to step B, which hands it to step C. If the Writer lacks enough evidence for a claim, the workflow can kick it back to retrieval instead of pushing weak material forward. The system can send the problem back to retrieval, change the query, collect another source, and try again. That feedback loop is what breaks the one-way pipeline and turns it into an agent-based research process.
Once retrieval can judge the quality of its own evidence, the next question is who decides what happens next. That is where the agent layer comes in.
How the Agents Divide the Work
Our script evolved from a linear pipeline into an iterative multi-agent workflow. The system breaks the job into specialized tasks handled by agents that continuously exchange feedback to refine the analysis. Each agent gets its own prompts and tools so that it can stay in its lane.
The internal framework relies on four primary agents:
- Planner Agent parses the user prompt, structures the outline, forms research hypotheses, and builds a sequence of queries for the web archive. It uses a large context window to keep the big picture in view.
- Retriever Agent interacts directly with the Wayback Machine CDX API. This agent handles rate limiting, parses HTML via Trafilatura, and initiates semantic chunking. It executes Python scripts to keep retrieval reliable.
- Writer Agent applies Self-RAG architecture to draft individual article sections using cleaned, verified historical data.
- Evaluator Agent performs strict fact-checking on generated content. This agent checks every claim against the source data and flags anything that doesn’t add up. When something doesn’t line up, it sends the section back for another pass.
This evaluation process mirrors an editorial fact-checking workflow. The system evaluates each section independently to maintain factual consistency.
The multi-agent setup structures the workload through an iterative feedback loop so research, retrieval, drafting, and verification dynamically inform each other, ensuring generated materials meet analytical requirements for internal review.
Editor Output and Evidence Verification
So editors don’t have to reconstruct the research path manually, the platform provides an evidence package alongside the generated text. Every drafted claim includes a metadata block: the exact source URL, the archived snapshot timestamp, the original supporting text passage, and the Evaluator Agent’s automated validation result. That lets an editor open the source, check the passage, and decide whether the claim is supported before approving the material for publication. Each collection run is also logged: request history, snapshot metadata, and validation steps stay attached to the draft, so any claim can be audited after publication.
What This Changes for the Research Team
The biggest time savings come from keeping the evidence attached to the work instead of rebuilding the research trail later. The system keeps supporting evidence linked directly to the text, giving editors direct access to the source used for each claim. Saved research templates and plans speed up future content updates.
That frees the team to focus on the story instead of chasing down sources. The workflow brings search, drafting, fact-checking, and publishing into one place.
If your team rebuilds the research trail for every article, we can map where evidence gets lost in your workflow in a one-hour call.
