How to build a real-time research agent with grounded web evidence

TL;DR

• To build a real-time research agent that survives enterprise review, you need four things a single-query search does not give you: multi-step retrieval, evidence deduplication, source-diversity checks, and citation-ready output.
• Linkup's /research endpoint runs the full asynchronous deep-research loop (1-10 min latency, 61% on SealQA-0, #1 across the board) and returns source-diverse, citation-ready results, so you do not have to orchestrate the retrieval loop yourself.
• Naive tutorials stop at one search call and one LLM summary. That pattern fails audits because it cannot show where each claim came from or prove the sources were not all one domain.
• Working Python below. Start free with 4,000 queries, then decide between the /search loop (you orchestrate) and the /research endpoint (Linkup orchestrates).

Building a real-time research agent with grounded web evidence means wiring an LLM to a live web search API, then adding multi-step retrieval, deduplication, source-diversity checks, and citations so every claim traces back to a real source. Public builds this week (Adeena Ramzan's ResearchMind is one) show developers stacking agents on grounding APIs, but most tutorials stop at a single search call and a summary. That pattern demos well and fails in production. This post shows the production-grade pattern and the exact code to run it.

Why single-query research agents fail in production

A single search call plus an LLM summary is a demo, not an agent. It breaks the moment a real question needs more than one angle. Research Bench II, which evaluates 132 grounded research tasks across 22 domains against human expert rubrics, found that most deployed deep research agents fail more than 50% of rubrics, with the largest deficits in information recall and analysis. The cause is structural, not a prompting problem.

Three failure modes recur:

1. Shallow recall. One query returns one slice of the web. Complex questions need decomposition into sub-questions, each with its own retrieval pass.
2. Source monoculture. Naive search often returns five results from the same domain or content farm. If your agent cites five pages that all trace to one press release, you have one source, not five. This is why source diversity in a search API is a correctness property, not a nice-to-have.
3. Unverifiable output. If a compliance reviewer asks "where did this number come from?" and your agent cannot map the claim to a URL, the report does not clear review.

A production research agent has to fix all three. The rest of this post is the pattern that does.

The production pattern: decompose, retrieve, dedupe, diversify, cite

A grounded web search agent that clears enterprise review runs five stages, not one. Each stage exists to close one of the failure modes above.

1. Decompose. Break the research question into 3-6 sub-questions with the LLM. Each sub-question drives its own retrieval pass, which fixes shallow recall.
2. Retrieve. Run a live web search per sub-question against a real-time evidence retrieval API. Latency matters here: Linkup's /search runs at 1-3 seconds synchronous, so parallel sub-question retrieval stays under a few seconds total.
3. Deduplicate. Normalise URLs and hash content snippets to drop near-duplicates across sub-questions. Two sub-questions often surface the same page.
4. Diversity check. Count distinct root domains in the surviving evidence set. If distinct domains fall below a threshold (say 4), trigger another retrieval pass with reformulated queries. This is the step naive tutorials skip entirely.
5. Synthesise with citations. Feed the deduplicated, diverse evidence set to the LLM with a strict instruction: every claim must carry the source index it came from. Output maps each sentence to a URL.

The payoff is an audit trail. When a reviewer asks where a figure came from, the answer is in the output, not lost in a black-box summary. For the difference between orchestrating this retrieval loop yourself and handing the reasoning to a model, see retrieval vs reasoning in your AI stack.

Working code: the /search orchestration loop

This is the version where you own the loop. Good when you need control over decomposition and the diversity threshold. Set LINKUP_API_KEY from your environment.

import os
from urllib.parse import urlparse
from linkup import LinkupClient
from openai import OpenAI

linkup = LinkupClient(api_key=os.environ["LINKUP_API_KEY"])
llm = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

def decompose(question, n=4):
prompt = f"Break this into {n} focused search sub-questions, one per line:\n{question}"
out = llm.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": prompt}],
)
return [l.strip("-* ") for l in out.choices[0].message.content.splitlines() if l.strip()]

def retrieve(subquestions):
evidence = {}
for sq in subquestions:
res = linkup.search(query=sq, depth="standard", output_type="searchResults")
for r in res.results:
evidence[r.url] = {"title": r.name, "snippet": r.content, "url": r.url}
return list(evidence.values()) # dedup by URL key

def diversity_ok(evidence, min_domains=4):
domains = {urlparse(e["url"]).netloc.replace("www.", "") for e in evidence}
return len(domains) >= min_domains

def synthesise(question, evidence):
sources = "\n".join(f"[{i}] {e['url']}: {e['snippet']}" for i, e in enumerate(evidence))
prompt = (
f"Answer using ONLY the sources. Cite every claim as [index].\n"
f"Question: {question}\n\nSources:\n{sources}"
)
out = llm.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": prompt}],
)
return out.choices[0].message.content

def research_agent(question):
evidence = retrieve(decompose(question))
if not diversity_ok(evidence):
evidence += retrieve(decompose(question + " alternative perspectives", n=3))
return synthesise(question, evidence)

print(research_agent("What are the 2026 EU AI Act obligations for high-risk systems?"))

The diversity gate is the part that separates this from a toy. It forces a second retrieval pass when the evidence set is too narrow, which is exactly the recall deficit Research Bench II penalises.

When to use the /research endpoint instead of building the loop

Use Linkup's /research endpoint when the depth of the question justifies a longer, asynchronous run and you do not want to maintain the orchestration code. It runs the full decompose-retrieve-dedupe-synthesise loop internally and returns citation-ready output. It scores 61% on SealQA-0, #1 across the board, and runs asynchronously at 1-10 minutes per request. See when to use the /research endpoint for the decision boundary.

The simplest possible version:
result = linkup.search(
query="Full competitive landscape for grounded research APIs in 2026, with sources",
depth="deep",
output_type="sourcedAnswer",
)
print(result.answer)
for s in result.sources:
print(s.name, s.url)

Decision guide:

• Use the /search loop (code above) when you need custom decomposition logic, a tunable diversity threshold, or sub-3-second latency per pass.
• Use the /research endpoint when the question is deep, latency of a few minutes is acceptable, and you want Linkup to own the retrieval loop.
• Use /fetch (<2s, $0.001-$0.005 per request) to pull full page content for any URL your agent decides to read in depth.

Raw SERP APIs and single-shot search APIs give you step 2 only. You still have to build decomposition, dedup, diversity, and citation formatting yourself, and most SERP providers return no clean snippet content to ground on. If you are weighing options, the best web search APIs compared covers the trade-offs across providers.

Compliance: what makes the output survive an enterprise review

A research agent used inside a regulated business has to pass more than an accuracy check. The output has to be traceable, and the pipeline has to satisfy procurement. This is where the grounding layer choice becomes a compliance decision, not just a quality one.

Traceable citations. Every claim maps to a URL in the evidence set. That is the artefact a compliance reviewer needs.

• Zero Data Retention. Research queries often contain sensitive internal context (a deal name, a legal question). Linkup offers ZDR at no extra cost (activation required), so your queries are not retained. ZDR explained for AI teams covers why this matters for research workloads.
• SOC 2 Type II and GDPR as the baseline on all plans, with EU data residency available.
• Private Link for banks and regulated industries where queries must travel over a private network, and BYOC as a custom enterprise option where Linkup deploys inside your own cloud.

Parallel AI targets enterprise research with evidence-backed output too, but goes shallow on the compliance stack. If your agent will face a procurement review, that gap matters. See Linkup vs Parallel in production for the detailed comparison.

Start with the free 4,000 queries and build the /search loop above, then move to the /research endpoint once your questions get deeper. The API docs have the full parameter reference for depth, output type, and source formatting.

FAQ

What is the best web search API for research agents?

For grounded research agents that need traceable citations, Linkup's /research endpoint scores 61% on SealQA-0 (#1 across the board) and returns source-diverse, citation-ready output, so you do not have to build the retrieval loop yourself. The /search endpoint (1-3s, from $5 per 1,000 queries) fits when you want to orchestrate decomposition yourself.

How is a real-time research agent different from a single-query search?

A single-query agent runs one search and summarises it. A real-time research agent decomposes the question into sub-questions, retrieves per sub-question, deduplicates results, checks source diversity, and cites every claim. The multi-step pattern is what passes recall and grounding rubrics.

Why do research agents need source-diversity checks?

Because naive search often returns several results that trace to the same domain or press release, which is one source pretending to be many. Counting distinct root domains and forcing a second retrieval pass when diversity is low is what keeps conclusions grounded in independent evidence.

Can I use Linkup instead of Parallel for a research agent?

Yes. Both return evidence-backed output, but Linkup goes deeper on the compliance stack (ZDR at no extra cost, SOC 2 Type II, Private Link, BYOC for enterprise), which matters when your agent's output faces a procurement or audit review.

How much does it cost to run a research agent on Linkup?

The /search endpoint costs $0.005-$0.006 per request (from $5 per 1,000 queries), /fetch costs $0.001-$0.005, and the deep /research endpoint costs $0.25-$2.50 per request. The first 4,000 queries are free.

What is the ResearchMind-style pattern?

It refers to the developer trend of building research agents on top of grounding APIs rather than closed deep-research products. The production version adds multi-step retrieval, deduplication, source-diversity checks, and citation-ready output on top of a live web search API.