At Doriot, one of the two systems I built was a research agent for questions about startups, investors, and funding activity. The obvious approach is a standard RAG pipeline: embed funding data and news, retrieve the closest chunks, hand them to an LLM. I tried that first. It fell apart quickly. Market research questions aren't one shape.
Someone asking "what was the last funding round for this company" wants an exact fact from a structured database. Someone asking "how does this company's funding trajectory compare to others in the same space" wants synthesis across a lot of unstructured context. A single generic retrieval path treats both identically and does a mediocre job on each.
Classify the Intent Before You Retrieve Anything
The fix was to stop treating retrieval as the first step. Put a classifier in front of it. I trained a custom SpaCy text classifier to route queries across more than 20 specialized categories: funding round details, competitor analysis, lead generation, market trend questions, and others specific to the platform.
The important detail: this is multi-label, not single-label. Real questions often touch more than one category at once. "How does this competitor's latest round compare to what's happening in their sector" is a funding-round question, a competitor-analysis question, and a market-trend question. Forcing that into one bucket loses information the routing layer needs. Multi-label classification lets one query trigger more than one downstream handler and combine the results. No guessing which single label wins.
intents = classifier.predict(query)
# intents might be: ["funding_round_details", "competitor_analysis"]
handlers = [INTENT_HANDLERS[i] for i in intents]
context = merge_contexts([h.retrieve(query) for h in handlers])
Routing to SQL or Vectors, Not Just One
Once intent is known, the retrieval path branches. Structured questions, like exact funding amounts, round types, or dates, go straight to SQL against the relational funding database. There's no reason to pay for an embedding lookup when the answer is just a row in a table. Narrative or comparative questions go a different way. They run through a semantic path: pgvector similarity search over 1536-dimensional embeddings from company descriptions, news coverage, and funding commentary.
A lot of queries need both. The agent runs the structured lookup and the semantic search in parallel, then merges results before generation. A comparative question gets exact numbers from SQL alongside qualitative context from the vector store. The model doesn't have to guess or hallucinate the numbers.
Generation and When to Escalate
For generation, the primary model is GPT-4 through Azure OpenAI. For queries that need deeper multi-step reasoning, like comparative analysis across several companies, the agent falls back to DeepSeek-R1. I cover that routing logic in a separate post. It deserves its own treatment rather than a paragraph here.
Keeping the Data Fresh Without Blocking Requests
None of this works if the underlying data is stale. A dedicated scraper pulls from TechCrunch. A NewsAPI integration continuously ingests funding announcements. Both run as background jobs on Celery workers backed by Redis. Scraping and embedding generation never sit in the request path of a user query. Prometheus tracks health separately from request serving: queue depth, job failures, latency.
What This Taught Me About RAG Design
The lesson I keep coming back to: a lot of RAG systems try to fix bad retrieval with better prompts. That's backwards. Classify what a query is asking for before you touch retrieval. That solves the problem earlier, and it's cheaper than patching it after generation with more context or more careful prompting. Intent routing isn't a nice-to-have on top of RAG. For a domain with genuinely different question shapes, it's the part of the architecture that does the most work.
Learn More
The full project is open source: Investment Market Research Agent on GitHub.