RAG is Not All You Need

Why Connecting LLMs to Business Apps Isn’t Just “RAG”

In the rush to apply large language models (LLMs) to enterprise systems, many companies fall into a seductive but flawed assumption: “Just add RAG”. A recent example of this shift is OpenAI’s integration of ChatGPT with OneDrive and SharePoint via the Microsoft 365 Copilot. Retrieval-Augmented Generation (RAG) works wonders in knowledge bases, documentation, and email threads (such as what OpenAI did with SharePoint) and there are literally tens of startups right now working on bringing these use-cases to the enterprise (and good luck to many of them following recent announcements by OpenAI and the like) but when it comes to structured systems like Salesforce, NetSuite, Data warehouses and/or thousands of business spreadsheets that your organization maintains, the game changes.

Business applications don’t express themselves in clean prose — they think in tangled webs of interrelated tables, messy records, redundant fields, and missing values. And spreadsheets, still the backbone of day-to-day business intelligence (yes, even now), are often worse: ad hoc, inconsistent, and lacking a consistent structure or schema.

You know how your kids have collected puzzle sets over the years — and every time they finish one, a few pieces inevitably go missing? Now imagine dumping the unmarked pieces from ten of those half-missing puzzles onto the floor and asking someone to “describe the picture.” That’s what simply throwing an LLM with just RAG at Salesforce, NetSuite,RedShift or Snowflake looks like.

It won’t work. It just can’t. Here is why:

1. The Data Is Tabular — And It’s a Jungle

Enterprise systems are built on sprawling relational databases. Salesforce, for example, contains objects like Accounts, Contacts, Opportunities, Leads, Activities, and dozens (often hundreds) of custom objects per implementation and integration. NetSuite adds its own layers of financial records, journal entries, invoices, and more. And I am not even talking about dataware houses with terabytes of tabular data from hundreds of different tables.

These aren’t well-written articles waiting to be summarized — they are rows with foreign keys, nullable fields, and nested hierarchies. Worse:

• Column names are cryptic: acct_id, cust_num, is_active__c
• Rows are incomplete: fields are left blank, populated inconsistently, or updated out of sync
• Redundancy is rampant: the same customer might appear in three systems with different names
• Connection between tables is not clear nor it is documented anywhere.

Throwing all of that into an embedding store and hoping RAG will solve it is a recipe for noise, hallucination, and confusion.

2. LLMs Need Structured Context — Not Just Raw Tables

Imagine asking an LLM, “Which of our enterprise customers saw declining engagement this quarter?” Now imagine handing it a dump of 170 interlinked tables, each with 40+ columns, inconsistent naming, and zero schema documentation.

LLMs don’t magically “understand” that engagement_score in one table loosely corresponds to last_login_count in another. They don’t inherently know that “enterprise” customers are defined by account size > $500k AND product tier = ‘premium’.

To answer even a basic business question, the model needs:

• Contextual definitions: what’s a customer, what’s engagement, what’s a quarter?
• Schema knowledge: how are Accounts related to Users, Sessions, Products?
• Business logic: how do we define “churned” vs. “inactive”? What classifies as “Declining”?
• Domain norms: what counts as good engagement in this industry? How do we measure ourselves?

This is not a retrieval problem — it’s a compositional reasoning problem.

3. RAG Breaks Down Without Semantic Understanding

RAG relies on chunking, embedding, and retrieving relevant documents. But what do you chunk when your source is a tangled graph of partial records and references?

If you give the LLM all the tables, it gets overwhelmed. If you give it just one, it lacks the relationships. And if you try to pre-process the data into text, you’re forced to flatten deeply relational knowledge into brittle summaries that don’t generalize.

This results in:

• Logical gaps: confusing correlation with causation, or omitting edge cases
• Shallow insights: the model might report “low usage” without understanding who used what
• Incorrect joins: pulling metrics from unlinked or misaligned objects

4. Business Applications Require Agents, Not Just Retrieval

To work effectively with Salesforce or NetSuite, an LLM needs to act more like an analyst than a search engine. That means:

• Understanding the schema: modeling how tables relate to one another
• Generating executable queries: translating questions into SQL or API calls
• Validating assumptions: asking follow-ups like, “By enterprise, do you mean revenue over $X?”
• Reconciling messy data: dealing with missing fields, overrides, duplicates
• Synthesizing across dimensions: time, teams, geographies, products\
• Perform Sanity checks: Does the answer to this question even make sense?

This is agentic behavior — dynamic, iterative, and context-aware — not document retrieval.

5. MCP helps, but it is far from solving the problem

Model Context Protocol (MCP) is a meaningful step forward in making LLMs enterprise-aware. Like APIs expose functionality to software, MCP exposes tools, capabilities,  schemas, and metadata to language models in a structured format. This gives LLMs access to external systems without relying solely on prompt tricks or brittle inference.

However, MCP enables access, not understanding. It shows what’s available and what you can do with it, but doesn’t help the model determine relevance, handle ambiguity, or reason through messy, inconsistent data. Just as APIs don’t explain when or why to call a function, MCP doesn’t guide logic or prioritization.

Critically, MCP doesn’t define relationships between data objects — it lists components, not how they connect. It also lacks business logic, which often lives outside the schema in the minds of teams and operational workflows.

In short, MCP is a valuable interface layer — but true enterprise-grade LLM performance still requires orchestration, reasoning, and semantic context on top.

You Need More Than a Model — You Need an intelligence layer

The real power of LLMs in enterprise systems comes not from what they retrieve, but from what they understand and compute.

This requires:

• Schema introspection: building a live, contextual map of the data model
• Business-aware prompting: encoding key metrics, definitions, and thresholds
• Code generation and execution: writing and running queries to fetch precise answers
• Cross-referencing and sanity check: making sure the answer makes sense in the context of the business and other data points.
• Interpretation and explanation layers: summarizing the results back to the user in natural language and be able to explain why this is the answer and how you go to it.

In other words, you need a data reasoning layer—a brain that knows how to ask, think, and respond.

Closing Thoughts

LLMs are transformational, but they don’t come preloaded with knowledge of your Salesforce org, your NetSuite setup, your data warehouse structure or your custom definitions of customer health and churn. They need help.

So when someone says “just do RAG on our CRM,” remind them: RAG is for finding answers in documents. Business data needs a thinking partner. One that understands not just language, but structure, logic, and messy human processes baked into messy digital tables.

Enterprise AI isn’t about feeding the model more data. It’s about giving it the tools and context to understand that data, however complex and imperfect it may be. Once that’s in place, and with today’s AI breakthroughs, the possibilities are limitless.