Data Preparation for AI Agents: 5 Steps That Most Affect Accuracy

The bitter truth: 85% of failed AI projects cite data quality as the root cause. MIT's Project NANDA research went even further: 95% of organizations that deployed generative AI saw zero measurable ROI.

Most "data preparation for AI" content still describes classic machine learning: missing values, outliers, normalization. However, nowadays, 9 out of 10 teams aren't training a model from scratch. They build an AI agent or chatbot on top of their own company documents. And that's a different game with different rules.

When you're building a classic ML model, "dirty data" means gaps, anomalies, and wrong types. When you're building an AI agent on RAG (retrieval-augmented generation), the model doesn't train on your data — it reads it in real time to form an answer. And dirty data looks different here:

Two documents with different answers to the same question. The old return policy and the new one both sit in the base. The agent pulls fragments from both and produces a contradictory or fabricated answer.

Outdated content nobody removed. Last quarter's pricing, a canceled promotion, a job posting that's already closed — all of it lives in the knowledge base and erodes trust in the answers.

Bad chunking. A document was split mechanically, ignoring where the meaning actually breaks — half the needed context ends up in a different fragment that search never reaches.

Missing metadata. The agent doesn't know which document applies to which region or language — and answers the wrong user.

PII and sensitive data without access control. If nobody flagged or restricted sensitive information, sooner or later the agent will quote exactly what it shouldn't have shown — and that's no longer a issue of data quality for AI, it's a compliance one.

Data cleaning for machine learning means handling missing values, outliers, duplicates, and inconsistent formats in structured training data, then normalizing it so a model can learn from it. Data labeling for machine learning means assigning the correct tags, classes, or annotations to training examples so a model learns the right associations. Both are essential — if you're training or fine-tuning a model, that groundwork still applies.

This U.S. auto-parts manufacturer's first attempt at an AI assistant — a Claude MCP connector plugged straight into NetSuite — looked like the fast path: plug-and-play access to ERP data, minimal engineering effort. It fell apart on contact with real data. NetSuite held order and customer information that wasn't standardized: PO numbers appeared as "PO-1234," "1234," or other variants depending on whether the order came through Shopify, phone, or email. 

The S&B Filters assistant couldn't reliably interpret or retrieve the right records, response times reached 4–6 minutes, and the whole thing ran on a 40-page prompt that was nearly impossible to maintain. The fix was making the underlying data usable for AI before asking an assistant to reason over it. Once BotsCrew rebuilt the data layer, order lookups that used to take minutes dropped to seconds.

Watch our video covering:

→ Why most enterprise AI pilots stall at the integration layer

→ What MCP is and how it works

→ MCP vs API — what's actually different

→ The four-layer MCP architecture enterprises are adopting

→ A real use case: cross-system reasoning in a single Claude conversation.

Here are five things worth doing for data preparation for AI before the agent ever gets access to your documents. 

Bring all your sources into one place: Confluence, Google Drive, PDFs buried in email, old versions of the website. A typical company keeps the same information in 3 to 5 places at once, and the versions drift apart. Build an inventory of everything you have and group documents that cover the same topic, either manually or through similarity search.

For every group, someone needs to make the call: "this version is the truth, the rest go to the archive." For instance, a company might have three versions of its return policy floating around at once — 2023, 2024, and a 2025 draft. Decide which single version goes into the agent's knowledge base. Until that decision gets made, the agent treats all three as equally true — and blends them into its answers.

Data cleaning for an AI agent isn't a one-time audit — it's something you do continuously. Every domain has its rhythm: someone owns product documentation, someone owns support policies, someone owns HR content. And the review cycle differs too — pricing and promotions can change weekly, while legal policies might update once a quarter (or even less).

The best signal that data has gone stale is the agent itself. If it keeps answering one topic with low confidence or contradicting itself, that's already a clue about where to look first. Log those cases and route them to the domain owner as a priority.

This is also why a narrow pilot beats a full rollout on day one. When Kravet — a fifth-generation family business with nearly 1,000 employees across Sales, Supply Chain, Operations, and HR — rolled out an internal AI assistant, BotsCrew started with a 3-week pilot on real data handed to real users, with one brief: tell us what doesn't work. The issues that surfaced were specific and cheap to fix at that stage — the kind of thing that becomes a six-figure rebuild if it's only discovered after a full launch.

Mechanically slicing text by character count is the number one reason an agent "can't see" an answer that's technically sitting right there in the base. Split a document into fixed-size blocks without regard for structure, and a sentence gets cut mid-word, or a table or list breaks apart into fragments that don't make sense on their own.

Chunks should follow the document's headings and logical sections — a chapter, a subsection, a list item — not an arbitrary character count. Chunk size and overlap should be tuned to the content type: a short FAQ answer can stand alone as its own chunk with no overlap, while technical documentation or a contract needs more surrounding context and 10-15% overlap, or the connection between neighboring sentences gets lost. The easiest way to check whether it worked is to test retrieval against real user queries — that's usually the fastest way to spot where fragments were cut badly.

This is where source format matters as much as source content. When BotsCrew built an internal chatbot for a global nonprofit operating in crisis response and humanitarian aid, the knowledge base was hundreds of pages of manuals and technical documentation full of interdependent tables. The team had to restructure those tables and rewrite retrieval logic so the AI could interpret the interdependencies, and separately convert manuals that relied on screenshots into text, since the model couldn't read the visual instructions at all. 

Only after that groundwork — and two rounds of testing with real employees to refine the knowledge base — did the chatbot reach a state where it could resolve 65% of employee inquiries on its own, with a 9/10 satisfaction score, from a pilot that launched in three weeks.

Every fragment in the knowledge base needs to "know" about itself: which product, region, language, and effective date it belongs to. Without that layer, retrieval pulls content that's technically relevant but factually wrong — shipping terms for the US served to a user in Europe, or enterprise pricing shown to someone on the free plan.

Start with a minimum set: source, last updated date, freshness status, access level. From there you can add product, region, customer segment, or language, depending on your business. What matters isn't just having the tags — it's whether they actually shape retrieval. If a query comes in from Europe, retrieval should either prioritize or strictly filter for documents tagged "EU."

Check your data for personal and sensitive information, and set clear access levels — not every agent user should be able to see everything, even if the document technically lives in the same base. Every category of data needs its own call: delete it, mask it, or restrict access to it.

This matters most in industries with strict requirements: healthcare (HIPAA), finance, personal data in the EU (GDPR). Furthermore, access control needs to live at the retrieval layer, not just in file storage: if a document is restricted in Google Drive but the agent indexes its content without checking permissions at query time, the restriction in storage means nothing.

Natera's GenAI Results Delivery Assistant is a clear illustration of what this looks like in a high-stakes environment. The assistant, embedded in Natera's patient portal, helps patients understand genetic test results — sensitive medical data with zero tolerance for inaccuracy. Its knowledge base was built exclusively from SME-approved content (official FAQs, patient materials, and real counselor queries, all reviewed by subject matter experts) and gated so it's reachable only by authenticated patients who actually have results on file. 

On top of that, a risk-tiered guardrails system (P0–P6) filters every query and validates every output before it reaches the patient, with automatic escalation to a human counselor for anything outside scope. None of that is a model feature — it's a data governance decision made before the assistant ever went live, and it's the reason the project could touch medical data safely at all.

When companies clean up their data before launching an AI solution, the difference in results shows up immediately. We've seen this play out across BotsCrew's projects. Building an internal AI assistant for one of the largest non-profit organisations (NDA), the core of the work wasn't prompt writing — it was structuring and cleaning the internal knowledge base, and the agent ended up resolving 65% of employees' repetitive internal questions.

S&B Filters saw a similar pattern from the opposite direction: a first attempt built on unstandardized NetSuite data produced 4–6 minute response times and an unmaintainable prompt, while a rebuilt, AI-ready data layer brought order lookups down to seconds. And in customer-facing settings, an agent built on an organized knowledge base saved a California beauty retailer $250,000 a year.

A typical scenario: a team is racing toward launch and decides "we'll clean the data later, if needed." The agent goes live, users start getting contradictory or outdated answers, and trust in the product drops before it's had a chance to prove itself. Only then does the team turn to the data — and now it's firefighting.

👉 Fixing data after the fact costs two to three times more: you're rebuilding the knowledge base architecture under live load, re-indexing content, and trying to win back the trust of users who already got a few bad answers. First impressions with an AI agent form in the first few interactions — and they are hard to reverse even after the data gets fixed.

Auditing and preparing data is part of the discovery phase of a project, not a post-launch fix. Kravet's three-week pilot and S&B Filters' rebuilt data layer both worked for the same reason: the questions "what's duplicated, what's outdated, what's missing" got answered before the system was asked to perform at scale. 

That's why in our own AI agent development process, we don't start by picking a model. Instead, we start by analyzing the client's existing data: what's there, what's duplicated, what's outdated, and how to structure it so the agent can actually use it.

If you are planning to roll out an AI agent built on your company's documents, the biggest investment worth making before development starts is a data audit. At BotsCrew, this is its structured step — Discovery Phase — where we analyze your data sources, find duplicates and gaps, and put together a clear plan for how much time and effort it'll take to get your knowledge base AI-ready.

If you need a team that can clean up your data and build the AI agent at the same time — that's what BotsCrew has been doing since 2016 for companies like Samsung NEXT, Honda, and FIBA. 

Data labeling for machine learning — how is it different from RAG data prep?

Data labeling means tagging or classifying training examples so a model learns to recognize patterns — labeling images, categorizing support tickets, marking sentiment in text. It's a step in training or fine-tuning a model. RAG data prep is different: the agent doesn't train on your documents, it retrieves and reads them at query time. There's nothing to label — the work instead is deduplication, chunking, metadata, and access control, so the agent finds and reads the right content on demand.

Do I still need data cleaning for machine learning if I'm building an AI agent?

Not in the traditional sense. Classic ML data cleaning — fixing missing values, removing outliers, normalizing formats — applies to structured training data. If you're building an AI agent on top of your company's documents rather than training a custom model, that kind of cleaning mostly doesn't apply. What you need instead is document-level cleanup: resolving conflicting versions, removing outdated content, and structuring text so retrieval can actually find and use it correctly.

Is data preparation for AI agents a one-time task, like data cleaning for machine learning often is?

No — this is one of the bigger differences. A training dataset gets cleaned once, before training runs. A RAG knowledge base needs continuous upkeep, since documents, pricing, and policies keep changing after the agent goes live.