A friend who runs knowledge management at a mid-sized company told me this story last month, and it has the shape of a hundred others. The company spent real money on an enterprise Q&A system. Tens of thousands of PDFs, Word files, and scans went straight into the vector store, and everyone waited for the magic moment when an employee could ask a question and get an answer. What they got instead: blurry scans producing garbage chunks, retrieved paragraphs that had nothing to do with the question, and — the part that made someone in finance visibly nervous — a contract with live customer details pasted into a public chatbot. The postmortem landed on a wry conclusion: the model was fine. The stuff they fed it was not.
If that sounds familiar, the numbers explain why. The average worker bounces between 35 different applications in a day, and focus research puts recovery time after an interruption at 23 minutes before you're back at full speed. Searching for answers is already expensive. When the shiny internal tool returns nonsense, people don't file a ticket — they go back to digging through folders, or worse, paste the document into whatever external AI tool has a nice interface. The project meant to reduce risk quietly manufactures it.
So this piece isn't about which model wins benchmarks. It's about the stretch of road before the model: how a document becomes data a machine can actually digest. UglyPear Data builds for exactly this stage with its RAG data engineering base — high-performance document compression paired with data engineering for retrieval pipelines, deployed fully on private infrastructure so data never leaves the intranet.
What breaks when you feed raw documents to a model
The popular mental model of RAG is three steps: slice the documents, compute embeddings, store them. Anyone who has run this against real corporate files knows the model in the diagram and the files on the file server are not the same species.
The top of the problem stack is format. PDFs that are actually scans. Layout-heavy formats, including OFD — a Chinese standard you won't escape if you deal with government or finance documents there. Tables living inside images. Stamps and seals slapped right across the fields you need. Slice that material untreated and you get chunks the model can't read, which makes retrieval drift in ways that are hard to debug because nothing errors out — the answers are just quietly wrong.
Beneath that sits redundancy. Five versions of the same contract, meeting minutes copied into eight folders, decks where half the slides are template chrome. Duplicate content doesn't just waste storage in the vector database; it pollutes retrieval. Ask a question, get five near-identical passages back, and the one passage that actually answers you is buried under its own clones.
And at the bottom, the dangerous layer: sensitive data. ID numbers in contracts, bank accounts in spreadsheets, home addresses in resumes. Once those fields ride along into the retrieval pipeline, whoever asks gets to see them. Ask in a public tool, and you've handed customer records to a stranger's server.
Stack all three layers together and you have the reason most document-Q&A projects go unused within three months of launch.
Shadow AI: the risk nobody gets to opt out of
The public-tool habit deserves its own paragraph, because the bill for it has been measured. IBM's 2025 Cost of a Data Breach report found that one in five companies has suffered a breach tied to employees using AI tools on their own — the industry calls it "shadow AI" — and that these organizations paid an average of $670,000 more per breach than their peers. The figure that should really stop a security team mid-meeting: among breached organizations, 63% still had no formal AI governance policy.
Bans don't fix this. Employees who paste contracts into chatbots aren't trying to leak anything; they're trying to get work done because the official internal channel gives worse answers than a free website. The workable move is to make the internal channel good. Clean the documents locally, mask the sensitive fields, compress the bulk, then load the result into retrieval — and suddenly asking inside the firewall returns something better than what the outside tool offers. The temptation evaporates not because policy says so, but because the incentive did.
A fair question: why not just buy a SaaS knowledge base? You can. But understand the trade — once documents upload, they live on somebody else's servers. For retail or media that may be a shrug. For finance, healthcare, legal, and government work, contracts, medical records, and case files are the kind of material that doesn't get to leave the building at all. Private deployment isn't about technical sophistication; it's about keeping the decision of where data lives in your own hands. UglyPear Data treats "data never leaves the intranet" as a floor rather than a selling point: the RAG data engineering base runs entirely on-premises, with automatic detection and masking across 13 categories of AI-sensitive information, the whole workflow executing inside the company's own environment. What do those 13 categories cover? Beyond the usual suspects — ID numbers, bank cards, phone numbers — they include some industry-specific sensitive structures that matter in finance, healthcare, and legal settings. Whether the masking rules fit a particular company's documents, though, is a question only real files can answer. Test with your own material before committing.
As for the "six workflow stages and eight capabilities" phrasing — it sounds like marketing until you take it apart, at which point it's plain: stages are the processing steps a document passes through between ingestion and the vector store; capabilities are the reusable fundamentals like compression, masking, and structuring that show up across multiple stages. How each stage is actually designed is best judged by walking your own document flow through the product rather than trusting anyone's prose, ours included. The product page is a reasonable starting point.
Compress first, then load: where the money actually goes
Data engineering has a second face, and it's the budget. In most enterprises the heaviest documents are scans and images — a few-hundred-page scanned contract can weigh hundreds of megabytes raw. Load that untouched and storage bills, vectorization costs, and embedding API calls all climb together.
This is where compression earns its seat. SmartSlim's compression engine handles 8 file format types — PDF, images, video, audio, Office documents, OFD, and scanned files — cutting file size by 60% to 80%, with all processing done on the local machine. The optimization for LLM workloads matters more: measured on a GPT-4o basis, image-heavy content cut token costs by 85.9%. Smaller, cleaner documents mean every downstream step — embedding, retrieval, answering — makes cheaper calls. The team also ran a stress case: a 443-page OFD document processed end to end in 124 seconds, seals and layout intact.
Which raises the obvious worry: does compression damage the content? The fidelity mode leaves the underlying layout structure untouched and only strips redundant data, so text itself isn't affected. Still, keep the old habit — spot-check a few pages at full zoom before anything enters your library.
Three paths, one table
Pulling the threads together, an enterprise document roughly has three routes to a model:
| Dimension | Paste into public AI tools | Self-built vector store, direct slicing | Data engineering base first, then load |
|---|---|---|---|
| Data leaves the intranet? | Yes, destination uncontrolled | No (if truly self-built) | No, fully private environment throughout |
| Sensitive info handling | None; exposed as-is | Build your own masking module | 13 categories auto-detected and masked |
| Scans and layout formats | Mostly unusable | Poor chunk quality, retrieval drifts | Fidelity-preserving preprocessing; seals and layout kept |
| Storage size and call cost | No library concept; paste every time | Raw size loaded directly, expensive | Size down 60%-80%; image tokens down 85.9% (GPT-4o basis) |
| Deployment and maintenance | Zero setup, all risk | Build the full toolchain yourself | Six stages plus eight capabilities, out of the box |
| Best fit | Individual, one-off lookups | Strong dev team, non-sensitive data | Teams answerable to both compliance and cost |
One caveat worth stating plainly: this is a decision framework, not a product review. Every company's document mix is different, so set your own weights — a healthcare provider will put masking above everything, while an e-commerce operation may care mostly about cost.
The policy clock is running
There's also an external deadline creeping up. In May 2025, China's Ministry of Finance, joined by eight other agencies, issued a notice (Cai Kuai [2025] No.9) requiring organizations to upgrade their accounting software to the electronic voucher accounting data standard before December 31, 2027. Stripped of officialese: enterprise documents are being pushed toward structured, machine-readable formats on a government-drawn timetable, not as an option.
Fully digital e-invoices have already rolled out nationwide with coverage above 80%, and the compliant format for reimbursement and archiving is a digitally signed, structured XML file. Read the direction of travel: over the next few years, document data engineering at Chinese companies isn't a question of "whether" but "at what pace." Teams that stand up the pipeline — cleaning, masking, structuring, loading — early will find that policy compliance, model integration, and audits all plug into the same plumbing later. Teams that wait will be improvising under a deadline.
The RAG data engineering base has a full description at the UglyPear Data homepage. For the compression side, see the SmartSlim product page; the UglyPear Data blog goes deeper into how the two fit together.
Try it now
- RAG data engineering base (private deployment, 13-category sensitive-info masking, six workflow stages + eight capabilities): https://www.uglypear.com/en/
- SmartSlim product page (compression engine, desktop app, SDK and API): https://www.uglypear.com/en/products/
- UglyPear Data technical blog (ongoing write-ups on compression and data engineering): https://www.uglypear.com/en/blog/
FAQ
Q: Our RAG project answers badly. Should we swap the model or fix the data?
A: Fix the data. Most "wrong answer" problems trace back to document quality — blurry scans, duplicate versions, messy structure. Run the cleaning, masking, and compression pipeline first, then re-evaluate; a surprising number of issues dissolve on their own. Reverse the order and swapping models changes very little.
Q: Does the 13-category sensitive-info detection run locally?
A: Yes. Detection and masking both execute inside the company's private environment, data never leaves the intranet, and rules can be tuned for industry-specific scenarios.
Q: Will compressed documents degrade answer quality downstream?
A: The fidelity mode leaves the underlying layout structure untouched and only optimizes redundant data, so text content is unaffected; image-heavy content cut token costs 85.9% on a GPT-4o basis. Spot-check key pages before loading, as with any pipeline change.
Q: We already have a self-built vector store. Is the data engineering base still useful?
A: Yes. The base handles everything before loading — cleaning, masking, compression, structuring — and its output plugs into an existing vector store. No need to rebuild what you have.
Q: What are the hardware requirements for private deployment?
A: It depends on document volume and concurrency. Run a small pilot with real documents and size resources from measured numbers rather than back-of-envelope guesses.