Ingesting Your Documentation Into 0xDoktor: PDFs, Markdown, and Web Pages

Before you ingest: thinking about knowledge base structure

How you structure your knowledge base affects answer quality. Before ingesting everything at once, consider:

One KB or multiple? For most teams starting out, a single knowledge base is fine. Separate knowledge bases make sense when you have meaningfully different audiences (end users vs. developers), different languages, or different product versions that should not cross-contaminate.

What content to prioritize? Start with the content that answers your most common questions. You can find this by looking at your support ticket history — the top 20 question types are your highest-priority KB content.

How fresh does it need to be? If your product changes frequently, plan for a re-ingestion process as part of your release workflow.

Ingesting PDF files

PDF is the most common format for formal documentation, policy documents, and manuals.

When 0xDoktor ingests a PDF:

1. Text is extracted using PDF parsing (tables and structured content are preserved where possible)
2. Boilerplate (page numbers, headers/footers, copyright notices) is removed
3. The clean text is chunked into overlapping segments
4. Each chunk is embedded and stored

Tips for better PDF ingestion:

• Text-based PDFs work better than scanned image-based PDFs (scanned PDFs require OCR and quality may vary)
• Well-structured PDFs with clear headings produce better chunks
• Avoid ingesting appendices with raw data tables that do not contain useful prose

Ingesting Markdown files

Markdown is the ideal format for technical documentation. Headers, code blocks, and lists are preserved during chunking, which helps the model understand structure.

When ingesting Markdown:

• # and ## headings are used as natural chunk boundaries
• Code blocks are kept intact within their surrounding prose context
• Internal links are stripped but link text is preserved

Tips for better Markdown ingestion:

• Well-structured Markdown with clear section headings produces the best chunks
• Include introductory sentences for each section — these help the model understand what the section is about
• If you have large auto-generated reference files (e.g., full API specs), consider splitting them by resource type

Ingesting web pages via URL

URL ingestion fetches a public web page and extracts the main content. This is useful for existing online documentation, help articles, or blog posts you want to include in the KB.

When ingesting from a URL:

• Navigation menus, sidebars, and footers are automatically removed
• The main content area is extracted
• Images are ignored (only text is ingested)

Tips for URL ingestion:

• Works best on clean documentation sites with clear content structure
• May not work well on pages with heavy JavaScript rendering (single-page apps where content loads dynamically)
• Check the preview after ingestion to confirm the right content was extracted

Managing updates

When your documentation changes, you need to re-ingest the updated content.

For files: re-upload the updated file. 0xDoktor automatically replaces the previous version of a document with the same filename.

For URLs: re-run the URL ingestion. 0xDoktor fetches the current version of the page and replaces the previous chunks.

Recommended workflow for documentation teams: add a step to your documentation update process that triggers re-ingestion for changed files. Many teams do this weekly or after each product release.

Checking ingestion quality

After ingesting content, use the test interface in the admin dashboard to verify quality. Ask questions that should be answered by the newly ingested content. If the answers are incorrect or missing:

• Check that the right content was ingested (review the chunk list for the document)
• Verify the document actually contains the answer — it may be missing from your documentation entirely
• If the document is large, consider splitting it into smaller, more focused files

Content that does not ingest well

Some content types produce poor results and should be excluded or handled differently:

• Raw data tables without prose: large tables of numbers without explanatory text are hard for the model to use
• Legal boilerplate: if your policy documents contain large sections of standard legal language that does not apply to user questions, exclude those sections
• Duplicate content: if the same information appears in multiple documents with slightly different wording, ingest the canonical version and remove duplicates
• Very long single-topic documents: extremely long documents (50+ pages on a single topic) should be split into sections

Keeping track of what is ingested

The knowledge base view in the admin dashboard shows every document that has been ingested, with its chunk count and ingestion date. Review this list periodically to ensure it reflects your current documentation state. Remove documents that are no longer accurate or relevant.