Developing Chunking Strategies for AI Retrieval

A developer can implement different chunking behavior for different content types, but teams usually don’t create a separate algorithm for every kind of document.

They normally build:

• A content classification or routing layer
• A small set of reusable chunking policies
• Rules that map each content type to the appropriate policy
• Retrieval tests that show whether those rules work

The documentation engineer helps define what information must stay together. The developer turns those decisions into ingestion code or configuration.

Start by separating file format from content type

File format and content type are related, but they aren’t the same.

A file format describes how content is stored or delivered:

• HTML
• Markdown
• PDF
• JSON
• YAML

A content type describes what the documentation is intended to do:

• Procedure
• Troubleshooting article
• API reference
• Conceptual explanation
• Release notes
• Frequently asked questions

A PDF could contain a conceptual guide, a procedure, a reference table, or all three. A Markdown developer document could be a tutorial, API overview, or troubleshooting page.

The file format determines what structure the ingestion pipeline can extract. The content type helps determine where the useful retrieval boundaries are.

For example, a PDF parser may detect headings, paragraphs, page numbers, images, and tables. The chunking policy then determines whether to divide that extracted content by heading, page, table, token length, or some combination.

Layout-aware tools can extract structural elements before chunking. Azure AI Search, for example, can use document layout to identify headings and divide content into sections based on paragraphs and sentences. Another example is the Unstructured library, which can partition documents into elements such as titles, narrative text, lists, and tables before applying a chunking policy.

What the collaboration looks like

Consider a company with the following documentation:

• Help center articles
• PDFs linked or embedded within those articles
• Developer guides
• API reference documentation
• Tutorials
• Troubleshooting pages

The developer and documentation engineer need to understand both the content and the technical capabilities of the ingestion pipeline before choosing chunking rules.

The documentation engineer describes the content

The documentation engineer identifies:

• Which content types exist
• How reliably they’re structured
• What information must remain together
• Which metadata is available
• What users commonly ask
• What source should appear in a citation

For example:

In troubleshooting articles, the symptom, likely cause, and resolution need to stay together. Retrieving only the resolution could produce irrelevant or unsafe advice.

For API reference documentation, the requirement might be different:

Parameters must retain the endpoint name, HTTP method, API version, and authentication requirements.

This is content and retrieval design work. It requires knowledge of how the documentation is written, how its parts relate to each other, and how users search for information.

The developer inspects what the pipeline can detect

The developer determines whether the ingestion process can identify:

• HTML headings
• Markdown sections
• Ordered lists
• Code blocks
• OpenAPI operations
• PDF pages
• Tables
• Image captions
• Document titles
• Parent-child relationships

The developer may use separate parsers for HTML, PDFs, Markdown, and OpenAPI. Those parsers can convert the source content into a shared internal structure containing the text, element type, source URL, heading path, and other metadata.

The chunking logic can then operate on that structure rather than treating every source as an undifferentiated block of text.

Together, they define a small set of policies

The team doesn’t necessarily need one policy for every content type. Several content types may share a policy when their structures and retrieval needs are similar.

A simplified configuration might look like this:

chunking_policies:
  help_procedure:
    split_on: headings
    preserve:
      - numbered_lists
      - warnings
      - prerequisites
    fallback_max_tokens: 700

  conceptual_doc:
    split_on: headings
    allow_section_merging: true
    fallback_max_tokens: 1000

  api_operation:
    split_on: operation
    include_metadata:
      - endpoint
      - http_method
      - api_version
      - authentication

  pdf_layout:
    split_on:
      - headings
      - tables
      - page_boundaries
    preserve_page_number: true

The numbers are illustrative. The team would test them against its documentation and retrieval use cases rather than treating them as universal defaults.

Example: A help center article with an embedded PDF

Consider a help article titled Configure single sign-on.

The article contains:

• Prerequisites
• Steps for configuring the identity provider
• A warning about certificate expiration
• Troubleshooting guidance
• A linked PDF containing a Security Assertion Markup Language, or SAML, attribute mapping table

The system may process the HTML article and the PDF separately.

Chunking the HTML article

The ingestion pipeline identifies the article as a help center procedure.

The chunking policy might:

• Start a new chunk at each major heading
• Keep a numbered sequence together when possible
• Keep a warning with the step it affects
• Add the article title and heading path to every chunk
• Use a size limit only when a section becomes too large

The resulting chunks might look like this:

Configure single sign-on
> Prerequisites

Configure single sign-on
> Configure your identity provider
> Steps 1–4

Configure single sign-on
> Configure your identity provider
> Steps 5–7
> Certificate warning

Configure single sign-on
> Troubleshoot sign-in failures

This is structure-aware chunking. The heading provides the first boundary, while a length limit handles unusually large sections.

Chunking the embedded PDF

The ingestion pipeline shouldn’t automatically paste the PDF into the article text and treat it as one long continuation. The PDF is a separate source with its own structure, pages, tables, and citation requirements.

The PDF pipeline might:

• Extract section headings
• Detect the SAML mapping table
• Keep the table separate from surrounding prose
• Preserve the PDF title and page number
• Add the parent help article URL as related-source metadata

The table might become a chunk like this:

Source: SAML attribute mapping guide
Page: 6
Related article: Configure single sign-on

Table: Required SAML attributes
emailAddress | Required | User's primary email
firstName    | Required | User's given name
...

Tables need special handling because flattening their cells into ordinary paragraph text can destroy the relationships between headings, columns, rows, and values. The Unstructured library, for example, keeps table elements separate from non-table content and creates dedicated table chunks when a table exceeds the configured size.

A user asking, “Which SAML attributes are required?” might retrieve the PDF table chunk. A user asking, “Why does SSO fail after certificate rotation?” might retrieve the troubleshooting section from the HTML article.

Both sources cover the same general subject, but their structure and retrieval needs require different treatment.

Example: Developer documentation

Developer documentation is too broad to map to one chunking strategy. A developer portal may contain API reference, tutorials, conceptual documentation, SDK guides, and code samples.

API reference

Suppose an OpenAPI specification defines this operation:

POST /v2/chat

The most useful structural unit may be the API operation rather than the rendered webpage.

A chunk could include:

• Operation summary
• HTTP method and path
• Authentication
• Request body overview
• Important parameters
• Response summary
• Error behavior

If the parameter or schema sections are large, the developer might create smaller child chunks:

Parent chunk:
POST /v2/chat
Overview, authentication, request and response summary

Child chunk:
POST /v2/chat
Request parameter: messages

Child chunk:
POST /v2/chat
Request parameter: stream

Child chunk:
POST /v2/chat
Response errors

Each child chunk inherits metadata such as:

{
  "doc_type": "api_reference",
  "operation_id": "chat",
  "method": "POST",
  "path": "/v2/chat",
  "api_version": "v2"
}

This structure allows the retrieval system to find a precise parameter description without losing the endpoint it belongs to.

A parent-child or hierarchical approach can retrieve a small chunk that closely matches the query, then supply the larger parent section as context. Amazon Bedrock supports hierarchical chunking with smaller child chunks and larger parent chunks. LlamaIndex provides a similar hierarchy in which child nodes retain references to their parent nodes.

Developer tutorials

Now consider a tutorial titled Build a streaming chat interface.

It contains:

• Prerequisites
• SDK installation
• Authentication setup
• A complete code example
• An explanation of streaming events
• Error handling

The article shouldn’t necessarily be chunked like API reference documentation.

The policy might preserve:

• Each code block with its explanation
• A warning with the affected step
• Prerequisites as their own section
• Closely related steps as a group

A search for “How do I handle the stream-end event?” should retrieve the explanation and relevant code. A search for “How do I install the Python SDK?” should retrieve the setup section.

The page may use the same general procedure-aware policy as a help center tutorial, even though the subject and source platform are different.

Conceptual developer guides

A page explaining How streaming responses work may need larger chunks because one paragraph introduces ideas used later in the section.

The team might divide the page by heading but allow short adjacent subsections to be combined. The Unstructured library’s title-based chunking follows this general pattern: it preserves section boundaries while allowing small elements within the same section to be combined until the chunk reaches its configured size.

This is one reason content type alone can’t determine the entire strategy. A developer tutorial and a conceptual developer guide may use the same file format, but their internal relationships differ.

Does the developer write separate code for every type?

Sometimes, but the system usually applies a limited number of parsers and policies through routing rules.

A simplified implementation might look like this:

if file_format == "pdf":
    elements = parse_pdf_with_layout(file)
elif file_format == "html":
    elements = parse_html(file)
elif source_type == "openapi":
    elements = parse_openapi(file)

if doc_type == "api_reference":
    chunks = chunk_by_api_operation(elements)
elif doc_type in ["procedure", "tutorial"]:
    chunks = chunk_by_heading_and_steps(elements)
elif doc_type == "troubleshooting":
    chunks = chunk_by_problem_resolution_unit(elements)
else:
    chunks = chunk_by_heading_with_size_fallback(elements)

A production implementation may use configuration rather than a long sequence of if statements. The underlying idea is the same: route the content to an appropriate parser, then apply a chunking policy based on its structure and retrieval needs.

A team may end up with four or five broad policies:

Policy	Typical content
Section-based	Conceptual documentation, product overviews, and general help articles
Procedure-aware	Tutorials, setup guides, and task-based help
Record-based	Frequently asked questions, release note entries, and glossary terms
Schema-aware	API reference, command-line interface commands, and configuration properties
Layout-aware	PDFs, tables, forms, and scanned documents

Several content types can share one policy. A separate policy is useful when a content type has different structural boundaries, context requirements, or retrieval risks.

How the team decides whether the strategy works

A chunking strategy isn’t finished when the ingestion job runs without errors. The team needs to test whether the resulting chunks provide useful retrieval context.

The documentation engineer can create representative questions such as:

• What SAML attributes are required?
• How do I rotate the SSO certificate?
• What happens when stream is set to true?
• Which API version supports this parameter?
• Why does the desktop client show error 403?
• What changed in version 4.8?

For each question, the team checks:

• Did the correct chunk appear?
• Did it contain enough context to answer the question?
• Did it include unrelated material?
• Were warnings and prerequisites preserved?
• Did the citation point to the right article, PDF page, or API operation?
• Did multiple chunks repeat the same content?
• Did a table survive extraction in a usable form?

The results may lead the team to adjust:

• Chunk boundaries
• Maximum chunk size
• Overlap
• Metadata
• Parent-child relationships
• Parser behavior
• The source documentation itself

Microsoft’s chunking guidance recommends evaluating the content’s structure and token sizes rather than assuming one splitting method or setting will work for every source.

The documentation engineer defines the meaningful information units and identifies what could be lost when the content is divided. The developer builds the parsers, routing rules, metadata handling, and chunking policies that preserve those units. Both evaluate the results using real user questions.

That shared process is the chunking strategy.