Text chunking

​

Overview

1. Manageability: Break down large text into smaller chunks that systems can handle efficiently.
2. Context Preservation: Try to keep related information together within the same chunk as much as possible.
3. Efficiency: Prepare text for tasks like searching, summarizing, or analysis by AI.

​

Core Concepts

1. Chunk: A smaller piece of text resulting from the splitting process.
2. Chunk Size: The target size for each chunk. This is often measured in tokens (the basic units AI models use to “read” text, roughly equivalent to words or parts of words) or sometimes just characters or words.
3. Chunk Overlap: To avoid losing context at the boundaries where text is split, we sometimes repeat a small portion of the end of one chunk at the beginning of the next chunk. Imagine the last sentence of one chapter being repeated as the first sentence of the next – this helps maintain the flow of information.
4. Segmentation: This is the first step in splitting. It’s about identifying the natural boundaries or units within the text before grouping them into chunks. Common segmentation methods include breaking the text down by:
   • Words
   • Sentences
   • Paragraphs
   • Specific structural elements (like Markdown headers or code blocks)

​

How Does Segmentation Work?

• Word Segmentation: Simply breaks the text into individual words. This is very basic and often loses the meaning conveyed by full sentences.
  • Example: “Text splitting is useful.” -> ["Text", "splitting", "is", "useful."]
• Sentence Segmentation: Uses punctuation (like periods, question marks, exclamation points) and grammatical rules to identify sentence boundaries. This is a very common and effective method as sentences usually represent complete thoughts.
  • Example: “Text splitting is useful. It helps AI process information.” -> ["Text splitting is useful.", "It helps AI process information."]
• Paragraph Segmentation: Treats blocks of text separated by blank lines as individual units. This is good for preserving topics discussed within a single paragraph.
  • Example:
    Copy

    Ask AI

    Text splitting is useful.
    It helps AI process information.
    
    Another key concept is chunk overlap.
    

    -> ["Text splitting is useful.\nIt helps AI process information.", "Another key concept is chunk overlap."]

​

How Does Splitting (Chunking) Work?

1. “Text splitting breaks down large documents.” (6 words)
2. “This makes them easier for AI to understand.” (8 words)
3. “AI models often have input limits.” (6 words)
4. “Therefore, chunking is essential for analysis.” (6 words)
5. “Proper splitting maintains context.” (4 words)

• Chunk 1:
  • Add Sentence 1: “Text splitting breaks down large documents.” (Current size: 6 words)
  • Add Sentence 2: “This makes them easier for AI to understand.” (Current size: 6 + 8 = 14 words)
  • Stop here, as adding the next sentence (6 words) would exceed the target.
  • Chunk 1 Text: “Text splitting breaks down large documents. This makes them easier for AI to understand.”
• Chunk 2:
  • Overlap: Take the last ~5 words from Chunk 1: “…for AI to understand.”
  • Add Sentence 3: “AI models often have input limits.” (Current size: 5 (overlap) + 6 = 11 words)
  • Add Sentence 4: “Therefore, chunking is essential for analysis.” (Current size: 11 + 6 = 17 words)
  • Stop here.
  • Chunk 2 Text: “…for AI to understand. AI models often have input limits. Therefore, chunking is essential for analysis.” (Note: The exact overlap text might vary based on the specific implementation, often using tokens instead of words).
• Chunk 3:
  • Overlap: Take the last ~5 words from Chunk 2: “…essential for analysis.”
  • Add Sentence 5: “Proper splitting maintains context.” (Current size: 3 (overlap) + 4 = 7 words)
  • End of text.
  • Chunk 3 Text: “…essential for analysis. Proper splitting maintains context.”

​

Different Splitting Strategies

1. Sentence Splitter:
   • How it works: Segments by sentence, then groups sentences into chunks based on size.
   • Best for: Plain text documents, articles, essays where sentence structure is important.
   • Example: The detailed example above used this logic.
2. Markdown Splitter:
   • How it works: Understands Markdown formatting (like # Headers, * Lists, code blocks, tables). It tries to keep these structural elements intact and often starts new chunks at major headers. It might segment by headers, then code blocks, then tables, then paragraphs, and finally sentences if needed to meet size constraints.
   • Best for: Documentation files (like README.md), web content written in Markdown, technical articles. It helps preserve the logical structure of the document.
   • Example: It would try not to split a code block or a table across two different chunks. A # Top Level Header would likely signal the start of a new chunk.
3. Dynamic Splitter:
   • How it works: Uses a chosen segmentation method (words, sentences, or paragraphs) and then employs a more sophisticated algorithm (often dynamic programming) to find the “optimal” places to split the text. It calculates a “cost” for different potential chunk combinations, aiming to create chunks that are as close as possible to the target Chunk Size while minimizing awkward breaks.
   • Best for: Situations where achieving highly consistent chunk sizes is critical, potentially offering a good balance even with varied text structures.

​

Choosing the Right Method

• For standard prose like articles or books, the Sentence Splitter is often a good starting point.
• If your document has clear Markdown structure (headers, code, lists), the Markdown Splitter is likely the best choice to preserve that structure.
• If you need very consistent chunk sizes and are willing to potentially trade off some structural awareness for size consistency, the Dynamic Splitter (often configured with sentence or paragraph segmentation) can be effective.