​

Overview

When working with large amounts of text, like long articles, books, or documents, we often need to break them down into smaller, more manageable pieces. This process is crucial for many applications, especially when feeding text into Artificial Intelligence (AI) models, which often have limits on how much text they can process at once.

Think of it like reading a very long book. Instead of trying to read it all in one go, you read it chapter by chapter, or even paragraph by paragraph. Text splitting does something similar for computers.

Key Goals:

1. Manageability: Break down large text into smaller chunks that systems can handle efficiently.
2. Context Preservation: Try to keep related information together within the same chunk as much as possible.
3. Efficiency: Prepare text for tasks like searching, summarizing, or analysis by AI.

​

Core Concepts

Let’s define a few important terms:

1. Chunk: A smaller piece of text resulting from the splitting process.
2. Chunk Size: The target size for each chunk. This is often measured in tokens (the basic units AI models use to “read” text, roughly equivalent to words or parts of words) or sometimes just characters or words.
3. Chunk Overlap: To avoid losing context at the boundaries where text is split, we sometimes repeat a small portion of the end of one chunk at the beginning of the next chunk. Imagine the last sentence of one chapter being repeated as the first sentence of the next – this helps maintain the flow of information.
4. Segmentation: This is the first step in splitting. It’s about identifying the natural boundaries or units within the text before grouping them into chunks. Common segmentation methods include breaking the text down by:
   • Words
   • Sentences
   • Paragraphs
   • Specific structural elements (like Markdown headers or code blocks)

​

How Does Segmentation Work?

Segmentation identifies the initial building blocks we’ll use to create our final chunks.

• Word Segmentation: Simply breaks the text into individual words. This is very basic and often loses the meaning conveyed by full sentences.
  • Example: “Text splitting is useful.” -> ["Text", "splitting", "is", "useful."]
• Sentence Segmentation: Uses punctuation (like periods, question marks, exclamation points) and grammatical rules to identify sentence boundaries. This is a very common and effective method as sentences usually represent complete thoughts.
  • Example: “Text splitting is useful. It helps AI process information.” -> ["Text splitting is useful.", "It helps AI process information."]
• Paragraph Segmentation: Treats blocks of text separated by blank lines as individual units. This is good for preserving topics discussed within a single paragraph.

  • Example:

    Copy

    Ask AI

    Text splitting is useful.
    It helps AI process information.
    
    Another key concept is chunk overlap.
    

    -> ["Text splitting is useful.\nIt helps AI process information.", "Another key concept is chunk overlap."]

​

How Does Splitting (Chunking) Work?

Once the text is segmented into initial units (like sentences or paragraphs), the splitting process groups these units together to form the final chunks, trying to respect the Chunk Size and Chunk Overlap settings.

Let’s walk through a simple example using Sentence Segmentation and aiming for a Chunk Size of around 15 words, with a Chunk Overlap of 5 words.

Original Text: “Text splitting breaks down large documents. This makes them easier for AI to understand. AI models often have input limits. Therefore, chunking is essential for analysis. Proper splitting maintains context.”

1. Segmentation (by Sentence):

1. “Text splitting breaks down large documents.” (6 words)
2. “This makes them easier for AI to understand.” (8 words)
3. “AI models often have input limits.” (6 words)
4. “Therefore, chunking is essential for analysis.” (6 words)
5. “Proper splitting maintains context.” (4 words)

2. Chunking (Target Size: ~15 words):

• Chunk 1:
  • Add Sentence 1: “Text splitting breaks down large documents.” (Current size: 6 words)
  • Add Sentence 2: “This makes them easier for AI to understand.” (Current size: 6 + 8 = 14 words)
  • Stop here, as adding the next sentence (6 words) would exceed the target.
  • Chunk 1 Text: “Text splitting breaks down large documents. This makes them easier for AI to understand.”
• Chunk 2:
  • Overlap: Take the last ~5 words from Chunk 1: “…for AI to understand.”
  • Add Sentence 3: “AI models often have input limits.” (Current size: 5 (overlap) + 6 = 11 words)
  • Add Sentence 4: “Therefore, chunking is essential for analysis.” (Current size: 11 + 6 = 17 words)
  • Stop here.
  • Chunk 2 Text: “…for AI to understand. AI models often have input limits. Therefore, chunking is essential for analysis.” (Note: The exact overlap text might vary based on the specific implementation, often using tokens instead of words).
• Chunk 3:
  • Overlap: Take the last ~5 words from Chunk 2: “…essential for analysis.”
  • Add Sentence 5: “Proper splitting maintains context.” (Current size: 3 (overlap) + 4 = 7 words)
  • End of text.
  • Chunk 3 Text: “…essential for analysis. Proper splitting maintains context.”

​

Different Splitting Strategies

Different situations call for different splitting strategies:

1. Sentence Splitter:
   • How it works: Segments by sentence, then groups sentences into chunks based on size.
   • Best for: Plain text documents, articles, essays where sentence structure is important.
   • Example: The detailed example above used this logic.
2. Markdown Splitter:
   • How it works: Understands Markdown formatting (like # Headers, * Lists, code blocks, tables). It tries to keep these structural elements intact and often starts new chunks at major headers. It might segment by headers, then code blocks, then tables, then paragraphs, and finally sentences if needed to meet size constraints.
   • Best for: Documentation files (like README.md), web content written in Markdown, technical articles. It helps preserve the logical structure of the document.
   • Example: It would try not to split a code block or a table across two different chunks. A # Top Level Header would likely signal the start of a new chunk.
3. Dynamic Splitter:
   • How it works: Uses a chosen segmentation method (words, sentences, or paragraphs) and then employs a more sophisticated algorithm (often dynamic programming) to find the “optimal” places to split the text. It calculates a “cost” for different potential chunk combinations, aiming to create chunks that are as close as possible to the target Chunk Size while minimizing awkward breaks.
   • Best for: Situations where achieving highly consistent chunk sizes is critical, potentially offering a good balance even with varied text structures.

​

Choosing the Right Method

• For standard prose like articles or books, the Sentence Splitter is often a good starting point.
• If your document has clear Markdown structure (headers, code, lists), the Markdown Splitter is likely the best choice to preserve that structure.
• If you need very consistent chunk sizes and are willing to potentially trade off some structural awareness for size consistency, the Dynamic Splitter (often configured with sentence or paragraph segmentation) can be effective.

By understanding segmentation and splitting, you can better prepare your text data for analysis, search, or AI processing, ensuring that the meaning and structure of your original documents are respected as much as possible.