Chunk Text Node Split text into chunks (list of text). Supports different chunking strategies like markdown-aware, sentence-based, or dynamic sizing.

​
Node Inputs

  1. Text: The text for chunking
    • Type: Text

​
Node Parameters

In the gear:
  1. Chunk Size: The size of each chunk of text in number of tokens. One token = 4 characters. The default value is 512 tokens. The value ranges from 1 to 4096.
    • Type: Text
  2. Chunk Overlap: The overlap of each chunk text in number of tokens. One token = 4 characters. The default value is 0. The value ranges from 0 to 4096.
    • Type: Text
  3. Chunk Strategy: Strategy for grouping segmented text into final chunks. sentence: groups sentences, markdown: respects markdown structure (headers, code), dynamic: optimizes breaks for size using chosen segmentation method (see below). The default option is Markdown.
    • Type: Dropdown
If Dynamic is selected as the chunk strategy:
  1. Segmentation Method: The method to break text into units before chunking. words: splits by word, sentences: splits by sentence boundary, paragraphs: splits by blank line/paragraph. The default option is words.
    • Type: Dropdown

​
Node Outputs

  1. Chunks: The chunked text in a list
    • Type: List<Text>
    • Example usage: {{chunking_0.chunks}}

​
Example

The below example shows a pipeline that takes a blog, chunks it into a list of text, and summarizes each chunk.
  1. Text Node: Contains the text
    • Text: The text from the blog
  2. Chunk Text Node: Splits the text into chunks of text based on the chunk size and overlap
    • Text: {{text_0.text}}
  3. Summarizer Node: Summarizes each chunk in the list (list mode applies the operation, in this case, summarization, onto each item in the list)
    • List Mode: True
    • Text for summarization: {{chunking_0.chunks}}
  4. Output: Display the list of summaries
    • Output: {{summarizer_0.summary}}
Chunk Text Example