Chat Memory
Give your Pipelines access to Chatbot conversation history using Chat Memory nodes.
Chat Memory
Inherently, LLMs do not have "memory" - they do not remember previous conversation history. However, having a memory in a Chatbot may be useful to refer to previous contexts and outputs.
Essentially, what the chat memory node does it is takes the conversation history and if connected to an LLM, will pass the history back to the LLM so the LLM can reference it / reason with it.
Note: within the no-code builder, you must run the pipeline as a chatbot (not the "standard" run interface) to reference previous messages.
Different types of chat memory
You can select the following types of chat memory with the full down menu within the chat memory node. The default is "Token Buffer"
Full - Formatted
Returns all previous chat history, with user messages prepended with "Human" and output messages prepended with "AI". Useful for handling short conversations, but may run into token limits once conversations start becoming longer.
Full - Raw
Returns a Python list with elements in the following format: {"type": type, "message": message}, where type is "input" (i.e., Human) or "output" (i.e., AI) and message contains the contents of the message. Useful for developers looking to execute custom logic with their chat history using Transformations.
Message Buffer
Returns a set number of previous consecutive messages. This number defaults to 10 and can be changed by right-clicking the Chat Memory node and clicking "Details". Both Human messages and AI messages count toward the limit.
Token Buffer
Returns previous consecutive messages until adding an additional message would cause the total history size to be larger than the Max Tokens. The default number of Max Tokens is 2048 and can be changed by right-clicking the Chat Memory node and clicking "Details".
Using the Chat memory node
The chat memory node doesn't have any inputs but has one output. The output is usually connected to the prompt field of a LLM node or a text node. Essentially, the chat memory takes the text in historical conversations and feeds it back into a prompt for a LLM. Hence, chatbots often have this structure:
Within the prompt (either in the LLM node or in a text node), we create a variable that the chat memory node can be connected to (in the below example, it is called "{{History}}". Then we label the variable as "Conversation History" so the LLM can understand.
As a result, when the pipeline is run, the LLM will get all the information it needs to answer based on what the user wants (input block), relevant data (from the Knowledge base node), and conversation history (from the chat memory node).
Last updated
