LLM Node

The LLM node allows you to connect to a variety of LLM providers and use them in your pipelines.

​

Node Inputs

1. System (Instructions): Specify how you would like the LLM to respond (e.g., the style). It is common to specify here how the LLM should utilize the data received in Prompt.
   • Type: Text
2. Prompt: Provide the data the LLM should consider. Type {{ to open the variable builder.
   • Type: Text

​

Node Parameters

On the face of the node:

1. Model: Select the model to use from the provider

In the gear:

1. Provider: Change the provider of the LLM.
2. Max tokens: The maximum number of output tokens for each LLM run.
3. Temperature: The diversity of the LLM generation. To have more diverse or creative generations, increase the temperature. To have more deterministic response, decrease the temperature.
4. Top P: The Top P parameter constrains how many tokens the LLM considers for generation at each step. For more diverse responses increase top p towards a maximum value of 1.0. This setting is found in the gear on the LLM node.
5. Stream Response: Check to have responses from the LLM stream. Ensure to change the Type on the output node to “Streamed Text”.
6. JSON Output: Check to ​​to have the model return a structured JSON output rather than pure text.
7. Show Sources: Display sources of documents used from the knowledge base.
8. Show Confidence: Show the confidence level of the LLM’s answer.
9. Toxic Input Filtration: Filter out toxic content; if the LLM receives a toxic message, the LLM will respond with a respectful one.
10. Detect PII: Detect and remove PII from being sent to the LLM.

​

Node Outputs

1. Response: The output of the LLM
   • Type: Text (or Stream<Text> if streaming is enabled)
   • Example usage: {{openai_0.response}}
2. Tokens_used: The total number of tokens used
   • Type: Integer
   • Example usage: {{openai_0.tokens_used}}
3. Input_tokens: The total number of input tokens used
   • Type: Integer
   • Example usage: {{openai_0.input_tokens}}
4. Output_tokens: The total number of output tokens used
   • Type: Integer
   • Example usage: {{openai_0.output_tokens}}
5. Credits_used: The total number of VectorShift AI credits used
   • Type: Decimal
   • Example usage: {{openai_0.credits_used}}

​

Example

The below example is a pipeline for chatting with a knowledge base. The LLM is used to generate the response to the user question.

1. Input Node: Represents the user message
2. Knowledge Base Reader Node: Queries the knowledge base semantically
   • Search Query: {{input_0.text}}
3. LLM Node: Responds to the user question
   • System (Instructions): Answer the question based on the context.
   • Prompt: Question: {{input_0.text}} Context: {{knowledge_base_1.chunks}}
4. Output Node: Displays the response
   • Output: {{openai_0.response}}