LLM Nodes

Node Inputs

1. System (Instructions): Specify how you would like the LLM to respond (e.g., the style). It is common to specify here how the LLM should utilize the data received in Prompt.

   1. Type: Text

2. Prompt: Provide the data the LLM should consider. Type “{{“ to open the variable builder.

   1. Type: Text

Node Parameters

Face of the node

1. Model: Select the model to use from the provider

In the gear

1. Provider: Change the provider of the LLM

2. Max tokens: The maximum number of output tokens for each LLM run

3. Temperature: The diversity of the LLM generation. To have more diverse or creative generations, increase the temperature. To have more deterministic response, decrease the temperature.

4. Top P: The Top P parameter constrains how many tokens the LLM considers for generation at each step. For more diverse responses increase top p towards a maximum value of 1.0. This setting is found in the gear on the LLM node.

5. Stream Response: Check to have responses from the LLM stream. Ensure to change the Type on the output node to “Streamed Text”.

6. JSON Output: Check to ​​to have the model return a structured JSON output rather than pure text.

7. Show Sources: Display sources of documents used from the knowledge base.

8. Show Confidence: Show the confidence level of the LLM’s answer.

9. Toxic Input Filtration: Filter out toxic content; if the LLM receives a toxic message, the LLM will respond with a respectful one.

10. Detect PII: Detect and remove PII from being sent to the LLM.

Node Outputs

1. Response: The output of the LLM

   1. Type: Text

   2. Example usage: {{openai_0.response}}

2. Tokens_used: The total number of tokens used

   1. Type: Integer

   2. Example usage: {{openai_0.tokens_used}}

3. Input_tokens: The total number of input tokens used

   1. Type: Integer

   2. Example usage: {{openai_0.input_tokens}}

4. Output_tokens: The total number of output tokens used

   1. Type: Integer

   2. Example usage: {{openai_0.output_tokens}}

5. Credits_used: The total number of VectorShift AI credits used

   1. Type: Decimal

   2. Example usage: {{openai_0.credits_used}}

Example

The below example is a pipeline for chatting with a knowledge base. The LLM node receives data from the input node (the user message) and context from the knowledge base.