Large Language Models (LLMs)

Large Language Models (LLMs) are AI models trained on large corpuses of data, and can generate text, images, videos, etc. In this section, we will discuss LLMs that generate text.

Through careful prompting LLMs can accomplish a variety of tasks. The VectorShift platform is LLM agnostic, meaning you can choose which model to use in your workflows (OpenAI, Anthropic, Google,m Mistral, Llama, etc.). You can choose the model and prompt best suited for your application.

The AI landscape changes very fast and we can expect new models to be released by various research labs. The VectorShift team will add new LLMs as soon as they are released. Within pipelines, you can then swap between LLM models / providers with ease.

How to use an LLM?

To use an LLM, you must do the following things:

1. System prompt: instruct how the LLM should behave in the system prompt within the LLM node or write it in a text box and connect it to the "System" input edge. Reference data sources you use in the Prompt within System (Answer the User Question using Context)

2. Prompt: define variables using double curly braces (open the variable builder by typing : "{{" in any text field)

LLM Settings

System and Prompt

Some models (e.g., OpenAI) are trained to take two inputs, a "system" prompt that contains instruction for the model to follow and a "prompt" input with various data sources (e.g., the user message, context, data sources, etc.). Other models (e.g., Gemini) have one singular prompt where you place both the instructions and the data sources.

Token Limits

Each model has a limit max number of input and output tokens. To adjust the limit for a particular model you can alter the max tokens parameter. Note: you cannot increase the max tokens beyond the maximum supported for a particular model. This setting is found in the gear on the LLM node.

Streaming

To stream output, click on the gear and then check "Stream Response". This setting is found in the gear on the LLM node.

Citations

You can display the sources the LLM uses by checking off "Show Sources" in the gear. This setting is found in the gear on the LLM node.

JSON Response

To have the model return a structured JSON output rather than pure text check the "Json output" box. This setting is found in the gear on the LLM node.

When using Json mode you can optionally provide the Json Schema. This will help the LLM know which json fields to generate.

For example if I want the output json to have a "temperature" field which is an integer and a unit field which is either Celsius or Fahrenheit I can define the schema as follows

{
    "type": "object",
    "properties": {
        "temperature": {
            "type": "integer",
            "description": "temperaure"
        },
        "unit": {
              "type": "string",
              "enum": ["celsius", "fahrenheit"],
              "description": "the temperature unit to use"
        }
    }
    "required":  ["temperature", "unit"]
}
        

Temperature

Temperature controls the diversity of LLM generation. You can adjust the temperature settings for your models. To have more diverse or creative generations increase the temperature. To have more deterministic response decrease the temperature. This setting is found in the gear on the LLM node.

Top P

The TopP parameter constrains how many tokens the LLM considers for generation at each step. For more diverse responses increase top p towards a maximum value of 1.0. This setting is found in the gear on the LLM node.

AI Model Costs

Model usage is billed based on the number of tokens that you use, both in the model input and the tokens generated in the model output. One token is equal to 4 characters.

Provider	Model	Price ($per 1000 tokens)
OpenAI	gpt-3.5-turbo	0.002
OpenAI	gpt-3.5-turbo-16k	0.002
OpenAI	gpt-3.5-turbo-instruct	0.002
OpenAI	gpt-4	0.06
OpenAI	gpt-4-32k	0.12
OpenAI	gpt-4-turbo	0.03
Anthropic	claude-v2	0.024
Anthropic	claude-instant	0.0024
Anthropic	claude-v2.1	0.024
Meta	llama2-13b	0.001
Meta	llama2-70b	0.00256
Meta	llama2-chat-13b	0.001
Meta	llama2-chat-70b	0.00256
Meta	llama-3-8b-chat-hf	0.0009
Meta	llama-3-70b-chat-hf	0.0009
Cohere	command	0.002
AWS	titan-text-express	0.0004
AWS	titan-text-lite	0.0016
Open Source	mistralai/Mistral-7B-v0.1	0.0002
Open Source	mistralai/Mistral-7B-Instruct-v0.1	0.0002
Open Source	mistralai/Mistral-7B-Instruct-v0.2	0.0002
Open Source	mistralai/Mixtral-8x7B-v0.1	0.0006
Open Source	mistralai/Mixtral-8x7B-Instruct-v0.1	0.0006
Google	gemini-pro	0.0005
Google	text-bison	0.0005
Google	text-bison-32k	0.0005
Google	text-unicorn	0.0075

Custom LLMs

Want to connect to a specialized model provider or a locally hosted LLM? Use the custom LLM node.

We have support for sending requests to models that are compatible with the Open AI chat api format. You can use models from your own accounts with LLM providers such as TogetherAI and Replicate. The custom LLM node requires the following parameters.

• model

• api key

• base url

For example using Together as an model provider the base_url will be "https://api.together.xyz". The API Key will be the key you find on your account. For the model you can choose from the available models for a provider.

Local Models

Models hosted locally on your computer are good for prototyping, experimenting with new models and cost savings. You can access your local models be setting up a connection to a locally running LLM server.

Make sure to find a secure way to forward your locally running server's port to the internet.

LM Studio

Follow the instructions to start a local LM studio server

The standard API key for LM studio is lm-studio

Ollama

Start a local LLM server using the Ollama cli.

Prompt Engineering Guidelines

Be as specific as possible - if the output should be one sentence or if the output should be in the first person, include the instructions in the text block connected to the system prompt. Within the system prompt, you can also mention things like:

• The tone you want the model to use (e.g., Respond in a professional manner).

• Reference data sources provided in the prompt and how the model should use them (e.g., Use datasource X when the question is related to sales; use datasource Y when the question is related to customer support).

• Provide specific information related to your company / situation that the model can reference (e.g., calendly link)

• Specific text that you want the model to output in certain situations (e.g., if you are unable to answer the question, respond with "I am unable to answer the question").

• What type of reasoning to use (here - it is important to think through step by step how you would actually perform the action. You then, want to encode this into the system prompt).

FAQs

• What is the best model for my tasks?

  • The best model depends on your use case as well as cost/latency constraints. Make sure to evaluate the performance of the model on your task.

• What should I to if I keep running into token limit errors?

  • Try reducing the amount of text passed to the LLM. You can use semantic search to feed only the most relevant input to the language model. See the vector db documentation and the Document Search Assistant tutorial

• Why won't the LLM follow my instructions?

  • Many language model applications require iterative development to find an effective prompt. Try stating your instruction in clear language. Additionally, you can try using a more powerful model (e.g., GPT 4 turbo)

For more questions about developing applications with LLMs check out the resource below or drop a question in our discord server.

Further Reading

• Understanding LLM Parameters

• Basics of LLM Prompt Engineering

• Chain of Thought Prompting

• Tree of Thought Prompting