Skip to main content
The Open Source LLM node connects your workflows to open-source language models hosted on Together AI, including Meta Llama, Mistral, Google Gemma, DeepSeek, and NVIDIA NeMo. Use it to access cost-effective, high-quality models for tasks like classifying financial transactions, generating research summaries, or building AI workflows with models you can evaluate and audit.

​
Core Functionality

  • Access open-source models from Meta Llama, Mistral, Google Gemma, DeepSeek, and others via Together AI
  • Process system instructions and dynamic prompts with variable interpolation
  • Stream responses in real time for long-running generations
  • Return structured JSON output with optional schema enforcement
  • Track token usage and credit consumption per run
  • Apply content moderation, PII detection, and safety guardrails
  • Retry failed executions automatically with configurable intervals

​
Tool Inputs

  • System Instructions — (String) Instructions that guide the model’s behavior, tone, and how it should use data provided in the prompt
  • Prompt — (String) The data sent to the model. Type {{ to open the variable builder and reference outputs from other nodes
  • Model * — (Enum (Dropdown), default: deepseek-ai/DeepSeek-R1) Select from available open-source models. Click Dropdown to view all options
  • Use Personal Api Key — (Boolean, default: No) Toggle to use your own Together AI API key
  • Api Key — (String) Your Together AI API key. Required when Use Personal Api Key is enabled — the node will show a validation error if left blank
  • JSON Schema — (String) JSON schema to enforce structured output. Only visible when JSON Response is enabled
* indicates a required field

​
Tool Outputs

  • response — (String (or Stream<String> when streaming)) The generated text response from the model
  • prompt_response — (String) The combined prompt and response content
  • tokens_used — (Integer) Total number of tokens consumed (input + output)
  • input_tokens — (Integer) Number of input tokens sent to the model
  • output_tokens — (Integer) Number of output tokens generated by the model
  • credits_used — (Decimal) VectorShift AI credits consumed for this run

​
Overview

The Open Source LLM node in workflows lets you place an open-source model on the canvas via Together AI’s hosting infrastructure. This gives you access to a wide range of models from different research labs, enabling cost optimization and model evaluation within your workflows.

​
Use Cases

  • Cost-optimized batch processing — Route high-volume, lower-complexity tasks like transaction tagging to cost-effective open-source models while reserving premium models for complex analysis.
  • Model evaluation and comparison — Compare outputs across different open-source model families to find the best quality-to-cost ratio for specific financial tasks.
  • Research summarization — Use DeepSeek or Llama models to summarize research papers, market reports, or regulatory updates at scale.
  • Regulatory document classification — Classify documents by type, jurisdiction, or urgency using open-source models that perform well on categorization tasks.
  • Internal knowledge Q&A — Build cost-effective internal chatbots using open-source models grounded in your organization’s documentation.

​
How It Works

  1. Add the node to your workflow. From the toolbar, open the AI category and drag the Open Source node onto the canvas.
Open Source node being dragged onto the canvas
  1. Write your System Instructions. Enter instructions in the System Instructions field to define the model’s behavior.
  2. Configure the Prompt. In the Prompt field, type {{ to open the variable builder and reference upstream node outputs.
  3. Select a model. Use the Model dropdown to choose an open-source model. Available options include deepseek-ai/DeepSeek-R1, meta-llama/Llama-3-70b-chat-hf, mistralai/Mistral-7B-Instruct-v0.1, mistralai/Mixtral-8x7B-Instruct-v0.1, google/gemma-2-27b-it, nvidia/Llama-3.1-Nemotron-Ultra-253B-v1, and others.
Open Source node showing the Model dropdown
  1. Provide an API key. Toggle Use Personal Api Key to Yes and enter your Together AI API key.
  2. Open settings. Click the gear icon (⚙) to configure token limits, temperature, retry behavior, and more.
Open Source node settings panel
  1. Connect outputs and run. Wire the response output to downstream nodes and execute the pipeline.

​
Settings

All settings below are accessed via the gear icon (⚙) on the node.
SettingTypeDefaultDescription
ProviderDropdownOpen SourceThe LLM provider (Together AI).
Max TokensInteger128000Maximum number of input + output tokens per run.
TemperatureFloat0.5Controls response creativity. Range: 0–1.
Top PFloat0.5Controls token sampling diversity. Range: 0–1.
Stream ResponseBooleanOffStream responses token-by-token.
JSON OutputBooleanOffReturn output as structured JSON.
Show SourcesBooleanOffDisplay source documents used for the response.
Toxic Input FiltrationBooleanOffFilter toxic input content.
Retry On FailureBooleanOffEnable automatic retries when execution fails.
Max RetriesInteger—Maximum retry attempts. Visible when Retry On Failure is enabled.
Max Interval b/w re-tryInteger—Interval in milliseconds between retry attempts.
PII Detection
NameBooleanOffDetect and redact personal names from input.
EmailBooleanOffDetect and redact email addresses from input.
PhoneBooleanOffDetect and redact phone numbers from input.
SSNBooleanOffDetect and redact Social Security numbers from input.
Credit CardBooleanOffDetect and redact credit card numbers from input.
Show Success/Failure OutputsBoolean—Display additional success and failure output ports.

​
Best Practices

  • Start with DeepSeek-R1 for reasoning tasks. It offers strong reasoning capabilities at a lower cost than commercial alternatives.
  • Use Llama models for general-purpose tasks. Meta’s Llama family provides excellent general performance for summarization and Q&A.
  • Compare models before committing. Swap the model dropdown to test different models on the same inputs and evaluate quality-to-cost tradeoffs.
  • Monitor token usage. Open-source models via Together AI are billed per token — track consumption for budgeting.
  • Enable PII detection for client data. Even with open-source models, apply PII toggles when processing sensitive financial information.

​
Common Issues

For troubleshooting common issues with this node, see the Common Issues documentation.