Vector Stores

Create a permanent embedded database for frequently reused data.
Vector Stores allow users to convert and store data as vector embeddings, which represent the meaning of the stored data. This representation allows users to ensure that they are always using the most relevant data for their Pipelines. You can access Vector Stores in the "Vector Stores" sub-tab in the Storage tab.
You can store a wide range of data in a Vector Store, including files, scraped URLs, GitHub repositories, and more. Each data source is converted into a specified number of vectors (depending on its length). Each non-Enterprise account is capped at 1,000 total vectors across all Vector Stores.

Create a New Vector Store

To create a new Vector Store, take the following steps:
  1. 1.
    Navigate to the "Vector Stores" sub-tab in Storage.
  2. 2.
    Click "New" >> "Create VectorStore", and fill in the Name and Description. Advanced Settings: If you want more control over how your data is converted into documents, you can change the "Chunk Size" and "Chunk Overlap" parameters. Chunk Size determines how many tokens are stored in each vector, and Chunk Overlap determines if there is any overlap between the tokens stored in each vector. Decreasing Chunk Size and/or increasing Chunk Overlap will result in the same data requiring more vectors to store, and Chunk Size determines how many tokens a query to your Vector Store will return (per document).
  3. 3.
    Add any relevant data to the Vector Store using the available data loaders, and click "Finish".

Edit an Existing Vector Store

To edit an existing Vector Store, take the following steps:
  1. 1.
    Either double click on the VectorStore or click once and click "edit" on the top bar.
  2. 2.
    (Optional) Edit any of the Vector Store parameters, if desired. Click on "Advanced Settings".
  3. 3.
    (Optional) Add or delete data from the Vector Store, if desired, and click "Finish". To delete, click on the trash can icon. To add, click on the "+" button.
Note, if you want to embed files into the Vector Store, you need to first upload them into our platform. In the storage tab, navigate to the "files" sub-tab. Then click "New" >> "Upload File". Once the file is loaded, click "+", select "File" as the Loader Type, and then select the files you want to embed.

Keeping Information Up to Date

As you choose which files you wish to load from your integrations, there may come times where the information you have embedded is no longer up to date.
In the bottom right-hand corner of the Vector Store page, there's a button labeled "Sync Vector Stores." Clicking this button will tell the VectorShift platform to check whether there have been any additions or updates to the files you have chosen to include in your vectorstore(s). Files that have been edited will be removed and re-embedded while new files located inside folders/directories you have chosen to include will be added as well.
By default, the platform will sync all vector stores once a week, but you may choose to sync your vectors more frequently if changes have been made in your integrations.
Keep in mind that this task may take a while to complete, depending on the number of files, vectors, and different integrations you have chosen to include in your vector store.