Skip to content

Commit 9524edc

Browse files
fzowlfzowl
authored
Merge pull request #1 from voyage-ai/voyageai
VoyageAI vectorizer and reranker
2 parents ce0f710 + 44decb6 commit 9524edc

File tree

15 files changed

+1659
-401
lines changed

15 files changed

+1659
-401
lines changed

‎.github/workflows/run_tests.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,7 @@ jobs:
6161
GCP_LOCATION: ${{ secrets.GCP_LOCATION }}
6262
GCP_PROJECT_ID: ${{ secrets.GCP_PROJECT_ID }}
6363
COHERE_API_KEY: ${{ secrets.COHERE_API_KEY }}
64+
VOYAGE_API_KEY: ${{ secrets.VOYAGE_API_KEY }}
6465
AZURE_OPENAI_API_KEY: ${{secrets.AZURE_OPENAI_API_KEY}}
6566
AZURE_OPENAI_ENDPOINT: ${{secrets.AZURE_OPENAI_ENDPOINT}}
6667
AZURE_OPENAI_DEPLOYMENT_NAME: ${{secrets.AZURE_OPENAI_DEPLOYMENT_NAME}}
@@ -80,6 +81,7 @@ jobs:
8081
GCP_LOCATION: ${{ secrets.GCP_LOCATION }}
8182
GCP_PROJECT_ID: ${{ secrets.GCP_PROJECT_ID }}
8283
COHERE_API_KEY: ${{ secrets.COHERE_API_KEY }}
84+
VOYAGE_API_KEY: ${{ secrets.VOYAGE_API_KEY }}
8385
AZURE_OPENAI_API_KEY: ${{secrets.AZURE_OPENAI_API_KEY}}
8486
AZURE_OPENAI_ENDPOINT: ${{secrets.AZURE_OPENAI_ENDPOINT}}
8587
AZURE_OPENAI_DEPLOYMENT_NAME: ${{secrets.AZURE_OPENAI_DEPLOYMENT_NAME}}

‎README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -216,6 +216,7 @@ Integrate with popular embedding models and providers to greatly simplify the pr
216216
- [OpenAI](https://www.redisvl.com/api/vectorizer.html#openaitextvectorizer)
217217
- [HuggingFace](https://www.redisvl.com/api/vectorizer.html#hftextvectorizer)
218218
- [GCP VertexAI](https://www.redisvl.com/api/vectorizer.html#vertexaitextvectorizer)
219+
- [VoyageAI](https://www.redisvl.com/api/vectorizer/html#voyageaitextvectorizer)
219220

220221
```python
221222
from redisvl.utils.vectorize import CohereTextVectorizer

‎docs/api/reranker.rst

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,3 +12,15 @@ CohereReranker
1212
.. autoclass:: CohereReranker
1313
:show-inheritance:
1414
:members:
15+
16+
17+
VoyageAIReranker
18+
================
19+
20+
.. _voyageaireranker_api:
21+
22+
.. currentmodule:: redisvl.utils.rerank.voyageai
23+
24+
.. autoclass:: VoyageAIReranker
25+
:show-inheritance:
26+
:members:

‎docs/api/vectorizer.rst

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,3 +49,14 @@ CohereTextVectorizer
4949
:show-inheritance:
5050
:members:
5151

52+
53+
VoyageAITextVectorizer
54+
====================
55+
56+
.. _voyageaitextvectorizer_api:
57+
58+
.. currentmodule:: redisvl.utils.vectorize.text.voyageai
59+
60+
.. autoclass:: VoyageAITextVectorizer
61+
:show-inheritance:
62+
:members:

‎docs/user_guide/getting_started_01.ipynb

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -413,7 +413,7 @@
413413
"source": [
414414
"## Creating `VectorQuery` Objects\n",
415415
"\n",
416-
"Next we will create a vector query object for our newly populated index. This example will use a simple vector to demonstrate how vector similarity works. Vectors in production will likely be much larger than 3 floats and often require Machine Learning models (i.e. Huggingface sentence transformers) or an embeddings API (Cohere, OpenAI). `redisvl` provides a set of [Vectorizers](https://www.redisvl.com/user_guide/vectorizers_04.html#openai) to assist in vector creation."
416+
"Next we will create a vector query object for our newly populated index. This example will use a simple vector to demonstrate how vector similarity works. Vectors in production will likely be much larger than 3 floats and often require Machine Learning models (i.e. Huggingface sentence transformers) or an embeddings API (Cohere, OpenAI, VoyageAI). `redisvl` provides a set of [Vectorizers](https://www.redisvl.com/user_guide/vectorizers_04.html#openai) to assist in vector creation."
417417
]
418418
},
419419
{

‎docs/user_guide/rerankers_06.ipynb

Lines changed: 45 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@
99
"\n",
1010
"In this notebook, we will show how to use RedisVL to rerank search results\n",
1111
"(documents or chunks or records) based on the input query. Today RedisVL\n",
12-
"supports reranking through the [Cohere /rerank API](https://docs.cohere.com/docs/rerank-2).\n",
12+
"supports reranking through the [Cohere /rerank API](https://docs.cohere.com/docs/rerank-2) or [VoyageAI /rerank API](https://docs.voyageai.com/docs/reranker).\n",
1313
"\n",
1414
"Before running this notebook, be sure to:\n",
1515
"1. Have installed ``redisvl`` and have that environment active for this notebook.\n",
@@ -75,7 +75,7 @@
7575
"cell_type": "markdown",
7676
"metadata": {},
7777
"source": [
78-
"### Init the Reranker\n",
78+
"### Init the Cohere Reranker\n",
7979
"\n",
8080
"Initialize the reranker. Install the cohere library and provide the right Cohere API Key."
8181
]
@@ -113,12 +113,54 @@
113113
]
114114
},
115115
{
116+
"cell_type": "markdown",
117+
"metadata": {},
118+
"source": [
119+
"### Init the VoyageAI Reranker\n",
120+
"\n",
121+
"Initialize the reranker. Install the voyageai library and provide the right VoyageAI API Key."
122+
]
123+
},
124+
{
125+
"cell_type": "code",
126+
"execution_count": null,
127+
"metadata": {},
128+
"outputs": [],
129+
"source": [
130+
"#!pip install voyageai"
131+
]
132+
},
133+
{
134+
"cell_type": "code",
135+
"execution_count": 3,
136+
"metadata": {},
137+
"outputs": [],
138+
"source": [
139+
"import getpass\n",
140+
"\n",
141+
"# setup the API Key\n",
142+
"api_key = os.environ.get(\"VOYAGE_API_KEY\") or getpass.getpass(\"Enter your VoyageAI API key: \")"
143+
]
144+
},
145+
{
146+
"cell_type": "code",
147+
"execution_count": 4,
148+
"metadata": {},
149+
"outputs": [],
150+
"source": [
151+
"from redisvl.utils.rerank import VoyageAIReranker\n",
152+
"\n",
153+
"reranker = VoyageAIReranker(model=\"rerank-lite-1\", limit=3, api_config={\"api_key\": api_key})",
154+
"# Please check the available models at https://docs.voyageai.com/docs/reranker"
155+
]
156+
},
157+
{
116158
"cell_type": "markdown",
117159
"metadata": {},
118160
"source": [
119161
"### Rerank documents\n",
120162
"\n",
121-
"Below we will use the `CohereReranker` to rerank and also truncate the list of\n",
163+
"Below we will use the `CohereReranker` or the `VoyageAIReranker` to rerank and also truncate the list of\n",
122164
"documents above based on relevance to the initial query."
123165
]
124166
},

‎docs/user_guide/vectorizers_04.ipynb

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@
1212
"2. HuggingFace\n",
1313
"3. Vertex AI\n",
1414
"4. Cohere\n",
15+
"5. VoyageAI\n",
1516
"\n",
1617
"Before running this notebook, be sure to\n",
1718
"1. Have installed ``redisvl`` and have that environment active for this notebook.\n",
@@ -501,6 +502,76 @@
501502
]
502503
},
503504
{
505+
"cell_type": "markdown",
506+
"metadata": {},
507+
"source": [
508+
"### VoyageAI\n",
509+
"\n",
510+
"[VoyageAI](https://dash.voyageai.com/) allows you to implement language AI into your product. The `VoyageAITextVectorizer` makes it simple to use RedisVL with the embeddings models at VoyageAI. For this you will need to install `voyageai`.\n",
511+
"\n",
512+
"```bash\n",
513+
"pip install voyageai\n",
514+
"```"
515+
]
516+
},
517+
{
518+
"cell_type": "code",
519+
"execution_count": 11,
520+
"metadata": {},
521+
"outputs": [],
522+
"source": [
523+
"import getpass\n",
524+
"# setup the API Key\n",
525+
"api_key = os.environ.get(\"VOYAGE_API_KEY\") or getpass.getpass(\"Enter your VoyageAI API key: \")"
526+
]
527+
},
528+
{
529+
"cell_type": "markdown",
530+
"metadata": {},
531+
"source": [
532+
"\n",
533+
"Special attention needs to be paid to the `input_type` parameter for each `embed` call. For example, for embedding \n",
534+
"queries, you should set `input_type='query'`; for embedding documents, set `input_type='document'`. See\n",
535+
"more information [here](https://docs.voyageai.com/docs/embeddings)"
536+
]
537+
},
538+
{
539+
"cell_type": "code",
540+
"execution_count": 12,
541+
"metadata": {},
542+
"outputs": [
543+
{
544+
"name": "stdout",
545+
"output_type": "stream",
546+
"text": [
547+
"Vector dimensions: 1024\n",
548+
"[0.015814896672964096, 0.046988241374492645, -0.00518248463049531, -0.05383478105068207, -0.015586535446345806, -0.0837097093462944, 0.03744547441601753, -0.007797810714691877, 0.00717928446829319, 0.06857716292142868]\n",
549+
"Vector dimensions: 1024\n",
550+
"[0.006725038401782513, 0.01441393606364727, -0.030212024226784706, -0.06782275438308716, -0.021446991711854935, -0.07667966187000275, 0.01804908737540245, -0.015767497941851616, -0.02152789570391178, 0.049741245806217194]\n"
551+
]
552+
}
553+
],
554+
"source": [
555+
"from redisvl.utils.vectorize import VoyageAITextVectorizer\n",
556+
"\n",
557+
"# create a vectorizer\n",
558+
"vo = VoyageAITextVectorizer(\n",
559+
" model=\"voyage-law-2\", # Please check the available models at https://docs.voyageai.com/docs/embeddings\n",
560+
" api_config={\"api_key\": api_key},\n",
561+
")\n",
562+
"\n",
563+
"# embed a search query\n",
564+
"test = vo.embed(\"This is a test sentence.\", input_type='query')\n",
565+
"print(\"Vector dimensions: \", len(test))\n",
566+
"print(test[:10])\n",
567+
"\n",
568+
"# embed a document\n",
569+
"test = vo.embed(\"This is a test sentence.\", input_type='document')\n",
570+
"print(\"Vector dimensions: \", len(test))\n",
571+
"print(test[:10])"
572+
]
573+
},
574+
{
504575
"cell_type": "markdown",
505576
"metadata": {},
506577
"source": [

0 commit comments

Comments
 (0)