Pinecone Scala client powered by Cequence

The Pinecone Scala client was released on April 27th 2023 and is completely open-source. The project is available on GitHub at https://github.com/cequence-io/pinecone-scala. Note that if you are reading this post in the future 🤨, the current version (0.0.1) is likely to be different. You can check for updates on GitHub or Maven.

In order to use the lib add the following dependency to your build.sbt:

"io.cequence" %% "pinecone-scala-client" % "0.0.1"

or to pom.xml (if you use maven)

<dependency>
 <groupId>io.cequence</groupId>
 <artifactId>pinecone-scala-client_2.12</artifactId>
 <version>0.0.1</version>
</dependency>

The currently supported Scala versions are 2.12, 2.13 and 3.

All available endpoints as defined here are provided in two convenient services called PineconeVectorService and PineconeIndexService.

Config ⚙️

Before using the library set the following env. variables: PINECONE_SCALA_CLIENT_API_KEY and PINECONE_SCALA_CLIENT_ENV. Alternatively, you can provide your own config file and override the configuration options (including timeouts) as specified in the default config pinecone-scala-client.conf. If you don’t have a Pinecone account yet register here.

Usage 👨‍🎓

First you need to provide an implicit execution context as well as akka materializer, e.g., as

implicit val ec = ExecutionContext.global
implicit val materializer = Materializer(ActorSystem())

Ia. Obtaining PineconeIndexService

• Default config (expects env. variable(s) to be set as defined in Config section)

val service = PineconeIndexServiceFactory()

• Custom config

val config = ConfigFactory.load("path_to_my_custom_config")
val service = PineconeIndexServiceFactory(config)

• Without config

val service = PineconeIndexServiceFactory(
  apiKey = "your_api_key",
  environment = "your_env" // e.g. "northamerica-northeast1-gcp
)

Ib. Obtaining PineconeVectorService

Same as with PineconeIndexService, you need to first provide implicit execution context and akka materializer. Then you can obtain a service in one of the following ways:

• Default config (expects env. variable(s) to be set as defined in Config section). Note that if the index with a given name is not available, the factory will return None.

PineconeVectorServiceFactory("index_name").map { serviceOption =>
  val service = serviceOption.getOrElse(
    throw new Exception(s"Index with a given name does not exist.")
  )
  // do something with the service
}

• Custom config

val config = ConfigFactory.load("path_to_my_custom_config")
PineconeVectorServiceFactory("index_name", config).map { service =>
  val service = serviceOption.getOrElse(
    throw new Exception(s"Index with a given name does not exist.")
  )
  // do something with the service
}

• Without config

PineconeVectorServiceFactory(
  apiKey = "your_api_key",
  indexName = "index_name", // e.g. "auto-gpt"
  pineconeIndexService = pineconeIndexService // index service to be used to find the index host URL 
).map { serviceOption =>
 val service = serviceOption.getOrElse(
   throw new Exception(s"Index with a given name does not exist.")
 )
 // do something with the service
}

II. Calling functions

Once we have created pineconeIndexService and/or pineconeVectorService we can start actually doing something useful. Since all the calls are async they return responses wrapped in Future.

• List indexes

pineconeIndexService.listIndexes.map(
  _.foreach(println)
)

• Create an index with default settings

import io.cequence.pineconescala.domain.response.CreateResponse
  
pineconeIndexService.createIndex(
  name = "auto-gpt-test",
  dimension = 1536
).map(
  _ match {
    case CreateResponse.Created => println("Index successfully created.")
    case CreateResponse.BadRequest => println("Index creation failed. Request exceeds quota or an invalid index name.")
    case CreateResponse.AlreadyExists => println("Index with a given name already exists.")
  }
)

• Upsert vectors

val dimension = 1536

pineconeVectorService.upsert(
  vectors = Seq(
    PVector(
      id = "666",
      values = Seq.fill(dimension)(Random.nextDouble),
      metadata = Map(
        "is_relevant" -> "not really but for testing it's ok, you know",
        "food_quality" -> "brunches are perfect but don't go there before closing time"
      )
    ),
    PVector(
      id = "777",
      values = Seq.fill(dimension)(Random.nextDouble),
      metadata = Map(
        "is_relevant" -> "very much so",
        "food_quality" -> "burritos are the best!"
      )
    )
  ),
  namespace = "my_namespace",
).map(vectorUpsertedCount =>
  println(s"Upserted $vectorUpsertedCount vectors.")
)

• Query vectors with custom settings

pineconeVectorService.query(
  vector = Seq.fill(1536)(Random.nextDouble), // some values/embeddings
  namespace = "my_namespace",
  settings = QuerySettings(
    topK = 5,
    includeValues = true,
    includeMetadata = true
  )
).map { queryResponse =>
  queryResponse.matches.foreach { matchInfo =>
    println(s"Matched vector id: ${matchInfo.id}")
    println(s"Matched vector values: ${matchInfo.values.take(20).mkString(", ")}..")
    println(s"Matched vector score: ${matchInfo.score}")
    println(s"Matched vector metadata: ${matchInfo.metadata}")
  }
}

We don’t want to pollute this post with an exhaustive list of all calls/examples (18 in total), therefore we recommend you to continue by reading Usage section of the README of the project on GitHub.

In addition, we implemented two ready-to-run demos as separate seed projects:

• Pinecone Scala Demo 🔥 — shows how to use Pinecone vector, index, and collection operations.
• Pinecone + OpenAI Scala Demo 🔥🔥 — shows how to generate and store OpenAI embeddings (with text-embedding-ada-002 model— 1536 dimensional) into Pinecone and query them afterwards. Based on the official tutorial from Pinecone.