Declare a schema and upsert documents¶

A document index is an index created with a schema that names its searchable fields. You write and read it through index.documents, and each record is a JSON document: a reserved _id plus your own fields. This page covers choosing fields, creating the index, and loading data.

If your index was created another way, it uses a different interface: top-level dimension=/metric= indexes take upsert() (see Upserting and querying vectors), and indexes from create_for_model() take the legacy Records API (see Legacy integrated indexes).

Choose your fields¶

Declare only the fields you search. Each kind of field is searched with its own score_by clause:

Field	Your documents carry	Search it with
`dense_vector`	a list of floats from your embedding model	`DenseVectorQuery`
`sparse_vector`	`{"indices": [...], "values": [...]}`	`SparseVectorQuery`
`string` with full-text search	text	`TextQuery`, `QueryStringQuery`
`string` with `embed`	text; Pinecone embeds it with a hosted dense model	`EmbedQuery`
`string` with `sparse_embed`	text; Pinecone embeds it with a hosted sparse model	`SparseEmbedQuery`

An index holds at most one dense field (a dense_vector, or a string with embed) and at most one sparse field (a sparse_vector, or a string with sparse_embed). Values you only filter on, such as a category or a year, stay out of the schema: put them on the documents and they are indexed for filtering when first written.

The schema cannot change after the index is created, so declare every searched field up front.

Create the index¶

SchemaBuilder lays out the field types and their options. This index stores your own embeddings and also supports keyword search on the title:

from pinecone import Pinecone, SchemaBuilder

pc = Pinecone(api_key="your-api-key")

schema = (
    SchemaBuilder()
    .add_dense_vector_field("embedding", dimension=1536, metric="cosine")
    .add_string_field("title", full_text_search={"language": "en", "stemming": True})
    .build()
)
pc.indexes.create(name="article-search", schema=schema)

dimension and metric must match the embedding model you use. create blocks until the index is ready; pass timeout=-1 to return immediately. Without deployment=, the index is serverless on AWS us-east-1; see Working with serverless indexes to choose another cloud or region.

To have Pinecone embed text for you, give a string field embed, sparse_embed, or both. True takes the default model:

schema = SchemaBuilder().add_string_field("body", embed=True, sparse_embed=True).build()
pc.indexes.create(name="support-articles", schema=schema)

The text of an embed field is stored and full-text searchable unless you pass store_text=False. See Integrated embedding for choosing a model and what store_text=False gives up.

build() returns a plain {"fields": {...}} dict, and writing that dict yourself is equally supported. The builder does not decide what the server accepts: an invalid combination is sent as written, and create raises ApiError with the server’s message.

Upsert documents¶

upsert() writes a list of documents to a namespace. A document whose _id already exists is replaced.

index = pc.index("article-search")

response = index.documents.upsert(
    namespace="articles-en",
    documents=[
        {
            "_id": "article-101",
            "embedding": [0.012, -0.087, 0.153],  # 1536 floats in practice
            "title": "How Roman aqueducts carried water",
            "category": "history",
            "year": 2024,
        },
        {
            "_id": "article-102",
            "embedding": [0.045, 0.021, -0.064],
            "title": "Gravity-fed irrigation today",
            "category": "engineering",
            "year": 2023,
        },
    ],
)
print(response.upserted_count)

Each document must carry every dense_vector field and the text of every embed field. Sparse fields, full-text-only fields, and metadata such as category are optional. The server rejects a document that is missing a required field with a 400 naming the document and the field.

For the support-articles index above, a document is just its _id, its text, and any metadata: {"_id": "article-101", "body": "Roman aqueducts carried water by gravity"}.

Documents can also be built as DocumentRecord objects, which check the _id where you construct them.

Load large datasets¶

upsert sends one request, and the server limits a request by document count and by size. For more documents than that, or documents of unknown size, use batch_upsert(). It splits the list into batches, sends them in parallel, and halves any batch the server rejects as too large. Here documents is a list of document dicts like the ones above:

result = index.documents.batch_upsert(namespace="articles-en", documents=documents)

for error in result.errors:
    print(error.batch_index, error.retryable, error.error_message)

A failed batch is recorded in the returned BatchResult instead of raised, so one bad batch does not stop the rest. Resend result.failed_items after fixing whatever result.errors reports. On an index with an embed or sparse_embed field, the server accepts fewer documents per request; the default batch_size fits, so raise it only on other indexes. See How bulk ingest behaves for tuning batch_size and max_concurrency.

Wait for documents to be visible¶

Upserts apply asynchronously: upsert returns once the server accepts the documents, and a search or fetch made right afterward may not see them yet. Code that reads its own writes, such as a test or a demo script, should poll until the documents appear:

import time

ids = ["article-101", "article-102"]
while len(index.documents.fetch(namespace="articles-en", ids=ids).documents) < len(ids):
    time.sleep(1)

Async client¶

AsyncDocuments on AsyncIndex has the same methods with the same arguments; await each call:

from pinecone import AsyncPinecone

async with AsyncPinecone(api_key="your-api-key") as pc:
    index = await pc.index(name="article-search")
    async with index:
        await index.documents.upsert(namespace="articles-en", documents=documents)