Data Loading

Initial Load Workflow

Every dataset endpoint is scoped to a team and dataset — prefix each operation below with /api/teams/{teamName}/datasets/{dataSetName}/.

1. PUT   (the dataset route)     → create or open dataset (201 created / 200 existed)
2. POST  analyze                 → discover fields from JSON
3. PUT   fields/configuration    → configure fields (204)
4. POST  load                    → load documents (204)
5. POST  index                   → start the index build (202 Accepted)
6. GET   status                  → poll until systemState == 4 (Ready)
7. POST  search                  → search

What each call does to data already there

The loading calls are not interchangeable. The difference is what happens to what the dataset already holds:

Two things follow from this:

• Don't call analyze on a live dataset to pick up a schema change. It takes the dataset offline and resets every field role. Use Replace instead.
• analyze decides which fields exist. A sample is enough, but only if it contains every field. A field missing from the analyzed body is unknown afterwards, and every later insert or update carrying it is refused.

Choosing a path

The server accepts request bodies up to 2 GB, so analyze, load and replace normally take a whole catalogue in one request. A proxy in front of the server can have a lower limit of its own: under IIS the shipped web.config raises it, and behind nginx raise client_max_body_size.

TypeScript Example

import { SystemState, QueryProxy, FieldProxy } from '@indxsearch/indx-types';

const team = 'my-team';      // team that owns the dataset
const dataset = 'products';
const base = `https://your-host/api/teams/${team}/datasets/${dataset}`;

const api = (op: string, init?: RequestInit) =>
  fetch(`${base}/${op}`, {
    headers: { Authorization: `Bearer ${token}`, 'Content-Type': 'application/json' },
    ...init,
  });

// 1. Create dataset (PUT on the dataset route itself — 201 created / 200 existed)
await fetch(base, { method: 'PUT', headers: { Authorization: `Bearer ${token}` } });

// 2. Analyze
await api('analyze', { method: 'POST', body: jsonString });

// 3. Configure fields (204 No Content)
const fields: FieldProxy[] = [
  { fieldName: 'name',        searchable: true, weight: 2.0 },
  { fieldName: 'description', searchable: true, weight: 1.0 },
  { fieldName: 'category',    filterable: true, facetable: true },
  { fieldName: 'price',       filterable: true, sortable: true },
];
await api('fields/configuration', { method: 'PUT', body: JSON.stringify(fields) });

// 4. Load (204 No Content)
await api('load', { method: 'POST', body: jsonString });

// 5. Index (202 Accepted — the build starts in the background)
await api('index', { method: 'POST' });

// 6. Poll until ready
let status;
do {
  await new Promise(r => setTimeout(r, 500));
  status = await api('status').then(r => r.json());
} while (status.systemState !== SystemState.Ready);

Field Configuration

PUT /api/teams/{teamName}/datasets/{dataSetName}/fields/configuration

Configure field properties, for any number of fields in a single call. null properties are ignored (leave current value unchanged). This is the one route that writes field roles; the role lists (GET fields/searchable and the rest) are read-only.

Before the dataset is indexed the answer is always 204 No Content. On a dataset that is Ready, a change that needs the index rebuilt answers 202 Accepted instead: the rebuild has been started in the background and the dataset keeps serving with the old configuration meanwhile. Poll GET status until shadowBuildInProgress is false, then check that shadowBuildError is null. See Endpoints.

[
  {
    "fieldName": "name",
    "searchable": true,
    "weight": 2.0,
    "bM25b": 0.75,
    "bM25k1": 1.2
  },
  {
    "fieldName": "category",
    "filterable": true,
    "facetable": true,
    "preloadFilters": true
  },
  {
    "fieldName": "embedding",
    "embeddable": true
  }
]

GET /api/teams/{teamName}/datasets/{dataSetName}/fields/configuration returns the same FieldProxy[] shape with all current values populated.

The single-role setters (PUT fields/searchable, PUT fields/filterable, PUT fields/facetable, PUT fields/sortable, PUT fields/word-indexing, PUT fields/embeddable) were removed in October 2026 and answer 405. Send the same roles to PUT fields/configuration. A field marked embeddable must be an array of numbers and can have no other role.

Dynamic Operations

Insert, update, and delete documents after the index is built. The index stays in sync without a full rebuild.

Insert and update refuse a field the configuration does not know. A document carrying a field that is not in the dataset's field configuration gets 400 invalidArgument, naming the field (nested fields by path, such as 'meta.batch'), and nothing is stored: one such document fails its whole batch, and an update leaves the document as it was. Typical case: a new field added in your CMS (Sanity, Contentful, ...) while the sync job keeps inserting. To adopt it, run a full Replace with a complete export. It reports the field under added, after which you assign roles in Field configuration. Missing non-key fields are accepted and treated as null; a missing key field rejects the whole batch.

Insert

POST /api/teams/{teamName}/datasets/{dataSetName}/documents/{documentKey} — single document (201)

curl -X POST https://your-host/api/teams/<team>/datasets/products/documents/1001 \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '"{ \"id\": 1001, \"name\": \"New Product\", \"price\": 49.99 }"'

POST /api/teams/{teamName}/datasets/{dataSetName}/documents — batch (201)

curl -X POST https://your-host/api/teams/<team>/datasets/products/documents \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '["{ \"id\": 1001, \"name\": \"A\" }", "{ \"id\": 1002, \"name\": \"B\" }"]'

Update

PUT /api/teams/{teamName}/datasets/{dataSetName}/documents/{documentKey} — single document (204)

PUT /api/teams/{teamName}/datasets/{dataSetName}/documents — batch (array of JSON strings; 204)

PATCH /api/teams/{teamName}/datasets/{dataSetName}/documents/{documentKey} — update one field (204)

{ "fieldName": "price", "value": 39.99 }

POST /api/teams/{teamName}/datasets/{dataSetName}/documents/update-by-filter — update a field on all matching documents

{
  "filter": { "hashString": "abc123..." },
  "fieldName": "inStock",
  "value": false
}

Returns the count of updated documents as {"count": n}.

Delete

DELETE /api/teams/{teamName}/datasets/{dataSetName}/documents/{documentKey} — single document (204)

DELETE /api/teams/{teamName}/datasets/{dataSetName}/documents — multiple documents (body: long[]; 204)

POST /api/teams/{teamName}/datasets/{dataSetName}/documents/delete-by-filter — all documents matching a filter (204)

{ "hashString": "abc123..." }

Real-time updates

Document inserts, updates and deletes (POST/PUT/PATCH/DELETE …/documents…) apply to the live index and are searchable immediately — no queue, no rebuild, no eventual consistency. A record published from your CMS is in the next search. Only adding new fields or changing a field's type needs a reload, which the Replace endpoint below does with zero downtime.

Replace (full reload)

POST /api/teams/{teamName}/datasets/{dataSetName}/replace

Atomically swaps all documents in the dataset for a fresh JSON payload — the streamed-body equivalent of a full reload. Use it to push a new export of the same dataset without the downtime and data loss of delete-and-recreate.

What it preserves and guarantees:

• Identity & settings survive — field configuration, boost rules, the declared key field, and the description all carry over (reconciled against the new schema). Only the documents change.
• Zero downtime — the old data keeps serving searches until the new index is built, then the engine is swapped atomically.
• All-or-nothing — if the new JSON fails to parse or index, the existing dataset is left completely untouched and still serving.
• State-transparent — works whether the dataset is Ready, hibernated or idle-evicted; the old documents are never reloaded just to be discarded.

curl -X POST https://your-host/api/teams/<team>/datasets/products/replace \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  --data-binary @products-latest.json

The body is the full new dataset as a JSON array of documents — the same shape you pass to load / load/text. No separate analyze / fields/configuration / index steps are needed; replace runs the whole pipeline and returns when the new index is live.

Response (200) — a summary of how the new schema differed from the previous field configuration:

{
  "added":       ["newField"],   // in the new JSON, not previously configured (loads unconfigured)
  "removed":     ["oldField"],   // previously configured, absent from the new JSON (its config is dropped)
  "typeChanged": ["price"],      // detected type changed; that field's roles were reset
  "lostRoles":   ["price (filterable, sortable)"],  // removed or retyped fields that HAD a role, with the roles — searches and filters on them now silently miss
  "keyFieldFallback": null       // set to a message when the declared key field is absent from the new data and the engine keyed by its default instead
}

Treat lostRoles and keyFieldFallback as warnings: they are the changes that alter search behaviour or document identity. A declared key field missing from the new data does not fail the replace — the engine keys by its default (an id field if present, else auto-generated), exactly as for a dataset with no declared key — but document keys may then differ from the previous load, so change the key under Field configuration → Document key if that is not what you want. A removed field with no role is just schema noise; a removed searchable field means queries that used to match on it no longer do.

A boost rule that references a field which disappears simply goes dormant — it is not deleted, and it re-applies if a later replace brings the field back.

Errors

From the web console — the same operation is the Replace data section of a dataset's Options tab: choose the JSON file, click Replace all data, and the schema-change summary is shown when it completes. New fields appear in Field configuration unconfigured; mark them searchable or filterable there to use them.

The key field is preserved across a replace, so documents keep stable identities between reloads — provided the declared key field is a unique whole number present on every document. If no field qualifies, the dataset uses auto-generated keys.

Lifecycle

POST /api/teams/{teamName}/datasets/{dataSetName}/hibernate — release memory while preserving the document store in the database (204).

POST /api/teams/{teamName}/datasets/{dataSetName}/wakeup — reload and re-index from the database (204). The engine will be back in Ready state when GET status returns systemState == 4.