Skip to content
Embabel Worlds

Ingest a page by URL as a job

POST
/api/v1/documents/ingests/url
curl --request POST \
  --url http://localhost:11044/api/v1/documents/ingests/url \
  --header 'Content-Type: application/json' \
  --header 'X-Embabel-Api-Key: <X-Embabel-Api-Key>' \
  --data '{ "fromOrgDomain": "example", "tags": [ "example" ], "url": "example" }'

Answers 202 at once with a job to follow at GET /documents/ingests/{id}.

Authorizations

Section titled “Authorizations”

Request Bodyrequired

Section titled “Request Bodyrequired”

What to fetch and ingest, read as by POST /api/v1/documents/url but run as a job: url is fetched and ingested after the 202, and names the job; its host’s domain is the publisher unless fromOrgDomain names another; tags are trimmed, blanks dropped and duplicates collapsed, and omitted leaves the document untagged

Media typeapplication/json

Body of POST /documents/url and POST /documents/ingests/url: a page or file to fetch and ingest into the caller’s world.

object
fromOrgDomain

Domain of the organisation the document is really from, recorded as its publisher. Absent or blank means the URL’s host

string
tags

Tags to put on the document and its chunks. Blanks are dropped and duplicates collapsed; absent means none

Array<string>
url
required

The URL to fetch — a web page, PDF or other document. It also becomes the document’s uri

string

Example generated

{
  "fromOrgDomain": "example",
  "tags": [
    "example"
  ],
  "url": "example"
}

Responses

Section titled “Responses”

202

Section titled “202”

Accepted; the job, queued

Media type*/*

One document ingest job as its owner sees it: its stage, embedding progress, and how it ended. Answered by POST /documents/ingests, POST /documents/ingests/url, POST /documents/embed-missing and GET /documents/ingests/{id}

object
chunks

Chunks the stored document was cut into; null until the job succeeds, and when the store does not say

integer format: int32
chunksWithoutEmbeddings

Chunks of the stored document with no embedding, which never match a semantic search; null until the job succeeds, and when the store does not say. POST /documents/embed-missing can embed them

integer format: int32
error

Why the job failed, in words; null unless state is failed

string
id
required

Job id, a UUID; the {id} of GET /documents/ingests/{id}

string
name
required

What the job is called: the uploaded file’s name, the URL being ingested, or for an embed-missing job the document’s title (its URI when it has none)

string
progress

Chunks embedded so far, of how many; null until the job starts embedding

object
done
required

Chunks embedded so far; never more than total

integer format: int32
total
required

Chunks to embed: an estimate from the document’s text while the job runs, corrected to the true count when it succeeds; raised rather than exceeded when the document cuts into more

integer format: int32
resumes
required

Times the appliance restarted while the job was unfinished and picked it up again under the same id; 0 for a job that ran straight through

integer format: int32
startedAt
required

When the job was submitted, ISO-8601 instant

string
state
required

Stage the job is at: queued, waiting_for_memory, converting, writing, embedding, succeeded or failed; the last two are final

string
Allowed values: queued waiting_for_memory converting writing embedding succeeded failed
title

Title of the stored document; null until the job succeeds, and when the document has none

string
updatedAt
required

When the job last changed stage or embedded a batch of chunks, ISO-8601 instant; a value that stops moving while the job is unfinished suggests it is stuck

string
uri

URI of the stored document; null until the job succeeds

string

409

Section titled “409”

Nothing can embed; not started

Media type*/*

A 409 from an ingest-job endpoint: this appliance has no embedding model yet, so nothing was spooled or started. Not set up, rather than broken

object
embeddings
required

The embedding status this was decided from: configured, model, dimensions and reason

object
key
additional properties
message
required

Why, in the operator’s words, naming the command that sets an embedding model

string
status
required

Always unavailable

string