Skip to content
Embabel Worlds

Ingest from URL

POST
/api/v1/documents/url
curl --request POST \
  --url http://localhost:11044/api/v1/documents/url \
  --header 'Content-Type: application/json' \
  --header 'X-Embabel-Api-Key: <X-Embabel-Api-Key>' \
  --data '{ "fromOrgDomain": "example", "tags": [ "example" ], "url": "example" }'

Fetch and ingest a document from a URL (web page, PDF, etc.). The optional fromOrgDomain field overrides the publisher domain inferred from the URL host.

Authorizations

Section titled “Authorizations”

Request Bodyrequired

Section titled “Request Bodyrequired”

What to fetch and ingest into the caller’s world, synchronously. url is fetched, converted and chunked; its host’s domain is recorded as the publisher unless fromOrgDomain names another (blank keeps the host). tags are trimmed, blanks dropped and duplicates collapsed; omitted or empty leaves the document untagged

Media typeapplication/json

Body of POST /documents/url and POST /documents/ingests/url: a page or file to fetch and ingest into the caller’s world.

object
fromOrgDomain

Domain of the organisation the document is really from, recorded as its publisher. Absent or blank means the URL’s host

string
tags

Tags to put on the document and its chunks. Blanks are dropped and duplicates collapsed; absent means none

Array<string>
url
required

The URL to fetch — a web page, PDF or other document. It also becomes the document’s uri

string

Example generated

{
  "fromOrgDomain": "example",
  "tags": [
    "example"
  ],
  "url": "example"
}

Responses

Section titled “Responses”

200

Section titled “200”

Document ingested. status is ingested, or ingested_incomplete when some chunks could not be embedded: the document is kept and text-searchable, chunksWithoutEmbeddings of chunks miss semantic search

Media type*/*

A document ingested. Kept and text-searchable even when some chunks could not be embedded

object
chunks

How many chunks were stored, null where the stored record could not be read back

integer format: int32
chunksWithoutEmbeddings
required

How many chunks have no embedding. Zero when every chunk was embedded

integer format: int32
message

Present only for ingested_incomplete: how many chunks miss embeddings, and that re-ingesting embeds them

string
status
required

ingested, or ingested_incomplete when some chunks could not be embedded

string
Allowed values: ingested ingested_incomplete
title

The document’s title. For inline text, the given name where no title was found

string
uri
required

The document’s URI

string

409

Section titled “409”

No embedding model is set, so nothing could be searched: the reason names the command that sets one

Media type*/*

An ingest refused because this appliance has no embedding model yet. Not set up, rather than broken

object
embeddings
required

The embedding status this was decided from: configured, model, dimensions and reason

object
key
additional properties
any
message
required

Why, in the operator’s words, naming the command that sets a model

string
status
required

Always unavailable

string

Example 

unavailable

500

Section titled “500”

Fetch or ingestion failed

Media type*/*

An ingest that failed: the word error and the reason

object
message
required

What failed — the fetch or the conversion — in the failure’s own words

string
status
required

Always error

string

Example 

error