Images and PDFs¶
Send images and PDF documents to a model alongside your text, on
/v1/chat/completions and /v1/messages. Files go inline in the request,
base64-encoded.
Check that the model accepts them¶
Reading images or PDFs is a property of the model, and not every model does. A model that does not read images refuses an image rather than ignoring it.
Each entry in GET /v1/models lists what the model reads in
input_modalities:
curl -s https://api.mangoboost.io/v1/models \
-H "Authorization: Bearer $MANGOINFERENCE_API_KEY" \
| jq '.data[] | {id, input_modalities}'
input_modalities contains |
The model reads |
|---|---|
text |
Text. |
image |
Images. |
document |
PDF documents. |
Send an image¶
Send the image inline as base64. Remote http(s):// image URLs are not
fetched, so a request that points at one is refused.
Add an image_url content part whose url is a data URL,
data:<media type>;base64,<data>:
import base64
from openai import OpenAI
client = OpenAI(base_url="https://api.mangoboost.io/v1", api_key="<your-api-key>")
with open("chart.png", "rb") as f:
b64 = base64.b64encode(f.read()).decode()
resp = client.chat.completions.create(
model="<model-id>",
messages=[{
"role": "user",
"content": [
{"type": "text", "text": "What does this chart show?"},
{"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64}"}},
],
}],
)
print(resp.choices[0].message.content)
Add an image content block with a base64 source:
import base64
import anthropic
client = anthropic.Anthropic(base_url="https://api.mangoboost.io", api_key="<your-api-key>")
with open("chart.png", "rb") as f:
b64 = base64.b64encode(f.read()).decode()
resp = client.messages.create(
model="<model-id>",
max_tokens=1024,
messages=[{
"role": "user",
"content": [
{"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": b64}},
{"type": "text", "text": "What does this chart show?"},
],
}],
)
print(resp.content[0].text)
Use the media type that matches the file: image/jpeg, image/png,
image/webp or image/gif.
Send a PDF¶
Send the PDF inline as base64, in the same position as an image.
with open("report.pdf", "rb") as f:
b64 = base64.b64encode(f.read()).decode()
content = [
{"type": "text", "text": "Summarize this report."},
# TODO: verify this part shape against a document-capable model
{"type": "file", "file": {"filename": "report.pdf", "file_data": f"data:application/pdf;base64,{b64}"}},
]
with open("report.pdf", "rb") as f:
b64 = base64.b64encode(f.read()).decode()
content = [
# TODO: verify this block shape against a document-capable model
{"type": "document", "source": {"type": "base64", "media_type": "application/pdf", "data": b64}},
{"type": "text", "text": "Summarize this report."},
]
Limits¶
| Limit | Value |
|---|---|
| How files are sent | Inline base64 only. Remote URLs are not fetched. |
| Image formats | JPEG, PNG, WebP, GIF |
| Image size | Longest side at most 8000 px |
| Images per request | 1000 |
| PDF pages per request | 100 |
| Request body | 32 MiB, including the base64-encoded files. See Request size limit. |
Base64 makes a file about 4/3 of its size on disk, so the 32 MiB body limit fits a file of about 24 MiB at most, less whatever the rest of the request takes.
Image and page content is input to the model. It counts against the model's
context window and is reported and billed in prompt_tokens like any other
input.
Refusals¶
A file the model cannot take is refused before generation starts, so nothing is billed. Do not retry the same request unchanged: it fails the same way.
| Status | code |
Meaning |
|---|---|---|
413 |
request_too_large |
The request body is over 32 MiB. The message states the limit. |
400 |
image_url_not_accepted |
The model refused an image_url part. The message says why. |
400 |
upstream_invalid_request |
The model refused the request without a more specific code. The message says why. |
On /v1/chat/completions the code is the one to branch on, and param, when
present, names the content part that was refused. On /v1/messages the error
carries no code: branch on the status and read message. See
Upstream 400 relay.