Skip to content

Images and PDFs

Send images and PDF documents to a model alongside your text, on /v1/chat/completions and /v1/messages. Files go inline in the request, base64-encoded.

Check that the model accepts them

Reading images or PDFs is a property of the model, and not every model does. A model that does not read images refuses an image rather than ignoring it.

Each entry in GET /v1/models lists what the model reads in input_modalities:

curl -s https://api.mangoboost.io/v1/models \
  -H "Authorization: Bearer $MANGOINFERENCE_API_KEY" \
  | jq '.data[] | {id, input_modalities}'
input_modalities contains The model reads
text Text.
image Images.
document PDF documents.

Send an image

Send the image inline as base64. Remote http(s):// image URLs are not fetched, so a request that points at one is refused.

Add an image_url content part whose url is a data URL, data:<media type>;base64,<data>:

import base64
from openai import OpenAI

client = OpenAI(base_url="https://api.mangoboost.io/v1", api_key="<your-api-key>")

with open("chart.png", "rb") as f:
    b64 = base64.b64encode(f.read()).decode()

resp = client.chat.completions.create(
    model="<model-id>",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "What does this chart show?"},
            {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64}"}},
        ],
    }],
)
print(resp.choices[0].message.content)

Add an image content block with a base64 source:

import base64
import anthropic

client = anthropic.Anthropic(base_url="https://api.mangoboost.io", api_key="<your-api-key>")

with open("chart.png", "rb") as f:
    b64 = base64.b64encode(f.read()).decode()

resp = client.messages.create(
    model="<model-id>",
    max_tokens=1024,
    messages=[{
        "role": "user",
        "content": [
            {"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": b64}},
            {"type": "text", "text": "What does this chart show?"},
        ],
    }],
)
print(resp.content[0].text)

Use the media type that matches the file: image/jpeg, image/png, image/webp or image/gif.

Send a PDF

Send the PDF inline as base64, in the same position as an image.

with open("report.pdf", "rb") as f:
    b64 = base64.b64encode(f.read()).decode()

content = [
    {"type": "text", "text": "Summarize this report."},
    # TODO: verify this part shape against a document-capable model
    {"type": "file", "file": {"filename": "report.pdf", "file_data": f"data:application/pdf;base64,{b64}"}},
]
with open("report.pdf", "rb") as f:
    b64 = base64.b64encode(f.read()).decode()

content = [
    # TODO: verify this block shape against a document-capable model
    {"type": "document", "source": {"type": "base64", "media_type": "application/pdf", "data": b64}},
    {"type": "text", "text": "Summarize this report."},
]

Limits

Limit Value
How files are sent Inline base64 only. Remote URLs are not fetched.
Image formats JPEG, PNG, WebP, GIF
Image size Longest side at most 8000 px
Images per request 1000
PDF pages per request 100
Request body 32 MiB, including the base64-encoded files. See Request size limit.

Base64 makes a file about 4/3 of its size on disk, so the 32 MiB body limit fits a file of about 24 MiB at most, less whatever the rest of the request takes.

Image and page content is input to the model. It counts against the model's context window and is reported and billed in prompt_tokens like any other input.

Refusals

A file the model cannot take is refused before generation starts, so nothing is billed. Do not retry the same request unchanged: it fails the same way.

Status code Meaning
413 request_too_large The request body is over 32 MiB. The message states the limit.
400 image_url_not_accepted The model refused an image_url part. The message says why.
400 upstream_invalid_request The model refused the request without a more specific code. The message says why.

On /v1/chat/completions the code is the one to branch on, and param, when present, names the content part that was refused. On /v1/messages the error carries no code: branch on the status and read message. See Upstream 400 relay.