Gemini PDF, audio and structured output

This page explains how to send PDFs and audio to Gemini and how to get structured output. For model IDs and endpoints, see the Gemini family overview.

PDF and audio

The native API takes base64-encoded files via inline_data, with a mime_type such as application/pdf, audio/wav or audio/mp3:

gemini_pdf.py


import os
from google import genai
from google.genai import types
 
client = genai.Client(
    api_key=os.environ["OFOX_API_KEY"],
    http_options={"api_version": "v1beta", "base_url": "https://api.ofox.io/gemini"},
)
 
pdf = open("invoice.pdf", "rb").read()
resp = client.models.generate_content(
    model="google/gemini-3.5-flash",
    contents=[types.Part.from_bytes(data=pdf, mime_type="application/pdf"), "What is the total amount on this invoice?"],
)
print(resp.text)

On the OpenAI-compatible API, PDFs use a file content block (file_data as data:application/pdf;base64,...), and audio uses an input_audio content block ({"data": "<base64>", "format": "wav"}).

Structured output

On the native API, set responseMimeType and responseSchema in generationConfig:

gemini_json_schema.py


import os
from google import genai
from google.genai import types
from pydantic import BaseModel
 
class Price(BaseModel):
    price: float
    currency: str
 
client = genai.Client(
    api_key=os.environ["OFOX_API_KEY"],
    http_options={"api_version": "v1beta", "base_url": "https://api.ofox.io/gemini"},
)
 
resp = client.models.generate_content(
    model="google/gemini-3.5-flash",
    contents="A bat and a ball cost $1.10 in total. The bat costs $1 more than the ball. How much does the ball cost?",
    config=types.GenerateContentConfig(response_mime_type="application/json", response_schema=Price),
)
print(resp.parsed)

On the OpenAI-compatible API, use response_format: {"type": "json_schema", ...}; see Structured output for the syntax.

For other errors, see Common errors.