Skip to content

External Storage

When RPC payloads grow large — multi-megabyte query results, bulk data imports, large model artifacts — they can exceed the capacity of the underlying transport. HTTP servers and reverse proxies typically enforce request and response body size limits (e.g., 10 MB). Even pipe-based transports become inefficient when serializing very large batches inline.

External storage solves this by transparently offloading oversized Arrow IPC batches to cloud storage (S3, GCS, or any compatible backend). The batch is replaced with a lightweight pointer batch — a zero-row batch carrying a download URL in its metadata. The receiving side resolves the pointer automatically, fetching the actual data in parallel chunks.

This works in both directions:

  • Large outputs — the server externalizes result batches that exceed a size threshold
  • Large inputs — the client uploads input data to a pre-signed URL and sends a pointer in the RPC request

From the caller's perspective, nothing changes. The proxy returns the same typed results; the externalization is invisible.

How It Works

Large Outputs (Server to Client)

When a server method returns a batch that exceeds externalize_threshold_bytes, the framework intercepts the response before writing it to the wire:

  1. The batch is serialized to Arrow IPC format
  2. Optionally compressed with zstd
  3. Uploaded to cloud storage via the configured ExternalStorage backend
  4. The storage backend returns a download URL (typically a pre-signed URL with an expiry)
  5. A pointer batch replaces the original — a zero-row batch with vgi_rpc.location metadata containing the URL
  6. The pointer batch is written to the wire instead of the full data

On the client side, pointer batches are detected and resolved transparently:

  1. The client reads the pointer batch and extracts the URL from vgi_rpc.location metadata
  2. A HEAD probe determines the content size and whether the server supports range requests
  3. For large payloads (>64 MB by default), parallel range-request fetching splits the download into chunks with speculative hedging for slow chunks
  4. For smaller payloads, a single GET request fetches the data
  5. If the data was compressed, it is decompressed automatically
  6. The Arrow IPC stream is parsed back into a RecordBatch and returned to the caller

This works for both unary results and streaming outputs. For streaming, all batches in a single output cycle (including any log batches) are serialized as one IPC stream, uploaded together, and resolved as a unit.

Large Inputs (Client to Server)

For the reverse direction — when a client needs to send large data to the server — the framework provides upload URLs. This is particularly important for HTTP transport where request body size limits are common.

  1. The client serializes the request to Arrow IPC, hashes it, and calls request_upload_urls() with one (size, sha256) entry per body
  2. Each UploadUrl contains an upload_url (for PUT) signed for exactly that size and digest, a download_url (for GET), and the headers the PUT must carry
  3. The client PUTs exactly those bytes to the upload_url with those headers, bypassing the RPC server entirely; storage refuses any other body where the backend can verify it (S3, R2, the blob vendor)
  4. The client sends the RPC request with a pointer batch referencing the download_url and carrying vgi_rpc.location.sha256
  5. The server verifies the digest while resolving the pointer, and refuses a request pointer without one

This pattern lets clients send arbitrarily large payloads even when the HTTP server enforces strict request size limits — the large data goes directly to cloud storage, and only a small pointer crosses the RPC boundary.

The upload URL endpoint is enabled by passing upload_url_provider to make_wsgi_app(). The server advertises this capability via the VGI-Upload-URL-Support: 2 response header (upload-URL protocol v2; the retired true was the unsized count protocol, which a v2 client refuses), and clients can discover it with http_capabilities() (upload_url_version == 2). The HTTP client does all of this automatically when a request exceeds the server's max_request_bytes.

Server Configuration

S3Storage requires pip install vgi-rpc[s3] (and GCSStorage requires [gcs]); the symbols are only exported from vgi_rpc when their backend dependency is installed.

from vgi_rpc import Compression, ExternalLocationConfig, FetchConfig, RpcServer, S3Storage

storage = S3Storage(bucket="my-bucket", prefix="rpc-data/")
config = ExternalLocationConfig(
    storage=storage,
    externalize_threshold_bytes=1_048_576,  # 1 MiB — batches above this go to S3
    compression=Compression(),              # zstd level 3 by default
    fetch_config=FetchConfig(
        chunk_size_bytes=8 * 1024 * 1024,   # 8 MiB parallel chunks
        max_parallel_requests=8,
    ),
)

server = RpcServer(MyService, MyServiceImpl(), external_location=config)

The same ExternalLocationConfig is passed to the client so it can resolve pointer batches in responses:

from vgi_rpc import connect

with connect(MyService, ["python", "worker.py"], external_location=config) as proxy:
    result = proxy.large_query()  # pointer batches resolved transparently

For HTTP transport with upload URL support:

from vgi_rpc import make_wsgi_app

app = make_wsgi_app(
    server,
    upload_url_provider=storage,       # enables __upload_url__ endpoint
    max_upload_bytes=100 * 1024 * 1024,  # refuse (400) uploads over 100 MiB
)

Client Upload URLs

import hashlib

import httpx2

from vgi_rpc import UploadUrl, request_upload_urls
from vgi_rpc.http import upload_put_headers

ipc_data = serialize_large_batch(my_batch)
digest = hashlib.sha256(ipc_data).hexdigest()

# One pre-signed URL pair, signed for exactly these bytes
urls: list[UploadUrl] = request_upload_urls(
    base_url="http://localhost:8080",
    uploads=[(len(ipc_data), digest)],
)

# Upload directly to storage (bypasses the RPC server), with the vended headers
httpx2.put(urls[0].upload_url, content=ipc_data, headers=upload_put_headers(urls[0]))

# Send the RPC request with a pointer naming urls[0].download_url and
# carrying vgi_rpc.location.sha256 = digest; the server verifies it

Clients can discover whether the server supports upload URLs:

from vgi_rpc import http_capabilities

caps = http_capabilities("http://localhost:8080")
if caps.upload_url_version == 2:
    urls = request_upload_urls("http://localhost:8080", uploads=[(len(body), digest)])

Compression

When compression is enabled, batches are compressed with zstd before upload and decompressed transparently on fetch:

from vgi_rpc import Compression

# Default: zstd level 3
config = ExternalLocationConfig(
    storage=storage,
    compression=Compression(),  # algorithm="zstd", level=3
)

# Custom compression level (higher = smaller but slower)
config = ExternalLocationConfig(
    storage=storage,
    compression=Compression(level=9),
)

# gzip — stdlib zlib, the right choice when consumers can't do zstd
config = ExternalLocationConfig(
    storage=storage,
    compression=Compression(algorithm="gzip"),  # level defaults to 6
)

# Disable compression
config = ExternalLocationConfig(
    storage=storage,
    compression=None,
)

Compression(algorithm=...) accepts "zstd" (default; needs zstandard on both writer and reader) or "gzip" (stdlib zlib, no extra dependency). The storage backend stores the Content-Encoding header alongside the object; on fetch the Content-Encoding: zstd / gzip header triggers automatic decompression.

zstd requires pip install vgi-rpc[external] (installs zstandard); gzip works with no extra dependency.

Parallel Fetching

For large externalized batches, the client fetches data in parallel chunks using range requests. This significantly reduces download time for multi-megabyte payloads:

  • A HEAD probe determines the total size and whether the server supports Accept-Ranges: bytes
  • If the payload exceeds parallel_threshold_bytes (default 64 MB) and ranges are supported, it is split into chunk_size_bytes chunks (default 8 MB) fetched concurrently
  • Speculative hedging: if a chunk takes longer than speculative_retry_multiplier times the median chunk time, a hedge request is launched in parallel — the first response wins
  • For smaller payloads or servers without range support, a single GET request is used
from vgi_rpc import FetchConfig

fetch_config = FetchConfig(
    parallel_threshold_bytes=64 * 1024 * 1024,  # 64 MiB
    chunk_size_bytes=8 * 1024 * 1024,            # 8 MiB chunks
    max_parallel_requests=8,                      # concurrent fetches
    timeout_seconds=60.0,                         # overall deadline
    max_fetch_bytes=256 * 1024 * 1024,            # 256 MiB hard cap
    speculative_retry_multiplier=2.0,             # hedge at 2x median
    max_speculative_hedges=4,                     # max hedge requests
)

Pre-published References

Per-call externalization re-serializes, re-compresses and re-uploads a result on every call. When a large result rarely changes — a worker's whole catalog, say — publish it once with publish_external(), keep the returned ExternalRef, and return that ref from the unary method on later calls. The server writes the pointer batch directly: no serialization, compression or upload happens during the call.

import threading
from typing import Protocol

import pyarrow as pa

from vgi_rpc import Compression, ExternalRef, ExternalStorage, publish_external, rpc_methods


class CatalogService(Protocol):
    def catalog(self) -> str: ...


class CatalogServiceImpl:
    def __init__(self, storage: ExternalStorage, compression: Compression | None) -> None:
        self._storage, self._compression = storage, compression
        self._ref: ExternalRef | None = None
        self._lock = threading.Lock()

    def catalog(self) -> str | ExternalRef:
        with self._lock:
            if self._ref is None:  # publish once per process / catalog version
                schema = rpc_methods(CatalogService)["catalog"].result_schema
                batch = pa.RecordBatch.from_pydict({"result": [build_catalog()]}, schema=schema)
                self._ref = publish_external(batch, self._storage, self._compression)
            return self._ref

A unary method may return an ExternalRef instead of its declared value. The return annotation may say so (-> str | ExternalRef); ExternalRef is stripped when the result schema is derived, so the wire contract (and protocol hash) is that of str. Returning a ref:

  • always answers with a pointer — whether or not the server has external storage configured, and regardless of externalize_threshold_bytes (a ref is never inlined, nor routed through shared memory);
  • skips building and validating the result value;
  • is not counted toward max_externalized_response_bytes (nothing is uploaded during the call); the tiny pointer still goes through the wire-body budget;
  • works on every transport (pipe, subprocess, Unix, TCP, HTTP). Stream methods are not supported.

publish_external(batch, storage, compression=None, *, include_sha256=True) serializes the 1-row result batch exactly as the per-call externalizer would, hashes the raw IPC bytes, compresses if asked, and calls storage.upload() once. With include_sha256=False the ref carries no digest, so clients skip the content check — for an object rewritten in place, or one too large to hash. Clients need no change: a ref's pointer is indistinguishable from any other.

The caller owns the ref and the object's lifecycle:

  • A long-lived ref must not point at an object under the short-TTL lifecycle rule used for per-call uploads (see Object Lifecycle) — publish under a different prefix.
  • A pre-signed URL expires; re-sign or rebuild the ref before it does.
  • Only return a ref to callers who are all entitled to the same content.

Object Lifecycle

Uploaded objects persist indefinitely — vgi-rpc does not delete them. Configure storage-level cleanup policies to auto-expire old data.

S3 lifecycle rule:

aws s3api put-bucket-lifecycle-configuration \
  --bucket MY_BUCKET \
  --lifecycle-configuration '{
    "Rules": [{
      "ID": "expire-vgi-rpc",
      "Filter": {"Prefix": "rpc-data/"},
      "Status": "Enabled",
      "Expiration": {"Days": 1}
    }]
  }'

GCS lifecycle rule:

gsutil lifecycle set <(cat <<EOF
{"rule": [{"action": {"type": "Delete"},
           "condition": {"age": 1, "matchesPrefix": ["rpc-data/"]}}]}
EOF
) gs://MY_BUCKET

Blob Vendor

The blob vendor is a small HTTP service that holds the object-storage credentials so workers never do. A worker asks it for a pre-signed URL for one object of a declared size and SHA-256, uploads straight to storage, and hands on a download URL. The storage (Cloudflare R2) enforces the signed Content-Length and x-amz-checksum-sha256, so whatever sits at the object's key is exactly the declared bytes. Buckets are per region (EU / US jurisdictions, or a location hint), chosen by the vendor from the caller's key, never from the request.

VendorStorage implements both ExternalStorage (server-side externalization) and UploadUrlProvider (client upload URLs):

from vgi_rpc.vendor import VendorStorage, external_config_from_env

storage = VendorStorage("https://blobs.example.com", token)

# Or, from the environment (returns None when VGI_BLOB_VENDOR_URL is unset):
config = external_config_from_env()
server = RpcServer(MyService, MyServiceImpl(), external_location=config)
app = make_wsgi_app(
    server,
    max_request_bytes=8 * 1024 * 1024,
    upload_url_provider=config.storage if config else None,
)

serve_http() takes the same upload_url_provider / max_upload_bytes arguments, and run_server() applies the environment automatically under --http.

For raw bytes that the caller references itself (e.g. large values in a key-value store), VendorBlobClient stores and verifies blobs by BlobRef:

from vgi_rpc.vendor import BlobScope, VendorBlobClient

with VendorBlobClient.from_env() as blobs:
    # Scope fields: 1-128 chars of [A-Za-z0-9._:-], so hex-encode binary ids.
    ref = blobs.put_bytes(value, scope=BlobScope(shard="s1", execution_id=exec_id_hex), idempotency_key="attempt-7:0")
    assert blobs.get_bytes(ref) == value  # size + SHA-256 checked against ref
    blobs.delete_scope("s1", exec_id_hex)

Environment variables

Variable Meaning
VGI_BLOB_VENDOR_URL Vendor base URL. Unset: the *_from_env configs return None.
VGI_BLOB_VENDOR_TOKEN Bearer token; falls back to VGI_CF_DO_TOKEN.
VGI_RPC_EXTERNALIZE_THRESHOLD_BYTES Result size above which the server externalizes (default 1 MiB).
VGI_RPC_EXTERNAL_COMPRESSION none (default), zstd or gzip, optionally :<level>.
VGI_RPC_MAX_UPLOAD_BYTES Upload size advertised to clients (VGI-Max-Upload-Bytes).
VGI_RPC_MAX_REQUEST_BYTES Request-body cap for run_server --http (--max-request-bytes); clients send larger requests through vendor upload URLs. Without a cap, clients never upload.

Host pinning

Configs built from the environment do not use https_only_validator, which accepts any host. They pin pointer and upload URLs to the vendor's storage hosts and URL prefixes (the region's bucket), read from its GET /v1/info and cached for ten minutes; a failed refresh keeps the last good list, and with no list at all every URL is rejected. VendorStorage also checks the URLs the vendor itself returns before using them. Build a pin by hand with pinned_hosts_validator:

from vgi_rpc.vendor import pinned_hosts_validator

validate = pinned_hosts_validator(["acct.eu.r2.cloudflarestorage.com"])

An entry matches only that exact host (no subdomains) on the default https port, or the port it names. Only https passes; plain http is accepted only when the vendor URL itself is a loopback http URL (a local test vendor).

Retries and secrets

Vendor and signed-URL requests retry transport errors, 429 and 5xx with exponential backoff (3 attempts by default); 4xx responses are not retried. Failures raise VendorError with the HTTP status and the vendor's code (not_found, not_uploaded, quota_exceeded, object_too_large, idempotency_conflict, ...) or the storage's (BadDigest). Pass idempotency_key to put_bytes so a retried logical write reuses one object. The bearer token goes only to the vendor, never to a signed URL. Every request to a storage host (the vendor clients' PUTs, and the HTTP client's upload of a large request) goes on a bare, single-purpose client: never the caller's httpx2.Client, so its default headers, cookies, auth flow and event hooks cannot reach the storage host. A session= passed to the vendor clients supplies only the transport for vendor API calls.

vgi-rpc's own logs and errors show URLs through redact_url (scheme, host, port and path only). httpx2 logs every request with its full URL at INFO on the global httpx2 logger, and for a signed URL the query string is the credential, so the vendor clients and the upload path install a filter on that logger (vgi_rpc.external_fetch.redact_httpx_logging(), idempotent) that redacts the URLs in those lines. The lines are still logged at the same level; call the helper yourself to cover other httpx2 use in the process.

Testing

vgi_rpc.conformance.fake_vendor is an in-process fake of the vendor API and of R2's signed-URL enforcement (declared size and checksum, method-bound signatures, range GETs), with fault injection:

from vgi_rpc.conformance.fake_vendor import serve_fake_vendor

with serve_fake_vendor({"tok": "tenant-a"}) as fake:
    storage = VendorStorage(fake.url, "tok")
    fake.vendor.fail_next("/v1/blobs/put", status=503)

The same fake runs standalone for test lanes in any language. It prints PORT:<n> and then URL:<base>, and needs only vgi-rpc's base dependencies:

uvx --from vgi-rpc vgi-rpc-fake-vendor --test-routes --token test-token=tenant-a

--test-routes adds unauthenticated routes. Each answers GET with a JSON body padded to 1,024 bytes, and a HEAD has no side effects:

  • GET /__stats__: call and byte counters (puts, gets, deletes, delete_scopes, upload_urls, storage_puts, storage_gets, objects, bytes, bytes_uploaded, bytes_downloaded), signed-PUT refusals (upload_refusals, upload_bad_digest, upload_incomplete_body, upload_signature_mismatch, uploads_corrupted), and counters by purpose: puts_rpc, puts_fs and puts_upload split the /v1/blobs/put grants issued into server rpc writes (results), function storage, and grants minted for a client upload; client_uploads, client_upload_bytes and client_upload_refusals count the signed PUTs to client-upload objects. A server marks a client-upload grant with the advisory origin: "client_upload" field, which VendorStorage.generate_upload_url sends and the real vendor ignores.
  • GET /__admin__/fault?purpose=fs|rpc&action=corrupt|missing&count=N: the next N objects of that purpose that are downloaded are damaged for every later GET. corrupt keeps the length and changes every byte; missing answers 404.
  • GET /__admin__/fault?purpose=upload&action=corrupt&count=N: the next N signed PUTs of client uploads are damaged in transit, so the store refuses them with 400 BadDigest. Server writes are never touched.
  • GET /__admin__/reset: clears all faults.

The standalone fake logs to stderr which stop signal it received, the sending pid and its command line (Linux), and its final counters.

In-process, the same faults are available as fake.vendor.set_download_fault(...) and reset_faults(), and the counters as test_stats().

URL Validation

By default, external location URLs are validated with https_only_validator, which rejects non-HTTPS URLs. This prevents pointer batches from being used to probe internal networks. You can provide a custom validator via ExternalLocationConfig.url_validator; for the blob vendor use the host-pinned validators described in Host pinning.

Client uploads to a vended URL are sent with Content-Type: application/octet-stream, the type pre-signed URLs that sign the content type (such as GCSStorage.generate_upload_url) expect.

API Reference

Configuration

ExternalLocationConfig module-attribute

ExternalLocationConfig = ServerExternalConfig

Compression dataclass

Compression(
    algorithm: Literal["zstd", "gzip"] = "zstd",
    level: int = 3,
)

Compression settings for externalized data.

ATTRIBUTE DESCRIPTION
algorithm

Compression algorithm. "zstd" (default) needs zstandard on both the writer and the reader; "gzip" uses stdlib zlib and is the right choice when consumers (e.g. browsers without a zstd polyfill, generic HTTP tooling) can't do zstd.

TYPE: Literal['zstd', 'gzip']

level

Compression level. Codec-specific — 1-22 for zstd (default 3), 1-9 for gzip (default 6 when level is left at the zstd-shaped default of 3, since gzip-3 produces noticeably worse ratios).

TYPE: int

FetchConfig dataclass

FetchConfig(
    parallel_threshold_bytes: int = 64 * 1024 * 1024,
    chunk_size_bytes: int = 8 * 1024 * 1024,
    max_parallel_requests: int = 8,
    timeout_seconds: float = 60.0,
    max_fetch_bytes: int = 256 * 1024 * 1024,
    max_decompressed_bytes: int | None = None,
    max_redirects: int = 5,
    speculative_retry_multiplier: float = 2.0,
    max_speculative_hedges: int = 4,
)

Configuration for parallel range-request fetching.

Maintains a persistent aiohttp.ClientSession backed by a daemon thread. Use as a context manager or call close() to release resources.

ATTRIBUTE DESCRIPTION
parallel_threshold_bytes

Below this size, use a single GET.

TYPE: int

chunk_size_bytes

Size of each Range request chunk.

TYPE: int

max_parallel_requests

Semaphore limit on concurrent requests.

TYPE: int

timeout_seconds

Overall deadline for the fetch.

TYPE: float

max_fetch_bytes

Hard cap on encoded/on-wire download size.

TYPE: int

max_decompressed_bytes

Hard cap after content decoding. None preserves the historical effective limit of 16 * max_fetch_bytes.

TYPE: int | None

max_redirects

Maximum redirects followed by each probe or data request. Every target is passed through the caller's URL validator before it is requested. Set to 0 to reject redirects.

TYPE: int

speculative_retry_multiplier

Launch a hedge request for chunks taking longer than multiplier * median. Set to 0 to disable hedging.

TYPE: float

max_speculative_hedges

Maximum number of hedge requests per fetch. Prevents runaway request amplification under adversarial or slow-backend conditions. 0 means no limit (bounded only by chunk count).

TYPE: int

close

close() -> None

Close the pooled session, stop the event loop, and join the thread.

Source code in vgi_rpc/external_fetch.py
def close(self) -> None:
    """Close the pooled session, stop the event loop, and join the thread."""
    pool = self._pool
    with pool.lock:
        loop = pool.loop
        thread = pool.thread
        if pool.session is not None and loop is not None and not loop.is_closed():
            asyncio.run_coroutine_threadsafe(pool.session.close(), loop).result(timeout=5)
        pool.session = None
        if loop is not None and not loop.is_closed():
            loop.call_soon_threadsafe(loop.stop)
        if thread is not None:
            thread.join(timeout=5)
            if thread.is_alive():
                raise RuntimeError("external fetch event-loop thread did not stop within 5s")
        # ``loop.stop()`` and joining its thread do not release the event
        # loop's selector or self-pipe descriptors.  Close it explicitly
        # before dropping the last owned reference; repeated short-lived
        # client configs otherwise exhaust macOS's common 256-FD limit.
        if loop is not None and not loop.is_closed():
            loop.close()
        pool.loop = None
        pool.thread = None

__del__

__del__() -> None

Safety net: close pool on garbage collection.

Source code in vgi_rpc/external_fetch.py
def __del__(self) -> None:
    """Safety net: close pool on garbage collection."""
    with contextlib.suppress(Exception):
        self.close()

__enter__

__enter__() -> FetchConfig

Enter context manager.

Source code in vgi_rpc/external_fetch.py
def __enter__(self) -> FetchConfig:
    """Enter context manager."""
    return self

__exit__

__exit__(
    exc_type: type[BaseException] | None,
    exc_val: BaseException | None,
    exc_tb: TracebackType | None,
) -> None

Exit context manager, closing the pool.

Source code in vgi_rpc/external_fetch.py
def __exit__(
    self,
    exc_type: type[BaseException] | None,
    exc_val: BaseException | None,
    exc_tb: TracebackType | None,
) -> None:
    """Exit context manager, closing the pool."""
    self.close()

Pre-published References

ExternalRef dataclass

ExternalRef(url: str, sha256: str | None = None)

A reference to an already-published unary result.

A unary method may return an ExternalRef in place of its declared result value. The server then answers with the ExternalLocation pointer batch for url directly — no result serialization, compression, or upload happens during the call, and the ref is used whether or not the server has external storage configured and regardless of externalize_threshold_bytes. Clients resolve it like any other pointer, so they need no change.

Build one with :func:publish_external (or by hand for an object published out of band). The object at url must be an Arrow IPC stream (optionally Content-Encoding-compressed) whose schema is the method's result schema and which holds exactly one 1-row data batch.

The caller owns caching the ref and the object's lifecycle: a long-lived ref must not point at an object under the short-TTL lifecycle rule used for per-call uploads, and a pre-signed URL expires — re-sign or rebuild the ref before then. Only return a ref to callers who are all entitled to the same content.

ATTRIBUTE DESCRIPTION
url

Where the published IPC stream lives.

TYPE: str

sha256

Lowercase hex SHA-256 of the raw (pre-compression) IPC stream bytes, sent as vgi_rpc.location.sha256. None omits the key, so clients skip the content check — use this for an object rewritten in place or one too large to hash.

TYPE: str | None

__post_init__

__post_init__() -> None

Validate the URL and digest shape.

RAISES DESCRIPTION
ValueError

If url is empty or sha256 is not 64 lowercase hex characters.

Source code in vgi_rpc/external.py
def __post_init__(self) -> None:
    """Validate the URL and digest shape.

    Raises:
        ValueError: If ``url`` is empty or ``sha256`` is not 64
            lowercase hex characters.

    """
    if not self.url:
        raise ValueError("ExternalRef.url must be non-empty")
    if self.sha256 is not None and (
        len(self.sha256) != 64 or any(c not in "0123456789abcdef" for c in self.sha256)
    ):
        raise ValueError("ExternalRef.sha256 must be 64 lowercase hex characters (or None)")

pointer_batch

pointer_batch(
    schema: Schema,
) -> tuple[RecordBatch, KeyValueMetadata]

Build the zero-row pointer batch announcing this ref.

PARAMETER DESCRIPTION
schema

The method's result schema.

TYPE: Schema

RETURNS DESCRIPTION
RecordBatch

(batch, custom_metadata) as from

KeyValueMetadata

func:make_external_location_batch.

Source code in vgi_rpc/external.py
def pointer_batch(self, schema: pa.Schema) -> tuple[pa.RecordBatch, pa.KeyValueMetadata]:
    """Build the zero-row pointer batch announcing this ref.

    Args:
        schema: The method's result schema.

    Returns:
        ``(batch, custom_metadata)`` as from
        :func:`make_external_location_batch`.

    """
    return make_external_location_batch(schema, self.url, sha256=self.sha256)

publish_external

publish_external(
    batch: RecordBatch,
    storage: ExternalStorage,
    compression: Compression | None = None,
    *,
    include_sha256: bool = True
) -> ExternalRef

Publish a unary result batch once and return a reusable reference.

Serializes batch exactly as the per-call externalizer does (an IPC stream of the schema plus this one batch), hashes the raw bytes, compresses when compression is given, and calls storage.upload once. Cache the returned :class:ExternalRef and return it from the unary method on later calls; the server writes the pointer directly.

Build batch against the method's result schema, e.g.::

schema = rpc_methods(MyService)["catalog"].result_schema
ref = publish_external(pa.RecordBatch.from_pydict({"result": [value]}, schema=schema), storage)
PARAMETER DESCRIPTION
batch

The 1-row result batch (single result column).

TYPE: RecordBatch

storage

Storage backend to upload to.

TYPE: ExternalStorage

compression

Optional compression applied before upload (pass the server's ServerExternalConfig.compression to match it).

TYPE: Compression | None DEFAULT: None

include_sha256

When False the ref carries no digest, so clients skip the content check.

TYPE: bool DEFAULT: True

RETURNS DESCRIPTION
An

class:ExternalRef for the uploaded object.

TYPE: ExternalRef

RAISES DESCRIPTION
ValueError

If batch does not have exactly one row.

Source code in vgi_rpc/external.py
def publish_external(
    batch: pa.RecordBatch,
    storage: ExternalStorage,
    compression: Compression | None = None,
    *,
    include_sha256: bool = True,
) -> ExternalRef:
    """Publish a unary result batch once and return a reusable reference.

    Serializes *batch* exactly as the per-call externalizer does (an IPC
    stream of the schema plus this one batch), hashes the raw bytes,
    compresses when *compression* is given, and calls ``storage.upload``
    once.  Cache the returned :class:`ExternalRef` and return it from the
    unary method on later calls; the server writes the pointer directly.

    Build *batch* against the method's result schema, e.g.::

        schema = rpc_methods(MyService)["catalog"].result_schema
        ref = publish_external(pa.RecordBatch.from_pydict({"result": [value]}, schema=schema), storage)

    Args:
        batch: The 1-row result batch (single ``result`` column).
        storage: Storage backend to upload to.
        compression: Optional compression applied before upload (pass the
            server's ``ServerExternalConfig.compression`` to match it).
        include_sha256: When ``False`` the ref carries no digest, so
            clients skip the content check.

    Returns:
        An :class:`ExternalRef` for the uploaded object.

    Raises:
        ValueError: If *batch* does not have exactly one row.

    """
    if batch.num_rows != 1:
        raise ValueError(f"publish_external expects a 1-row result batch, got {batch.num_rows} rows")
    url, data_sha256, _raw_size = _upload_ipc_bytes(
        _serialize_single_batch(batch, None), batch.schema, storage, compression
    )
    return ExternalRef(url=url, sha256=data_sha256 if include_sha256 else None)

Storage Protocol

ExternalStorage

Bases: Protocol

Pluggable storage interface for externalizing large batches.

Implementations must be thread-safe — upload() may be called concurrently from different server threads.

.. important:: Object lifecycle — vgi-rpc does not manage the lifecycle of uploaded objects. Data uploaded via upload() or generate_upload_url() persists indefinitely unless the server operator configures cleanup. Use storage-level lifecycle rules (e.g. S3 Lifecycle Policies, GCS Object Lifecycle Management) or bucket-level TTLs to automatically expire and delete stale objects.

upload

upload(
    data: bytes,
    schema: Schema,
    *,
    content_encoding: str | None = None
) -> str

Upload serialized IPC data and return a URL for retrieval.

The uploaded object is not automatically deleted — server operators are responsible for configuring object cleanup via storage lifecycle rules or TTLs.

PARAMETER DESCRIPTION
data

Complete Arrow IPC stream bytes.

TYPE: bytes

schema

The schema of the data being uploaded.

TYPE: Schema

content_encoding

Optional encoding applied to data (e.g. "zstd"). Backends should store this so that fetchers can decompress correctly.

TYPE: str | None DEFAULT: None

RETURNS DESCRIPTION
str

A URL (typically pre-signed) that can be fetched to retrieve

str

the uploaded data.

Source code in vgi_rpc/external.py
def upload(self, data: bytes, schema: pa.Schema, *, content_encoding: str | None = None) -> str:
    """Upload serialized IPC data and return a URL for retrieval.

    The uploaded object is not automatically deleted — server
    operators are responsible for configuring object cleanup via
    storage lifecycle rules or TTLs.

    Args:
        data: Complete Arrow IPC stream bytes.
        schema: The schema of the data being uploaded.
        content_encoding: Optional encoding applied to *data*
            (e.g. ``"zstd"``).  Backends should store this so that
            fetchers can decompress correctly.

    Returns:
        A URL (typically pre-signed) that can be fetched to retrieve
        the uploaded data.

    """
    ...

UploadUrlProvider

Bases: Protocol

Generates pre-signed upload URL pairs, each for one sized upload.

Implementations must be thread-safe — generate_upload_url() may be called concurrently from different server threads.

.. important:: Object lifecycle — vgi-rpc does not manage the lifecycle of uploaded objects. Data uploaded via these URLs persists indefinitely unless the server operator configures cleanup. Use storage-level lifecycle rules (e.g. S3 Lifecycle Policies, GCS Object Lifecycle Management) or bucket-level TTLs to automatically expire and delete stale objects.

generate_upload_url

generate_upload_url(
    size: int, sha256_hex: str
) -> UploadUrl

Generate a pre-signed upload/download URL pair for one upload.

The PUT URL must be signed for exactly size bytes and, where the backend can verify it at storage, for sha256_hex, so storage itself refuses any other body. The uploaded object is not automatically deleted — server operators are responsible for configuring object cleanup via storage lifecycle rules or TTLs.

PARAMETER DESCRIPTION
size

Exact byte length the client will PUT (>= 1).

TYPE: int

sha256_hex

SHA-256 of exactly those bytes, 64 lowercase hex.

TYPE: str

RETURNS DESCRIPTION
UploadUrl

An UploadUrl with PUT and GET URLs for the same object and

UploadUrl

the headers the PUT must carry.

Source code in vgi_rpc/external.py
def generate_upload_url(self, size: int, sha256_hex: str) -> UploadUrl:
    """Generate a pre-signed upload/download URL pair for one upload.

    The PUT URL must be signed for exactly *size* bytes and, where the
    backend can verify it at storage, for *sha256_hex*, so storage
    itself refuses any other body.  **The uploaded object is not
    automatically deleted** — server operators are responsible for
    configuring object cleanup via storage lifecycle rules or TTLs.

    Args:
        size: Exact byte length the client will PUT (``>= 1``).
        sha256_hex: SHA-256 of exactly those bytes, 64 lowercase hex.

    Returns:
        An ``UploadUrl`` with PUT and GET URLs for the same object and
        the headers the PUT must carry.

    """
    ...

UploadUrl dataclass

UploadUrl(
    upload_url: str,
    download_url: str,
    expires_at: datetime,
    headers: Mapping[str, str] = dict(),
)

Pre-signed URL pair for one client-side upload of known size and digest.

S3/GCS pre-signed URLs are signed per HTTP method, so a PUT URL cannot be used for GET — hence two URLs for the same storage object. Since upload-URL protocol v2 the PUT URL is signed for the exact size (and, where the backend can verify one, the SHA-256) of the body, and headers lists what the client must send with it.

ATTRIBUTE DESCRIPTION
upload_url

Pre-signed PUT URL for uploading data.

TYPE: str

download_url

Pre-signed GET URL for downloading the uploaded data.

TYPE: str

expires_at

Expiration time for the pre-signed URLs (UTC).

TYPE: datetime

headers

Headers the client MUST send on the PUT, verbatim (for example x-amz-checksum-sha256). Never Content-Length: the client sends exactly the declared size with a plain Content-Length, which the signature covers. May be empty.

TYPE: Mapping[str, str]

Validation

https_only_validator

https_only_validator(url: str) -> None

Reject URLs that do not use the https scheme.

This is the default url_validator for ExternalLocationConfig. It prevents the client from issuing requests over plain HTTP or other schemes (ftp, file, etc.) when resolving external-location pointers.

PARAMETER DESCRIPTION
url

The URL to validate.

TYPE: str

RAISES DESCRIPTION
ValueError

If the URL scheme is not https.

Source code in vgi_rpc/external.py
def https_only_validator(url: str) -> None:
    """Reject URLs that do not use the ``https`` scheme.

    This is the default ``url_validator`` for ``ExternalLocationConfig``.
    It prevents the client from issuing requests over plain HTTP or other
    schemes (``ftp``, ``file``, etc.) when resolving external-location
    pointers.

    Args:
        url: The URL to validate.

    Raises:
        ValueError: If the URL scheme is not ``https``.

    """
    from urllib.parse import urlparse

    parsed = urlparse(url)
    if parsed.scheme != "https":
        raise ValueError(f"URL scheme '{parsed.scheme}' not allowed (only 'https' is permitted)")

S3 Backend

Requires pip install vgi-rpc[s3].

S3Storage dataclass

S3Storage(
    bucket: str,
    prefix: str = "vgi-rpc/",
    presign_expiry_seconds: int = 3600,
    region_name: str | None = None,
    endpoint_url: str | None = None,
)

S3-backed ExternalStorage using boto3.

.. important:: Object lifecycle — uploaded objects persist indefinitely. Configure an S3 Lifecycle Policy_ on the bucket to expire objects under prefix (default vgi-rpc/) after a suitable retention period. See :mod:vgi_rpc.external for full details and examples.

ATTRIBUTE DESCRIPTION
bucket

S3 bucket name.

TYPE: str

prefix

Key prefix for uploaded objects.

TYPE: str

presign_expiry_seconds

Lifetime of pre-signed GET URLs.

TYPE: int

region_name

AWS region (None uses boto3 default).

TYPE: str | None

endpoint_url

Custom endpoint for S3-compatible services (e.g. MinIO, LocalStack).

TYPE: str | None

.. _S3 Lifecycle Policy: https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html

upload

upload(
    data: bytes,
    schema: Schema,
    *,
    content_encoding: str | None = None
) -> str

Upload IPC data to S3 and return a pre-signed GET URL.

PARAMETER DESCRIPTION
data

Serialized Arrow IPC stream bytes.

TYPE: bytes

schema

Schema of the data (unused but required by protocol).

TYPE: Schema

content_encoding

Optional encoding applied to data (e.g. "zstd").

TYPE: str | None DEFAULT: None

RETURNS DESCRIPTION
str

A pre-signed URL that can be used to download the data.

Source code in vgi_rpc/s3.py
def upload(self, data: bytes, schema: pa.Schema, *, content_encoding: str | None = None) -> str:
    """Upload IPC data to S3 and return a pre-signed GET URL.

    Args:
        data: Serialized Arrow IPC stream bytes.
        schema: Schema of the data (unused but required by protocol).
        content_encoding: Optional encoding applied to *data*
            (e.g. ``"zstd"``).

    Returns:
        A pre-signed URL that can be used to download the data.

    """
    client = self._get_client()

    ext = ".arrow.zst" if content_encoding == "zstd" else ".arrow"
    key = f"{self.prefix}{uuid.uuid4().hex}{ext}"

    put_kwargs: dict[str, str | bytes] = {
        "Bucket": self.bucket,
        "Key": key,
        "Body": data,
        "ContentType": "application/octet-stream",
    }
    if content_encoding is not None:
        put_kwargs["ContentEncoding"] = content_encoding

    t0 = time.monotonic()
    try:
        client.put_object(**put_kwargs)
    except Exception as exc:
        _logger.error(
            "S3 upload failed: bucket=%s key=%s",
            self.bucket,
            key,
            exc_info=True,
            extra={"bucket": self.bucket, "key": key, "error_type": type(exc).__name__},
        )
        raise
    duration_ms = (time.monotonic() - t0) * 1000

    url: str = client.generate_presigned_url(
        "get_object",
        Params={"Bucket": self.bucket, "Key": key},
        ExpiresIn=self.presign_expiry_seconds,
    )
    _logger.debug(
        "S3 upload completed: bucket=%s key=%s (%d bytes, %.1fms)",
        self.bucket,
        key,
        len(data),
        duration_ms,
        extra={"bucket": self.bucket, "key": key, "size_bytes": len(data), "duration_ms": round(duration_ms, 2)},
    )
    return url

generate_upload_url

generate_upload_url(
    size: int, sha256_hex: str
) -> UploadUrl

Generate pre-signed PUT and GET URLs for one sized client upload.

The PUT URL signs Content-Length and x-amz-checksum-sha256 (SigV4), so S3 rejects any body that is not exactly size bytes hashing to sha256_hex (BadDigest). The created S3 object is not automatically deleted. Configure S3 Lifecycle Policies on the bucket to expire objects after a suitable retention period.

PARAMETER DESCRIPTION
size

Exact byte length the client will PUT.

TYPE: int

sha256_hex

SHA-256 of those bytes, 64 lowercase hex.

TYPE: str

RETURNS DESCRIPTION
UploadUrl

An UploadUrl with PUT and GET pre-signed URLs for the

UploadUrl

same S3 object and the x-amz-checksum-sha256 header the

UploadUrl

PUT must carry.

Source code in vgi_rpc/s3.py
def generate_upload_url(self, size: int, sha256_hex: str) -> UploadUrl:
    """Generate pre-signed PUT and GET URLs for one sized client upload.

    The PUT URL signs ``Content-Length`` and ``x-amz-checksum-sha256``
    (SigV4), so S3 rejects any body that is not exactly *size* bytes
    hashing to *sha256_hex* (``BadDigest``).  The created S3 object is
    not automatically deleted.  Configure S3 Lifecycle Policies on the
    bucket to expire objects after a suitable retention period.

    Args:
        size: Exact byte length the client will PUT.
        sha256_hex: SHA-256 of those bytes, 64 lowercase hex.

    Returns:
        An ``UploadUrl`` with PUT and GET pre-signed URLs for the
        same S3 object and the ``x-amz-checksum-sha256`` header the
        PUT must carry.

    """
    checksum = sha256_b64_from_hex(sha256_hex)
    client = self._get_client()
    key = f"{self.prefix}{uuid.uuid4().hex}.arrow"
    params = {"Bucket": self.bucket, "Key": key}
    expires_at = datetime.now(UTC) + timedelta(seconds=self.presign_expiry_seconds)

    put_url: str = client.generate_presigned_url(
        "put_object",
        Params={**params, "ContentLength": size, "ChecksumSHA256": checksum},
        ExpiresIn=self.presign_expiry_seconds,
    )
    get_url: str = client.generate_presigned_url(
        "get_object",
        Params=params,
        ExpiresIn=self.presign_expiry_seconds,
    )

    _logger.debug(
        "S3 upload URL generated: bucket=%s key=%s size=%d",
        self.bucket,
        key,
        size,
        extra={"bucket": self.bucket, "key": key, "size_bytes": size},
    )
    return UploadUrl(
        upload_url=put_url,
        download_url=get_url,
        expires_at=expires_at,
        headers={"x-amz-checksum-sha256": checksum},
    )

GCS Backend

Requires pip install vgi-rpc[gcs].

GCSStorage dataclass

GCSStorage(
    bucket: str,
    prefix: str = "vgi-rpc/",
    presign_expiry_seconds: int = 3600,
    project: str | None = None,
)

GCS-backed ExternalStorage using google-cloud-storage.

.. important:: Object lifecycle — uploaded objects persist indefinitely. Configure Object Lifecycle Management_ on the bucket to delete objects under prefix (default vgi-rpc/) after a suitable retention period. See :mod:vgi_rpc.external for full details and examples.

ATTRIBUTE DESCRIPTION
bucket

GCS bucket name.

TYPE: str

prefix

Key prefix for uploaded objects.

TYPE: str

presign_expiry_seconds

Lifetime of signed GET URLs.

TYPE: int

project

GCS project ID (None uses Application Default Credentials default project).

TYPE: str | None

.. _Object Lifecycle Management: https://cloud.google.com/storage/docs/lifecycle

upload

upload(
    data: bytes,
    schema: Schema,
    *,
    content_encoding: str | None = None
) -> str

Upload IPC data to GCS and return a signed GET URL.

PARAMETER DESCRIPTION
data

Serialized Arrow IPC stream bytes.

TYPE: bytes

schema

Schema of the data (unused but required by protocol).

TYPE: Schema

content_encoding

Optional encoding applied to data (e.g. "zstd").

TYPE: str | None DEFAULT: None

RETURNS DESCRIPTION
str

A signed URL that can be used to download the data.

Source code in vgi_rpc/gcs.py
def upload(self, data: bytes, schema: pa.Schema, *, content_encoding: str | None = None) -> str:
    """Upload IPC data to GCS and return a signed GET URL.

    Args:
        data: Serialized Arrow IPC stream bytes.
        schema: Schema of the data (unused but required by protocol).
        content_encoding: Optional encoding applied to *data*
            (e.g. ``"zstd"``).

    Returns:
        A signed URL that can be used to download the data.

    """
    client = self._get_client()
    bucket = client.bucket(self.bucket)
    ext = ".arrow.zst" if content_encoding == "zstd" else ".arrow"
    blob_name = f"{self.prefix}{uuid.uuid4().hex}{ext}"
    blob = bucket.blob(blob_name)
    if content_encoding is not None:
        blob.content_encoding = content_encoding
    t0 = time.monotonic()
    try:
        blob.upload_from_string(data, content_type="application/octet-stream")
    except Exception as exc:
        _logger.error(
            "GCS upload failed: bucket=%s key=%s",
            self.bucket,
            blob_name,
            exc_info=True,
            extra={"bucket": self.bucket, "key": blob_name, "error_type": type(exc).__name__},
        )
        raise
    duration_ms = (time.monotonic() - t0) * 1000

    url: str = blob.generate_signed_url(
        version="v4",
        expiration=timedelta(seconds=self.presign_expiry_seconds),
        method="GET",
    )
    _logger.debug(
        "GCS upload completed: bucket=%s key=%s (%d bytes, %.1fms)",
        self.bucket,
        blob_name,
        len(data),
        duration_ms,
        extra={
            "bucket": self.bucket,
            "key": blob_name,
            "size_bytes": len(data),
            "duration_ms": round(duration_ms, 2),
        },
    )
    return url

generate_upload_url

generate_upload_url(
    size: int, sha256_hex: str
) -> UploadUrl

Generate signed PUT and GET URLs for one sized client upload.

GCS cannot verify a SHA-256 at storage, so the PUT URL signs the length (x-goog-content-length-range: size,size) and the content type; the server's vgi_rpc.location.sha256 check on the request pointer catches a wrong body. The created GCS object is not automatically deleted. Configure GCS Object Lifecycle Management on the bucket to expire objects after a suitable retention period.

PARAMETER DESCRIPTION
size

Exact byte length the client will PUT.

TYPE: int

sha256_hex

SHA-256 of those bytes, 64 lowercase hex (validated; GCS cannot enforce it).

TYPE: str

RETURNS DESCRIPTION
UploadUrl

An UploadUrl with PUT and GET signed URLs for the same GCS

UploadUrl

object and the signed headers the PUT must carry.

RAISES DESCRIPTION
ValueError

If sha256_hex is not 64 lowercase hex characters.

Source code in vgi_rpc/gcs.py
def generate_upload_url(self, size: int, sha256_hex: str) -> UploadUrl:
    """Generate signed PUT and GET URLs for one sized client upload.

    GCS cannot verify a SHA-256 at storage, so the PUT URL signs the
    length (``x-goog-content-length-range: size,size``) and the content
    type; the server's ``vgi_rpc.location.sha256`` check on the request
    pointer catches a wrong body.  The created GCS object is not
    automatically deleted.  Configure GCS Object Lifecycle Management on
    the bucket to expire objects after a suitable retention period.

    Args:
        size: Exact byte length the client will PUT.
        sha256_hex: SHA-256 of those bytes, 64 lowercase hex (validated;
            GCS cannot enforce it).

    Returns:
        An ``UploadUrl`` with PUT and GET signed URLs for the same GCS
        object and the signed headers the PUT must carry.

    Raises:
        ValueError: If *sha256_hex* is not 64 lowercase hex characters.

    """
    if not is_sha256_hex(sha256_hex):
        raise ValueError("sha256 must be 64 lowercase hex characters")
    client = self._get_client()
    bucket = client.bucket(self.bucket)
    blob_name = f"{self.prefix}{uuid.uuid4().hex}.arrow"
    blob = bucket.blob(blob_name)
    expiration = timedelta(seconds=self.presign_expiry_seconds)
    expires_at = datetime.now(UTC) + expiration
    put_headers = {"x-goog-content-length-range": f"{size},{size}"}

    put_url: str = blob.generate_signed_url(
        version="v4",
        expiration=expiration,
        method="PUT",
        content_type="application/octet-stream",
        headers=put_headers,
    )
    get_url: str = blob.generate_signed_url(
        version="v4",
        expiration=expiration,
        method="GET",
    )

    _logger.debug(
        "GCS upload URL generated: bucket=%s key=%s size=%d",
        self.bucket,
        blob_name,
        size,
        extra={"bucket": self.bucket, "key": blob_name, "size_bytes": size},
    )
    return UploadUrl(
        upload_url=put_url,
        download_url=get_url,
        expires_at=expires_at,
        headers={"Content-Type": "application/octet-stream", **put_headers},
    )

Blob Vendor

Requires pip install vgi-rpc[http,external].

VendorStorage

VendorStorage(
    url: str,
    token: str,
    *,
    purpose: Purpose = "rpc",
    timeout: float = _DEFAULT_TIMEOUT,
    session: Client | None = None,
    max_attempts: int = 3,
    backoff_seconds: float = 0.25,
    info_ttl_seconds: float = _DEFAULT_INFO_TTL,
    pin_hosts: bool = True
)

Bases: _VendorClientBase

ExternalStorage and UploadUrlProvider backed by the blob vendor.

upload() (server-side externalization) asks the vendor for a sized, checksummed grant, PUTs the bytes with exactly the grant's required headers, and returns the grant's download URL. generate_upload_url() (client uploads, upload-URL protocol v2) asks for the same kind of sized grant for the client's declared size and digest.

Thread-safe. Objects are cleaned up by the vendor (registry alarm plus storage lifecycle rules); vgi-rpc never deletes them.

Create a vendor-backed storage.

PARAMETER DESCRIPTION
url

Vendor base URL (https; http only for loopback).

TYPE: str

token

Bearer token.

TYPE: str

purpose

Object purpose; "rpc" (default) gets a download URL and the short lifecycle.

TYPE: Purpose DEFAULT: 'rpc'

timeout

Per-request timeout in seconds.

TYPE: float DEFAULT: _DEFAULT_TIMEOUT

session

Optional httpx2.Client for vendor API requests (its transport only: requests carry exactly the vendor headers, never its default headers, cookies or auth). Storage-host requests always use a separate bare client.

TYPE: Client | None DEFAULT: None

max_attempts

Attempts per request (transport errors, 429 and 5xx are retried with exponential backoff).

TYPE: int DEFAULT: 3

backoff_seconds

Base backoff between attempts.

TYPE: float DEFAULT: 0.25

info_ttl_seconds

How long the /v1/info host pin is cached.

TYPE: float DEFAULT: _DEFAULT_INFO_TTL

pin_hosts

Check vended upload/download URLs against the vendor's hosts before using them.

TYPE: bool DEFAULT: True

Source code in vgi_rpc/vendor.py
def __init__(
    self,
    url: str,
    token: str,
    *,
    purpose: Purpose = "rpc",
    timeout: float = _DEFAULT_TIMEOUT,
    session: httpx2.Client | None = None,
    max_attempts: int = 3,
    backoff_seconds: float = 0.25,
    info_ttl_seconds: float = _DEFAULT_INFO_TTL,
    pin_hosts: bool = True,
) -> None:
    """Create a vendor-backed storage.

    Args:
        url: Vendor base URL (``https``; ``http`` only for loopback).
        token: Bearer token.
        purpose: Object purpose; ``"rpc"`` (default) gets a download URL
            and the short lifecycle.
        timeout: Per-request timeout in seconds.
        session: Optional ``httpx2.Client`` for vendor API requests
            (its transport only: requests carry exactly the vendor
            headers, never its default headers, cookies or auth).
            Storage-host requests always use a separate bare client.
        max_attempts: Attempts per request (transport errors, 429 and
            5xx are retried with exponential backoff).
        backoff_seconds: Base backoff between attempts.
        info_ttl_seconds: How long the ``/v1/info`` host pin is cached.
        pin_hosts: Check vended upload/download URLs against the
            vendor's hosts before using them.

    """
    super().__init__(
        url,
        token,
        timeout=timeout,
        session=session,
        max_attempts=max_attempts,
        backoff_seconds=backoff_seconds,
        info_ttl_seconds=info_ttl_seconds,
    )
    self._purpose: Purpose = purpose
    self._pin = pin_hosts

from_env classmethod

from_env(
    *,
    purpose: Purpose = "rpc",
    env: Mapping[str, str] | None = None
) -> VendorStorage

Build from VGI_BLOB_VENDOR_URL and the vendor token variables.

PARAMETER DESCRIPTION
purpose

Object purpose.

TYPE: Purpose DEFAULT: 'rpc'

env

Environment mapping (defaults to os.environ).

TYPE: Mapping[str, str] | None DEFAULT: None

RETURNS DESCRIPTION
VendorStorage

A configured :class:VendorStorage.

RAISES DESCRIPTION
ValueError

If the URL or token is missing.

Source code in vgi_rpc/vendor.py
@classmethod
def from_env(cls, *, purpose: Purpose = "rpc", env: Mapping[str, str] | None = None) -> VendorStorage:
    """Build from ``VGI_BLOB_VENDOR_URL`` and the vendor token variables.

    Args:
        purpose: Object purpose.
        env: Environment mapping (defaults to ``os.environ``).

    Returns:
        A configured :class:`VendorStorage`.

    Raises:
        ValueError: If the URL or token is missing.

    """
    url, token = _env_url_and_token(_resolve_env(env))
    return cls(url, token, purpose=purpose)

upload

upload(
    data: bytes,
    schema: Schema,
    *,
    content_encoding: str | None = None
) -> str

Upload data via a vendor grant and return its download URL.

PARAMETER DESCRIPTION
data

Bytes to store (already compressed when content_encoding is set).

TYPE: bytes

schema

Arrow schema of the stream (unused).

TYPE: Schema

content_encoding

Encoding applied to data, stored with the object.

TYPE: str | None DEFAULT: None

RETURNS DESCRIPTION
str

The pre-signed download URL.

RAISES DESCRIPTION
VendorError

If the vendor or the signed upload fails.

Source code in vgi_rpc/vendor.py
def upload(self, data: bytes, schema: pa.Schema, *, content_encoding: str | None = None) -> str:
    """Upload *data* via a vendor grant and return its download URL.

    Args:
        data: Bytes to store (already compressed when *content_encoding*
            is set).
        schema: Arrow schema of the stream (unused).
        content_encoding: Encoding applied to *data*, stored with the
            object.

    Returns:
        The pre-signed download URL.

    Raises:
        VendorError: If the vendor or the signed upload fails.

    """
    grant = self._api.put(
        size=len(data),
        sha256_b64=_b64_sha256(data),
        purpose=self._purpose,
        content_encoding=content_encoding,
    )
    if self._pin:
        self._check(grant.upload_url)
    self._api.put_object(grant, data, content_encoding=content_encoding)
    download_url = grant.download_url
    if download_url is None:
        download_url = self._api.get(grant.object_id).download_url
    if self._pin:
        self._check(download_url)
    _logger.debug("Vendored %d bytes as %s", len(data), redact_url(download_url))
    return download_url

generate_upload_url

generate_upload_url(
    size: int, sha256_hex: str
) -> UploadUrl

Ask the vendor for one sized, checksummed PUT/GET pair (purpose=rpc).

Uses the same /v1/blobs/put grant as :meth:upload, so the PUT URL signs Content-Length and x-amz-checksum-sha256 and R2 rejects any other body. The request carries the advisory origin: "client_upload" (:data:CLIENT_UPLOAD_ORIGIN): the vendor ignores it, test fakes use it to tell client uploads from the server's own writes. The grant's required_headers become the URL's headers, minus Content-Length (the client sends the body's own).

PARAMETER DESCRIPTION
size

Exact byte length the client will PUT.

TYPE: int

sha256_hex

SHA-256 of those bytes, 64 lowercase hex.

TYPE: str

RETURNS DESCRIPTION
UploadUrl

The vendored :class:~vgi_rpc.external.UploadUrl.

RAISES DESCRIPTION
VendorError

If the vendor fails or returns no download URL.

Source code in vgi_rpc/vendor.py
def generate_upload_url(self, size: int, sha256_hex: str) -> UploadUrl:
    """Ask the vendor for one sized, checksummed PUT/GET pair (``purpose=rpc``).

    Uses the same ``/v1/blobs/put`` grant as :meth:`upload`, so the PUT
    URL signs ``Content-Length`` and ``x-amz-checksum-sha256`` and R2
    rejects any other body.  The request carries the advisory
    ``origin: "client_upload"`` (:data:`CLIENT_UPLOAD_ORIGIN`): the vendor
    ignores it, test fakes use it to tell client uploads from the
    server's own writes.  The grant's ``required_headers`` become
    the URL's headers, minus ``Content-Length`` (the client sends the
    body's own).

    Args:
        size: Exact byte length the client will PUT.
        sha256_hex: SHA-256 of those bytes, 64 lowercase hex.

    Returns:
        The vendored :class:`~vgi_rpc.external.UploadUrl`.

    Raises:
        VendorError: If the vendor fails or returns no download URL.

    """
    grant = self._api.put(
        size=size, sha256_b64=sha256_b64_from_hex(sha256_hex), purpose="rpc", origin=CLIENT_UPLOAD_ORIGIN
    )
    if grant.download_url is None:
        raise VendorError("Blob vendor put for purpose 'rpc' returned no download_url")
    if self._pin:
        self._check(grant.upload_url)
        self._check(grant.download_url)
    return UploadUrl(
        upload_url=grant.upload_url,
        download_url=grant.download_url,
        expires_at=min(grant.expires_at, grant.download_expires_at or grant.expires_at),
        headers={k: v for k, v in grant.required_headers.items() if k.lower() != "content-length"},
    )

VendorBlobClient

VendorBlobClient(
    url: str,
    token: str,
    *,
    purpose: Purpose = "fs",
    timeout: float = _DEFAULT_TIMEOUT,
    session: Client | None = None,
    max_attempts: int = 3,
    backoff_seconds: float = 0.25,
    info_ttl_seconds: float = _DEFAULT_INFO_TTL,
    fetch_config: FetchConfig | None = None
)

Bases: _VendorClientBase

Raw byte blobs through the vendor, referenced by :class:BlobRef.

For callers that store references themselves (e.g. a key-value store that externalizes large values). Objects use purpose "fs" by default: no download URL at put; get_bytes asks the vendor for a fresh one.

Create a blob client.

PARAMETER DESCRIPTION
url

Vendor base URL (https; http only for loopback).

TYPE: str

token

Bearer token.

TYPE: str

purpose

Object purpose (default "fs").

TYPE: Purpose DEFAULT: 'fs'

timeout

Per-request timeout in seconds.

TYPE: float DEFAULT: _DEFAULT_TIMEOUT

session

Optional httpx2.Client for vendor API requests (transport only; storage requests use a separate bare client).

TYPE: Client | None DEFAULT: None

max_attempts

Attempts per vendor request.

TYPE: int DEFAULT: 3

backoff_seconds

Base backoff between attempts.

TYPE: float DEFAULT: 0.25

info_ttl_seconds

How long the /v1/info host pin is cached.

TYPE: float DEFAULT: _DEFAULT_INFO_TTL

fetch_config

Download settings; max_fetch_bytes caps the size of a blob get_bytes will fetch. A config created here is closed by :meth:close.

TYPE: FetchConfig | None DEFAULT: None

Source code in vgi_rpc/vendor.py
def __init__(
    self,
    url: str,
    token: str,
    *,
    purpose: Purpose = "fs",
    timeout: float = _DEFAULT_TIMEOUT,
    session: httpx2.Client | None = None,
    max_attempts: int = 3,
    backoff_seconds: float = 0.25,
    info_ttl_seconds: float = _DEFAULT_INFO_TTL,
    fetch_config: FetchConfig | None = None,
) -> None:
    """Create a blob client.

    Args:
        url: Vendor base URL (``https``; ``http`` only for loopback).
        token: Bearer token.
        purpose: Object purpose (default ``"fs"``).
        timeout: Per-request timeout in seconds.
        session: Optional ``httpx2.Client`` for vendor API requests
            (transport only; storage requests use a separate bare client).
        max_attempts: Attempts per vendor request.
        backoff_seconds: Base backoff between attempts.
        info_ttl_seconds: How long the ``/v1/info`` host pin is cached.
        fetch_config: Download settings; ``max_fetch_bytes`` caps the
            size of a blob ``get_bytes`` will fetch.  A config created
            here is closed by :meth:`close`.

    """
    super().__init__(
        url,
        token,
        timeout=timeout,
        session=session,
        max_attempts=max_attempts,
        backoff_seconds=backoff_seconds,
        info_ttl_seconds=info_ttl_seconds,
    )
    self._purpose: Purpose = purpose
    self._own_fetch = fetch_config is None
    self._fetch = fetch_config if fetch_config is not None else FetchConfig()

from_env classmethod

from_env(
    *,
    purpose: Purpose = "fs",
    env: Mapping[str, str] | None = None
) -> VendorBlobClient

Build from VGI_BLOB_VENDOR_URL and the vendor token variables.

PARAMETER DESCRIPTION
purpose

Object purpose.

TYPE: Purpose DEFAULT: 'fs'

env

Environment mapping (defaults to os.environ).

TYPE: Mapping[str, str] | None DEFAULT: None

RETURNS DESCRIPTION
VendorBlobClient

A configured :class:VendorBlobClient.

RAISES DESCRIPTION
ValueError

If the URL or token is missing.

Source code in vgi_rpc/vendor.py
@classmethod
def from_env(cls, *, purpose: Purpose = "fs", env: Mapping[str, str] | None = None) -> VendorBlobClient:
    """Build from ``VGI_BLOB_VENDOR_URL`` and the vendor token variables.

    Args:
        purpose: Object purpose.
        env: Environment mapping (defaults to ``os.environ``).

    Returns:
        A configured :class:`VendorBlobClient`.

    Raises:
        ValueError: If the URL or token is missing.

    """
    url, token = _env_url_and_token(_resolve_env(env))
    return cls(url, token, purpose=purpose)

put_bytes

put_bytes(
    data: bytes,
    *,
    scope: BlobScope | None = None,
    idempotency_key: str | None = None
) -> BlobRef

Store data and return its reference.

PARAMETER DESCRIPTION
data

The bytes to store.

TYPE: bytes

scope

The execution the blob belongs to, for delete_scope.

TYPE: BlobScope | None DEFAULT: None

idempotency_key

Makes a retried logical write reuse the same object: the same key with the same size and SHA-256 returns the same object_id; with different bytes the vendor refuses (VendorError.code == "idempotency_conflict").

TYPE: str | None DEFAULT: None

RETURNS DESCRIPTION
A

class:BlobRef naming the object, its size and SHA-256.

TYPE: BlobRef

RAISES DESCRIPTION
VendorError

If the vendor or the signed upload fails.

Source code in vgi_rpc/vendor.py
def put_bytes(self, data: bytes, *, scope: BlobScope | None = None, idempotency_key: str | None = None) -> BlobRef:
    """Store *data* and return its reference.

    Args:
        data: The bytes to store.
        scope: The execution the blob belongs to, for ``delete_scope``.
        idempotency_key: Makes a retried logical write reuse the same
            object: the same key with the same size and SHA-256 returns
            the same ``object_id``; with different bytes the vendor
            refuses (``VendorError.code == "idempotency_conflict"``).

    Returns:
        A :class:`BlobRef` naming the object, its size and SHA-256.

    Raises:
        VendorError: If the vendor or the signed upload fails.

    """
    digest = hashlib.sha256(data).digest()
    grant = self._api.put(
        size=len(data),
        sha256_b64=base64.b64encode(digest).decode("ascii"),
        purpose=self._purpose,
        scope=scope,
        idempotency_key=idempotency_key,
    )
    try:
        self._validator(grant.upload_url)
    except ValueError as exc:
        raise VendorError(f"Blob vendor returned a URL off its pinned hosts: {exc}") from None
    self._api.put_object(grant, data)
    return BlobRef(object_id=grant.object_id, size=len(data), sha256=digest.hex())

get_bytes

get_bytes(ref: BlobRef) -> bytes

Fetch and verify the blob ref names.

The download is pinned to the vendor's storage hosts, capped at the fetch config's max_fetch_bytes, and checked against the size and SHA-256 in ref (not the vendor's own report).

PARAMETER DESCRIPTION
ref

The reference from :meth:put_bytes.

TYPE: BlobRef

RETURNS DESCRIPTION
bytes

The blob bytes.

RAISES DESCRIPTION
VendorError

If the vendor fails, the object is missing, the download fails, or the bytes do not match ref.

Source code in vgi_rpc/vendor.py
def get_bytes(self, ref: BlobRef) -> bytes:
    """Fetch and verify the blob *ref* names.

    The download is pinned to the vendor's storage hosts, capped at the
    fetch config's ``max_fetch_bytes``, and checked against the size and
    SHA-256 in *ref* (not the vendor's own report).

    Args:
        ref: The reference from :meth:`put_bytes`.

    Returns:
        The blob bytes.

    Raises:
        VendorError: If the vendor fails, the object is missing, the
            download fails, or the bytes do not match *ref*.

    """
    if ref.size > self._fetch.max_fetch_bytes:
        raise VendorError(
            f"Blob {ref.object_id} is {ref.size} bytes, over max_fetch_bytes ({self._fetch.max_fetch_bytes})"
        )
    location = self._api.get(ref.object_id)
    if location.content_encoding not in (None, "identity"):
        raise VendorError(f"Blob {ref.object_id} has unsupported content_encoding {location.content_encoding!r}")
    if location.size != ref.size:
        raise VendorError(
            f"Blob {ref.object_id} size mismatch: vendor reports {location.size}, ref has {ref.size}",
            code="size_mismatch",
        )
    try:
        data = fetch_url(location.download_url, self._fetch, url_validator=self._validator)
    except ValueError as exc:
        raise VendorError(f"Blob {ref.object_id} download rejected: {exc}") from None
    except RuntimeError as exc:
        raise VendorError(f"Blob {ref.object_id} download failed: {exc}") from None
    except Exception as exc:
        # aiohttp errors can embed the signed URL; report the type only.
        status = getattr(exc, "status", None)
        raise VendorError(
            f"Blob {ref.object_id} download failed ({type(exc).__name__}"
            f"{f' HTTP {status}' if isinstance(status, int) else ''}): {redact_url(location.download_url)}",
            status if isinstance(status, int) else None,
        ) from None
    if len(data) != ref.size:
        raise VendorError(
            f"Blob {ref.object_id} size mismatch: expected {ref.size}, got {len(data)}", code="size_mismatch"
        )
    actual = hashlib.sha256(data).hexdigest()
    if actual != ref.sha256.lower():
        raise VendorError(
            f"Blob {ref.object_id} SHA-256 mismatch: expected {ref.sha256}, got {actual}", code="sha256_mismatch"
        )
    return data

delete

delete(refs: Iterable[BlobRef | str]) -> int

Delete objects by reference or object id (missing objects are ignored).

PARAMETER DESCRIPTION
refs

References or object ids.

TYPE: Iterable[BlobRef | str]

RETURNS DESCRIPTION
int

How many objects the vendor deleted.

Source code in vgi_rpc/vendor.py
def delete(self, refs: Iterable[BlobRef | str]) -> int:
    """Delete objects by reference or object id (missing objects are ignored).

    Args:
        refs: References or object ids.

    Returns:
        How many objects the vendor deleted.

    """
    ids = [r.object_id if isinstance(r, BlobRef) else r for r in refs]
    step = self._api.MAX_DELETE_IDS
    return sum(self._api.delete(ids[start : start + step]) for start in range(0, len(ids), step))

delete_scope

delete_scope(shard: str, execution_id: str) -> int

Delete every object put under (shard, execution_id).

PARAMETER DESCRIPTION
shard

The storage shard key.

TYPE: str

execution_id

The execution id, encoded as in :class:BlobScope.

TYPE: str

RETURNS DESCRIPTION
int

How many objects the vendor deleted.

Source code in vgi_rpc/vendor.py
def delete_scope(self, shard: str, execution_id: str) -> int:
    """Delete every object put under ``(shard, execution_id)``.

    Args:
        shard: The storage shard key.
        execution_id: The execution id, encoded as in :class:`BlobScope`.

    Returns:
        How many objects the vendor deleted.

    """
    return self._api.delete_scope(BlobScope(shard=shard, execution_id=execution_id))

close

close() -> None

Close the HTTP session and any fetch config this client created.

Source code in vgi_rpc/vendor.py
def close(self) -> None:
    """Close the HTTP session and any fetch config this client created."""
    super().close()
    if self._own_fetch:
        self._fetch.close()

BlobRef dataclass

BlobRef(object_id: str, size: int, sha256: str)

A reference to one vendored blob, small enough to store inline.

ATTRIBUTE DESCRIPTION
object_id

The vendor's object id.

TYPE: str

size

Exact byte length of the blob.

TYPE: int

sha256

Lowercase hex SHA-256 of the blob bytes.

TYPE: str

BlobScope dataclass

BlobScope(shard: str, execution_id: str)

The execution a blob belongs to, for scoped clean-up.

The vendor accepts 1-128 characters of [A-Za-z0-9._:-] for each field, so encode binary ids as hex (base64's +/= are refused).

ATTRIBUTE DESCRIPTION
shard

The storage shard key.

TYPE: str

execution_id

The execution identifier (e.g. hex of the raw id bytes).

TYPE: str

VendorError

VendorError(
    message: str,
    status: int | None = None,
    code: str | None = None,
)

Bases: RuntimeError

A blob-vendor or signed-URL request failed.

Messages never contain the bearer token or a signed query string.

ATTRIBUTE DESCRIPTION
status

The HTTP status of the failing response, or None for a transport failure or a malformed response.

TYPE: int | None

code

The vendor's machine-readable error code (e.g. "not_found", "not_uploaded", "quota_exceeded", "idempotency_conflict"), or the storage's XML error code (e.g. "BadDigest"), when there was one. Checks made by :meth:VendorBlobClient.get_bytes itself use "size_mismatch" and "sha256_mismatch".

TYPE: str | None

Create the error.

PARAMETER DESCRIPTION
message

Diagnostic text (already redacted).

TYPE: str

status

HTTP status, when there was a response.

TYPE: int | None DEFAULT: None

code

Machine-readable error code, when there was one.

TYPE: str | None DEFAULT: None

Source code in vgi_rpc/vendor.py
def __init__(self, message: str, status: int | None = None, code: str | None = None) -> None:
    """Create the error.

    Args:
        message: Diagnostic text (already redacted).
        status: HTTP status, when there was a response.
        code: Machine-readable error code, when there was one.

    """
    super().__init__(message)
    self.status = status
    self.code = code

pinned_hosts_validator

pinned_hosts_validator(
    hosts: Iterable[str],
    *,
    allow_http: bool = False,
    prefixes: Iterable[str] = ()
) -> Callable[[str], None]

Build a url_validator that accepts only URLs on hosts.

A URL passes when its scheme is https (or http with allow_http), it carries no userinfo, and its host and port are exactly one of hosts. An entry is host or host:port ([v6]:port for IPv6); an entry without a port matches the scheme's default port only. Subdomains do not match. With prefixes, the URL (without its query) must also start with one of them, e.g. a region's bucket path.

PARAMETER DESCRIPTION
hosts

Allowed hosts.

TYPE: Iterable[str]

allow_http

Also accept http URLs. Only for loopback test vendors; never enable it for a real deployment.

TYPE: bool DEFAULT: False

prefixes

Optional URL prefixes the URL must start with.

TYPE: Iterable[str] DEFAULT: ()

RETURNS DESCRIPTION
Callable[[str], None]

A callable that raises ValueError for a disallowed URL.

RAISES DESCRIPTION
ValueError

If hosts is empty or an entry will not parse.

Source code in vgi_rpc/vendor.py
def pinned_hosts_validator(
    hosts: Iterable[str], *, allow_http: bool = False, prefixes: Iterable[str] = ()
) -> Callable[[str], None]:
    """Build a ``url_validator`` that accepts only URLs on *hosts*.

    A URL passes when its scheme is ``https`` (or ``http`` with
    *allow_http*), it carries no userinfo, and its host and port are exactly
    one of *hosts*.  An entry is ``host`` or ``host:port`` (``[v6]:port`` for
    IPv6); an entry without a port matches the scheme's default port only.
    Subdomains do not match.  With *prefixes*, the URL (without its query)
    must also start with one of them, e.g. a region's bucket path.

    Args:
        hosts: Allowed hosts.
        allow_http: Also accept ``http`` URLs.  Only for loopback test
            vendors; never enable it for a real deployment.
        prefixes: Optional URL prefixes the URL must start with.

    Returns:
        A callable that raises ``ValueError`` for a disallowed URL.

    Raises:
        ValueError: If *hosts* is empty or an entry will not parse.

    """
    allowed: set[tuple[str, int]] = set()
    allowed_schemes = ("https", "http") if allow_http else ("https",)
    for entry in hosts:
        parts = urlsplit(f"//{entry.strip()}")
        if not parts.hostname or parts.path or parts.query or parts.username is not None:
            raise ValueError(f"Invalid pinned host entry: {entry!r}")
        if parts.port is None:
            for scheme in allowed_schemes:
                allowed.add(_host_key(parts.hostname, None, scheme))
        else:
            allowed.add(_host_key(parts.hostname, parts.port, "https"))
    if not allowed:
        raise ValueError("pinned_hosts_validator needs at least one host")
    frozen = frozenset(allowed)
    url_prefixes = tuple(p for p in prefixes if p)

    def validate(url: str) -> None:
        try:
            parts = urlsplit(url)
            port = parts.port
        except ValueError:
            raise ValueError("URL is malformed") from None
        if parts.scheme not in allowed_schemes:
            raise ValueError(f"URL scheme '{parts.scheme}' not allowed (only {' or '.join(allowed_schemes)})")
        if parts.username is not None or parts.password is not None:
            raise ValueError("URL userinfo is not allowed")
        if not parts.hostname or _host_key(parts.hostname, port, parts.scheme) not in frozen:
            raise ValueError(f"URL host is not a pinned storage host: {redact_url(url)}")
        if url_prefixes:
            bare = f"{parts.scheme}://{parts.netloc}{parts.path}"
            if ".." in parts.path.split("/") or not any(bare.startswith(p) for p in url_prefixes):
                raise ValueError(f"URL is outside the pinned storage prefixes: {redact_url(url)}")

    return validate

vendor_url_validator

vendor_url_validator(
    url: str,
    token: str,
    *,
    ttl_seconds: float = _DEFAULT_INFO_TTL,
    session: Client | None = None
) -> Callable[[str], None]

Build a url_validator pinned to the vendor's storage hosts.

The hosts come from the vendor's GET /v1/info, fetched on first use and cached for ttl_seconds; a failed refresh keeps the last good list. Only https URLs pass (http only when the vendor itself is a loopback http URL, i.e. a local test vendor).

PARAMETER DESCRIPTION
url

Vendor base URL.

TYPE: str

token

Bearer token.

TYPE: str

ttl_seconds

Cache lifetime of the host list.

TYPE: float DEFAULT: _DEFAULT_INFO_TTL

session

Optional httpx2.Client.

TYPE: Client | None DEFAULT: None

RETURNS DESCRIPTION
Callable[[str], None]

A callable raising ValueError for a URL off the vendor's hosts.

Source code in vgi_rpc/vendor.py
def vendor_url_validator(
    url: str,
    token: str,
    *,
    ttl_seconds: float = _DEFAULT_INFO_TTL,
    session: httpx2.Client | None = None,
) -> Callable[[str], None]:
    """Build a ``url_validator`` pinned to the vendor's storage hosts.

    The hosts come from the vendor's ``GET /v1/info``, fetched on first use
    and cached for *ttl_seconds*; a failed refresh keeps the last good list.
    Only ``https`` URLs pass (``http`` only when the vendor itself is a
    loopback ``http`` URL, i.e. a local test vendor).

    Args:
        url: Vendor base URL.
        token: Bearer token.
        ttl_seconds: Cache lifetime of the host list.
        session: Optional ``httpx2.Client``.

    Returns:
        A callable raising ``ValueError`` for a URL off the vendor's hosts.

    """
    api = _VendorApi(url, token, timeout=_DEFAULT_TIMEOUT, session=session, max_attempts=3, backoff_seconds=0.25)
    return _VendorHostsValidator(api, ttl_seconds)

external_config_from_env

external_config_from_env(
    env: Mapping[str, str] | None = None,
) -> ServerExternalConfig | None

Server external-location config from the blob-vendor environment.

Uses :class:VendorStorage for uploads, pins inbound pointer URLs to the vendor's hosts, and reads VGI_RPC_EXTERNALIZE_THRESHOLD_BYTES (default 1 MiB) and VGI_RPC_EXTERNAL_COMPRESSION (default none).

Pass config.storage as upload_url_provider to serve_http / make_wsgi_app to also vend client upload URLs.

PARAMETER DESCRIPTION
env

Environment mapping (defaults to os.environ).

TYPE: Mapping[str, str] | None DEFAULT: None

RETURNS DESCRIPTION
ServerExternalConfig | None

The config, or None when VGI_BLOB_VENDOR_URL is unset.

RAISES DESCRIPTION
ValueError

If the vendor URL is set but the token is missing, or a variable will not parse.

Source code in vgi_rpc/vendor.py
def external_config_from_env(env: Mapping[str, str] | None = None) -> ServerExternalConfig | None:
    """Server external-location config from the blob-vendor environment.

    Uses :class:`VendorStorage` for uploads, pins inbound pointer URLs to the
    vendor's hosts, and reads ``VGI_RPC_EXTERNALIZE_THRESHOLD_BYTES``
    (default 1 MiB) and ``VGI_RPC_EXTERNAL_COMPRESSION`` (default none).

    Pass ``config.storage`` as ``upload_url_provider`` to ``serve_http`` /
    ``make_wsgi_app`` to also vend client upload URLs.

    Args:
        env: Environment mapping (defaults to ``os.environ``).

    Returns:
        The config, or ``None`` when ``VGI_BLOB_VENDOR_URL`` is unset.

    Raises:
        ValueError: If the vendor URL is set but the token is missing, or a
            variable will not parse.

    """
    environ = _resolve_env(env)
    if not environ.get(ENV_URL, "").strip():
        return None
    storage = VendorStorage.from_env(env=environ)
    threshold = _parse_positive_int(environ, ENV_THRESHOLD, _DEFAULT_THRESHOLD)
    assert threshold is not None
    return ServerExternalConfig(
        storage=storage,
        externalize_threshold_bytes=threshold,
        compression=_parse_compression(environ.get(ENV_COMPRESSION, "")),
        url_validator=storage.url_validator,
    )

client_external_config_from_env

client_external_config_from_env(
    env: Mapping[str, str] | None = None,
) -> ClientExternalConfig | None

Client external-location config pinned to the blob vendor's hosts.

For clients that hold a vendor token (e.g. a worker calling another worker): pointer and upload URLs are accepted only on the vendor's storage hosts.

PARAMETER DESCRIPTION
env

Environment mapping (defaults to os.environ).

TYPE: Mapping[str, str] | None DEFAULT: None

RETURNS DESCRIPTION
ClientExternalConfig | None

The config, or None when VGI_BLOB_VENDOR_URL is unset.

RAISES DESCRIPTION
ValueError

If the vendor URL is set but the token is missing.

Source code in vgi_rpc/vendor.py
def client_external_config_from_env(env: Mapping[str, str] | None = None) -> ClientExternalConfig | None:
    """Client external-location config pinned to the blob vendor's hosts.

    For clients that hold a vendor token (e.g. a worker calling another
    worker): pointer and upload URLs are accepted only on the vendor's
    storage hosts.

    Args:
        env: Environment mapping (defaults to ``os.environ``).

    Returns:
        The config, or ``None`` when ``VGI_BLOB_VENDOR_URL`` is unset.

    Raises:
        ValueError: If the vendor URL is set but the token is missing.

    """
    environ = _resolve_env(env)
    if not environ.get(ENV_URL, "").strip():
        return None
    url, token = _env_url_and_token(environ)
    return ClientExternalConfig(url_validator=vendor_url_validator(url, token))

max_upload_bytes_from_env

max_upload_bytes_from_env(
    env: Mapping[str, str] | None = None,
) -> int | None

Read VGI_RPC_MAX_UPLOAD_BYTES (advertised client-upload size).

PARAMETER DESCRIPTION
env

Environment mapping (defaults to os.environ).

TYPE: Mapping[str, str] | None DEFAULT: None

RETURNS DESCRIPTION
int | None

The size, or None when unset.

Source code in vgi_rpc/vendor.py
def max_upload_bytes_from_env(env: Mapping[str, str] | None = None) -> int | None:
    """Read ``VGI_RPC_MAX_UPLOAD_BYTES`` (advertised client-upload size).

    Args:
        env: Environment mapping (defaults to ``os.environ``).

    Returns:
        The size, or ``None`` when unset.

    """
    return _parse_positive_int(_resolve_env(env), ENV_MAX_UPLOAD_BYTES, None)