External Storage¶
When RPC payloads grow large — multi-megabyte query results, bulk data imports, large model artifacts — they can exceed the capacity of the underlying transport. HTTP servers and reverse proxies typically enforce request and response body size limits (e.g., 10 MB). Even pipe-based transports become inefficient when serializing very large batches inline.
External storage solves this by transparently offloading oversized Arrow IPC batches to cloud storage (S3, GCS, or any compatible backend). The batch is replaced with a lightweight pointer batch — a zero-row batch carrying a download URL in its metadata. The receiving side resolves the pointer automatically, fetching the actual data in parallel chunks.
This works in both directions:
- Large outputs — the server externalizes result batches that exceed a size threshold
- Large inputs — the client uploads input data to a pre-signed URL and sends a pointer in the RPC request
From the caller's perspective, nothing changes. The proxy returns the same typed results; the externalization is invisible.
How It Works¶
Large Outputs (Server to Client)¶
When a server method returns a batch that exceeds externalize_threshold_bytes, the framework intercepts the response before writing it to the wire:
- The batch is serialized to Arrow IPC format
- Optionally compressed with zstd
- Uploaded to cloud storage via the configured
ExternalStoragebackend - The storage backend returns a download URL (typically a pre-signed URL with an expiry)
- A pointer batch replaces the original — a zero-row batch with
vgi_rpc.locationmetadata containing the URL - The pointer batch is written to the wire instead of the full data
On the client side, pointer batches are detected and resolved transparently:
- The client reads the pointer batch and extracts the URL from
vgi_rpc.locationmetadata - A HEAD probe determines the content size and whether the server supports range requests
- For large payloads (>64 MB by default), parallel range-request fetching splits the download into chunks with speculative hedging for slow chunks
- For smaller payloads, a single GET request fetches the data
- If the data was compressed, it is decompressed automatically
- The Arrow IPC stream is parsed back into a
RecordBatchand returned to the caller
This works for both unary results and streaming outputs. For streaming, all batches in a single output cycle (including any log batches) are serialized as one IPC stream, uploaded together, and resolved as a unit.
Large Inputs (Client to Server)¶
For the reverse direction — when a client needs to send large data to the server — the framework provides upload URLs. This is particularly important for HTTP transport where request body size limits are common.
- The client serializes the request to Arrow IPC, hashes it, and calls
request_upload_urls()with one(size, sha256)entry per body - Each
UploadUrlcontains anupload_url(for PUT) signed for exactly that size and digest, adownload_url(for GET), and theheadersthe PUT must carry - The client PUTs exactly those bytes to the
upload_urlwith those headers, bypassing the RPC server entirely; storage refuses any other body where the backend can verify it (S3, R2, the blob vendor) - The client sends the RPC request with a pointer batch referencing the
download_urland carryingvgi_rpc.location.sha256 - The server verifies the digest while resolving the pointer, and refuses a request pointer without one
This pattern lets clients send arbitrarily large payloads even when the HTTP server enforces strict request size limits — the large data goes directly to cloud storage, and only a small pointer crosses the RPC boundary.
The upload URL endpoint is enabled by passing upload_url_provider to make_wsgi_app(). The server advertises this capability via the VGI-Upload-URL-Support: 2 response header (upload-URL protocol v2; the retired true was the unsized count protocol, which a v2 client refuses), and clients can discover it with http_capabilities() (upload_url_version == 2). The HTTP client does all of this automatically when a request exceeds the server's max_request_bytes.
Server Configuration¶
S3Storage requires pip install vgi-rpc[s3] (and GCSStorage requires [gcs]); the symbols are only exported from vgi_rpc when their backend dependency is installed.
from vgi_rpc import Compression, ExternalLocationConfig, FetchConfig, RpcServer, S3Storage
storage = S3Storage(bucket="my-bucket", prefix="rpc-data/")
config = ExternalLocationConfig(
storage=storage,
externalize_threshold_bytes=1_048_576, # 1 MiB — batches above this go to S3
compression=Compression(), # zstd level 3 by default
fetch_config=FetchConfig(
chunk_size_bytes=8 * 1024 * 1024, # 8 MiB parallel chunks
max_parallel_requests=8,
),
)
server = RpcServer(MyService, MyServiceImpl(), external_location=config)
The same ExternalLocationConfig is passed to the client so it can resolve pointer batches in responses:
from vgi_rpc import connect
with connect(MyService, ["python", "worker.py"], external_location=config) as proxy:
result = proxy.large_query() # pointer batches resolved transparently
For HTTP transport with upload URL support:
from vgi_rpc import make_wsgi_app
app = make_wsgi_app(
server,
upload_url_provider=storage, # enables __upload_url__ endpoint
max_upload_bytes=100 * 1024 * 1024, # refuse (400) uploads over 100 MiB
)
Client Upload URLs¶
import hashlib
import httpx2
from vgi_rpc import UploadUrl, request_upload_urls
from vgi_rpc.http import upload_put_headers
ipc_data = serialize_large_batch(my_batch)
digest = hashlib.sha256(ipc_data).hexdigest()
# One pre-signed URL pair, signed for exactly these bytes
urls: list[UploadUrl] = request_upload_urls(
base_url="http://localhost:8080",
uploads=[(len(ipc_data), digest)],
)
# Upload directly to storage (bypasses the RPC server), with the vended headers
httpx2.put(urls[0].upload_url, content=ipc_data, headers=upload_put_headers(urls[0]))
# Send the RPC request with a pointer naming urls[0].download_url and
# carrying vgi_rpc.location.sha256 = digest; the server verifies it
Clients can discover whether the server supports upload URLs:
from vgi_rpc import http_capabilities
caps = http_capabilities("http://localhost:8080")
if caps.upload_url_version == 2:
urls = request_upload_urls("http://localhost:8080", uploads=[(len(body), digest)])
Compression¶
When compression is enabled, batches are compressed with zstd before upload and decompressed transparently on fetch:
from vgi_rpc import Compression
# Default: zstd level 3
config = ExternalLocationConfig(
storage=storage,
compression=Compression(), # algorithm="zstd", level=3
)
# Custom compression level (higher = smaller but slower)
config = ExternalLocationConfig(
storage=storage,
compression=Compression(level=9),
)
# gzip — stdlib zlib, the right choice when consumers can't do zstd
config = ExternalLocationConfig(
storage=storage,
compression=Compression(algorithm="gzip"), # level defaults to 6
)
# Disable compression
config = ExternalLocationConfig(
storage=storage,
compression=None,
)
Compression(algorithm=...) accepts "zstd" (default; needs zstandard on both writer and reader) or "gzip" (stdlib zlib, no extra dependency). The storage backend stores the Content-Encoding header alongside the object; on fetch the Content-Encoding: zstd / gzip header triggers automatic decompression.
zstd requires pip install vgi-rpc[external] (installs zstandard); gzip works with no extra dependency.
Parallel Fetching¶
For large externalized batches, the client fetches data in parallel chunks using range requests. This significantly reduces download time for multi-megabyte payloads:
- A HEAD probe determines the total size and whether the server supports
Accept-Ranges: bytes - If the payload exceeds
parallel_threshold_bytes(default 64 MB) and ranges are supported, it is split intochunk_size_byteschunks (default 8 MB) fetched concurrently - Speculative hedging: if a chunk takes longer than
speculative_retry_multipliertimes the median chunk time, a hedge request is launched in parallel — the first response wins - For smaller payloads or servers without range support, a single GET request is used
from vgi_rpc import FetchConfig
fetch_config = FetchConfig(
parallel_threshold_bytes=64 * 1024 * 1024, # 64 MiB
chunk_size_bytes=8 * 1024 * 1024, # 8 MiB chunks
max_parallel_requests=8, # concurrent fetches
timeout_seconds=60.0, # overall deadline
max_fetch_bytes=256 * 1024 * 1024, # 256 MiB hard cap
speculative_retry_multiplier=2.0, # hedge at 2x median
max_speculative_hedges=4, # max hedge requests
)
Pre-published References¶
Per-call externalization re-serializes, re-compresses and re-uploads a result on
every call. When a large result rarely changes — a worker's whole catalog, say —
publish it once with publish_external(),
keep the returned ExternalRef, and return that
ref from the unary method on later calls. The server writes the pointer batch
directly: no serialization, compression or upload happens during the call.
import threading
from typing import Protocol
import pyarrow as pa
from vgi_rpc import Compression, ExternalRef, ExternalStorage, publish_external, rpc_methods
class CatalogService(Protocol):
def catalog(self) -> str: ...
class CatalogServiceImpl:
def __init__(self, storage: ExternalStorage, compression: Compression | None) -> None:
self._storage, self._compression = storage, compression
self._ref: ExternalRef | None = None
self._lock = threading.Lock()
def catalog(self) -> str | ExternalRef:
with self._lock:
if self._ref is None: # publish once per process / catalog version
schema = rpc_methods(CatalogService)["catalog"].result_schema
batch = pa.RecordBatch.from_pydict({"result": [build_catalog()]}, schema=schema)
self._ref = publish_external(batch, self._storage, self._compression)
return self._ref
A unary method may return an ExternalRef instead of its declared value. The
return annotation may say so (-> str | ExternalRef); ExternalRef is stripped
when the result schema is derived, so the wire contract (and protocol hash) is
that of str. Returning a ref:
- always answers with a pointer — whether or not the server has external storage
configured, and regardless of
externalize_threshold_bytes(a ref is never inlined, nor routed through shared memory); - skips building and validating the result value;
- is not counted toward
max_externalized_response_bytes(nothing is uploaded during the call); the tiny pointer still goes through the wire-body budget; - works on every transport (pipe, subprocess, Unix, TCP, HTTP). Stream methods are not supported.
publish_external(batch, storage, compression=None, *, include_sha256=True)
serializes the 1-row result batch exactly as the per-call externalizer would,
hashes the raw IPC bytes, compresses if asked, and calls storage.upload() once.
With include_sha256=False the ref carries no digest, so clients skip the
content check — for an object rewritten in place, or one too large to hash.
Clients need no change: a ref's pointer is indistinguishable from any other.
The caller owns the ref and the object's lifecycle:
- A long-lived ref must not point at an object under the short-TTL lifecycle rule used for per-call uploads (see Object Lifecycle) — publish under a different prefix.
- A pre-signed URL expires; re-sign or rebuild the ref before it does.
- Only return a ref to callers who are all entitled to the same content.
Object Lifecycle¶
Uploaded objects persist indefinitely — vgi-rpc does not delete them. Configure storage-level cleanup policies to auto-expire old data.
S3 lifecycle rule:
aws s3api put-bucket-lifecycle-configuration \
--bucket MY_BUCKET \
--lifecycle-configuration '{
"Rules": [{
"ID": "expire-vgi-rpc",
"Filter": {"Prefix": "rpc-data/"},
"Status": "Enabled",
"Expiration": {"Days": 1}
}]
}'
GCS lifecycle rule:
gsutil lifecycle set <(cat <<EOF
{"rule": [{"action": {"type": "Delete"},
"condition": {"age": 1, "matchesPrefix": ["rpc-data/"]}}]}
EOF
) gs://MY_BUCKET
Blob Vendor¶
The blob vendor is a small HTTP service that holds the object-storage
credentials so workers never do. A worker asks it for a pre-signed URL for one
object of a declared size and SHA-256, uploads straight to storage, and hands on
a download URL. The storage (Cloudflare R2) enforces the signed
Content-Length and x-amz-checksum-sha256, so whatever sits at the object's
key is exactly the declared bytes. Buckets are per region (EU / US
jurisdictions, or a location hint), chosen by the vendor from the caller's key,
never from the request.
VendorStorage implements both ExternalStorage (server-side externalization)
and UploadUrlProvider (client upload URLs):
from vgi_rpc.vendor import VendorStorage, external_config_from_env
storage = VendorStorage("https://blobs.example.com", token)
# Or, from the environment (returns None when VGI_BLOB_VENDOR_URL is unset):
config = external_config_from_env()
server = RpcServer(MyService, MyServiceImpl(), external_location=config)
app = make_wsgi_app(
server,
max_request_bytes=8 * 1024 * 1024,
upload_url_provider=config.storage if config else None,
)
serve_http() takes the same upload_url_provider / max_upload_bytes
arguments, and run_server() applies the environment automatically under
--http.
For raw bytes that the caller references itself (e.g. large values in a
key-value store), VendorBlobClient stores and verifies blobs by BlobRef:
from vgi_rpc.vendor import BlobScope, VendorBlobClient
with VendorBlobClient.from_env() as blobs:
# Scope fields: 1-128 chars of [A-Za-z0-9._:-], so hex-encode binary ids.
ref = blobs.put_bytes(value, scope=BlobScope(shard="s1", execution_id=exec_id_hex), idempotency_key="attempt-7:0")
assert blobs.get_bytes(ref) == value # size + SHA-256 checked against ref
blobs.delete_scope("s1", exec_id_hex)
Environment variables¶
| Variable | Meaning |
|---|---|
VGI_BLOB_VENDOR_URL |
Vendor base URL. Unset: the *_from_env configs return None. |
VGI_BLOB_VENDOR_TOKEN |
Bearer token; falls back to VGI_CF_DO_TOKEN. |
VGI_RPC_EXTERNALIZE_THRESHOLD_BYTES |
Result size above which the server externalizes (default 1 MiB). |
VGI_RPC_EXTERNAL_COMPRESSION |
none (default), zstd or gzip, optionally :<level>. |
VGI_RPC_MAX_UPLOAD_BYTES |
Upload size advertised to clients (VGI-Max-Upload-Bytes). |
VGI_RPC_MAX_REQUEST_BYTES |
Request-body cap for run_server --http (--max-request-bytes); clients send larger requests through vendor upload URLs. Without a cap, clients never upload. |
Host pinning¶
Configs built from the environment do not use https_only_validator, which
accepts any host. They pin pointer and upload URLs to the vendor's storage
hosts and URL prefixes (the region's bucket), read from its GET /v1/info and
cached for ten minutes; a failed
refresh keeps the last good list, and with no list at all every URL is
rejected. VendorStorage also checks the URLs the vendor itself returns before
using them. Build a pin by hand with pinned_hosts_validator:
from vgi_rpc.vendor import pinned_hosts_validator
validate = pinned_hosts_validator(["acct.eu.r2.cloudflarestorage.com"])
An entry matches only that exact host (no subdomains) on the default https
port, or the port it names. Only https passes; plain http is accepted only
when the vendor URL itself is a loopback http URL (a local test vendor).
Retries and secrets¶
Vendor and signed-URL requests retry transport errors, 429 and 5xx with
exponential backoff (3 attempts by default); 4xx responses are not retried.
Failures raise VendorError with the HTTP status and the vendor's code
(not_found, not_uploaded, quota_exceeded, object_too_large,
idempotency_conflict, ...) or the storage's (BadDigest). Pass
idempotency_key to put_bytes so a retried logical write reuses one object.
The bearer token goes only to the vendor, never to a signed URL. Every request
to a storage host (the vendor clients' PUTs, and the HTTP client's upload of a
large request) goes on a bare, single-purpose client: never the caller's
httpx2.Client, so its default headers, cookies, auth flow and event hooks
cannot reach the storage host. A session= passed to the vendor clients
supplies only the transport for vendor API calls.
vgi-rpc's own logs and errors show URLs through redact_url (scheme, host, port
and path only). httpx2 logs every request with its full URL at INFO on the
global httpx2 logger, and for a signed URL the query string is the
credential, so the vendor clients and the upload path install a filter on that
logger (vgi_rpc.external_fetch.redact_httpx_logging(), idempotent) that
redacts the URLs in those lines. The lines are still logged at the same level;
call the helper yourself to cover other httpx2 use in the process.
Testing¶
vgi_rpc.conformance.fake_vendor is an in-process fake of the vendor API and
of R2's signed-URL enforcement (declared size and checksum, method-bound
signatures, range GETs), with fault injection:
from vgi_rpc.conformance.fake_vendor import serve_fake_vendor
with serve_fake_vendor({"tok": "tenant-a"}) as fake:
storage = VendorStorage(fake.url, "tok")
fake.vendor.fail_next("/v1/blobs/put", status=503)
The same fake runs standalone for test lanes in any language. It prints
PORT:<n> and then URL:<base>, and needs only vgi-rpc's base dependencies:
--test-routes adds unauthenticated routes. Each answers GET with a JSON
body padded to 1,024 bytes, and a HEAD has no side effects:
GET /__stats__: call and byte counters (puts,gets,deletes,delete_scopes,upload_urls,storage_puts,storage_gets,objects,bytes,bytes_uploaded,bytes_downloaded), signed-PUT refusals (upload_refusals,upload_bad_digest,upload_incomplete_body,upload_signature_mismatch,uploads_corrupted), and counters by purpose:puts_rpc,puts_fsandputs_uploadsplit the/v1/blobs/putgrants issued into server rpc writes (results), function storage, and grants minted for a client upload;client_uploads,client_upload_bytesandclient_upload_refusalscount the signed PUTs to client-upload objects. A server marks a client-upload grant with the advisoryorigin: "client_upload"field, whichVendorStorage.generate_upload_urlsends and the real vendor ignores.GET /__admin__/fault?purpose=fs|rpc&action=corrupt|missing&count=N: the nextNobjects of that purpose that are downloaded are damaged for every laterGET.corruptkeeps the length and changes every byte;missinganswers 404.GET /__admin__/fault?purpose=upload&action=corrupt&count=N: the nextNsigned PUTs of client uploads are damaged in transit, so the store refuses them with400 BadDigest. Server writes are never touched.GET /__admin__/reset: clears all faults.
The standalone fake logs to stderr which stop signal it received, the sending pid and its command line (Linux), and its final counters.
In-process, the same faults are available as
fake.vendor.set_download_fault(...) and reset_faults(), and the counters as
test_stats().
URL Validation¶
By default, external location URLs are validated with https_only_validator, which rejects non-HTTPS URLs. This prevents pointer batches from being used to probe internal networks. You can provide a custom validator via ExternalLocationConfig.url_validator; for the blob vendor use the host-pinned validators described in Host pinning.
Client uploads to a vended URL are sent with Content-Type: application/octet-stream, the type pre-signed URLs that sign the content type (such as GCSStorage.generate_upload_url) expect.
API Reference¶
Configuration¶
Compression
dataclass
¶
Compression settings for externalized data.
| ATTRIBUTE | DESCRIPTION |
|---|---|
algorithm |
Compression algorithm.
TYPE:
|
level |
Compression level. Codec-specific — 1-22 for zstd
(default 3), 1-9 for gzip (default 6 when
TYPE:
|
FetchConfig
dataclass
¶
FetchConfig(
parallel_threshold_bytes: int = 64 * 1024 * 1024,
chunk_size_bytes: int = 8 * 1024 * 1024,
max_parallel_requests: int = 8,
timeout_seconds: float = 60.0,
max_fetch_bytes: int = 256 * 1024 * 1024,
max_decompressed_bytes: int | None = None,
max_redirects: int = 5,
speculative_retry_multiplier: float = 2.0,
max_speculative_hedges: int = 4,
)
Configuration for parallel range-request fetching.
Maintains a persistent aiohttp.ClientSession backed by a daemon
thread. Use as a context manager or call close() to release
resources.
| ATTRIBUTE | DESCRIPTION |
|---|---|
parallel_threshold_bytes |
Below this size, use a single GET.
TYPE:
|
chunk_size_bytes |
Size of each Range request chunk.
TYPE:
|
max_parallel_requests |
Semaphore limit on concurrent requests.
TYPE:
|
timeout_seconds |
Overall deadline for the fetch.
TYPE:
|
max_fetch_bytes |
Hard cap on encoded/on-wire download size.
TYPE:
|
max_decompressed_bytes |
Hard cap after content decoding.
TYPE:
|
max_redirects |
Maximum redirects followed by each probe or data
request. Every target is passed through the caller's URL validator
before it is requested. Set to
TYPE:
|
speculative_retry_multiplier |
Launch a hedge request for chunks
taking longer than
TYPE:
|
max_speculative_hedges |
Maximum number of hedge requests per
fetch. Prevents runaway request amplification under
adversarial or slow-backend conditions.
TYPE:
|
close
¶
Close the pooled session, stop the event loop, and join the thread.
Source code in vgi_rpc/external_fetch.py
__del__
¶
__enter__
¶
__enter__() -> FetchConfig
__exit__
¶
__exit__(
exc_type: type[BaseException] | None,
exc_val: BaseException | None,
exc_tb: TracebackType | None,
) -> None
Exit context manager, closing the pool.
Pre-published References¶
ExternalRef
dataclass
¶
A reference to an already-published unary result.
A unary method may return an ExternalRef in place of its declared
result value. The server then answers with the ExternalLocation
pointer batch for url directly — no result serialization,
compression, or upload happens during the call, and the ref is used
whether or not the server has external storage configured and
regardless of externalize_threshold_bytes. Clients resolve it
like any other pointer, so they need no change.
Build one with :func:publish_external (or by hand for an object
published out of band). The object at url must be an Arrow IPC
stream (optionally Content-Encoding-compressed) whose schema is the
method's result schema and which holds exactly one 1-row data batch.
The caller owns caching the ref and the object's lifecycle: a long-lived ref must not point at an object under the short-TTL lifecycle rule used for per-call uploads, and a pre-signed URL expires — re-sign or rebuild the ref before then. Only return a ref to callers who are all entitled to the same content.
| ATTRIBUTE | DESCRIPTION |
|---|---|
url |
Where the published IPC stream lives.
TYPE:
|
sha256 |
Lowercase hex SHA-256 of the raw (pre-compression) IPC
stream bytes, sent as
TYPE:
|
__post_init__
¶
Validate the URL and digest shape.
| RAISES | DESCRIPTION |
|---|---|
ValueError
|
If |
Source code in vgi_rpc/external.py
pointer_batch
¶
Build the zero-row pointer batch announcing this ref.
| PARAMETER | DESCRIPTION |
|---|---|
schema
|
The method's result schema.
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
RecordBatch
|
|
KeyValueMetadata
|
func: |
Source code in vgi_rpc/external.py
publish_external
¶
publish_external(
batch: RecordBatch,
storage: ExternalStorage,
compression: Compression | None = None,
*,
include_sha256: bool = True
) -> ExternalRef
Publish a unary result batch once and return a reusable reference.
Serializes batch exactly as the per-call externalizer does (an IPC
stream of the schema plus this one batch), hashes the raw bytes,
compresses when compression is given, and calls storage.upload
once. Cache the returned :class:ExternalRef and return it from the
unary method on later calls; the server writes the pointer directly.
Build batch against the method's result schema, e.g.::
schema = rpc_methods(MyService)["catalog"].result_schema
ref = publish_external(pa.RecordBatch.from_pydict({"result": [value]}, schema=schema), storage)
| PARAMETER | DESCRIPTION |
|---|---|
batch
|
The 1-row result batch (single
TYPE:
|
storage
|
Storage backend to upload to.
TYPE:
|
compression
|
Optional compression applied before upload (pass the
server's
TYPE:
|
include_sha256
|
When
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
An
|
class:
TYPE:
|
| RAISES | DESCRIPTION |
|---|---|
ValueError
|
If batch does not have exactly one row. |
Source code in vgi_rpc/external.py
Storage Protocol¶
ExternalStorage
¶
Bases: Protocol
Pluggable storage interface for externalizing large batches.
Implementations must be thread-safe — upload() may be called
concurrently from different server threads.
.. important:: Object lifecycle — vgi-rpc does not manage the
lifecycle of uploaded objects. Data uploaded via upload()
or generate_upload_url() persists indefinitely unless the
server operator configures cleanup. Use storage-level lifecycle
rules (e.g. S3 Lifecycle Policies, GCS Object Lifecycle
Management) or bucket-level TTLs to automatically expire and
delete stale objects.
upload
¶
Upload serialized IPC data and return a URL for retrieval.
The uploaded object is not automatically deleted — server operators are responsible for configuring object cleanup via storage lifecycle rules or TTLs.
| PARAMETER | DESCRIPTION |
|---|---|
data
|
Complete Arrow IPC stream bytes.
TYPE:
|
schema
|
The schema of the data being uploaded.
TYPE:
|
content_encoding
|
Optional encoding applied to data
(e.g.
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
str
|
A URL (typically pre-signed) that can be fetched to retrieve |
str
|
the uploaded data. |
Source code in vgi_rpc/external.py
UploadUrlProvider
¶
Bases: Protocol
Generates pre-signed upload URL pairs, each for one sized upload.
Implementations must be thread-safe — generate_upload_url()
may be called concurrently from different server threads.
.. important:: Object lifecycle — vgi-rpc does not manage the lifecycle of uploaded objects. Data uploaded via these URLs persists indefinitely unless the server operator configures cleanup. Use storage-level lifecycle rules (e.g. S3 Lifecycle Policies, GCS Object Lifecycle Management) or bucket-level TTLs to automatically expire and delete stale objects.
generate_upload_url
¶
generate_upload_url(
size: int, sha256_hex: str
) -> UploadUrl
Generate a pre-signed upload/download URL pair for one upload.
The PUT URL must be signed for exactly size bytes and, where the backend can verify it at storage, for sha256_hex, so storage itself refuses any other body. The uploaded object is not automatically deleted — server operators are responsible for configuring object cleanup via storage lifecycle rules or TTLs.
| PARAMETER | DESCRIPTION |
|---|---|
size
|
Exact byte length the client will PUT (
TYPE:
|
sha256_hex
|
SHA-256 of exactly those bytes, 64 lowercase hex.
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
UploadUrl
|
An |
UploadUrl
|
the headers the PUT must carry. |
Source code in vgi_rpc/external.py
UploadUrl
dataclass
¶
UploadUrl(
upload_url: str,
download_url: str,
expires_at: datetime,
headers: Mapping[str, str] = dict(),
)
Pre-signed URL pair for one client-side upload of known size and digest.
S3/GCS pre-signed URLs are signed per HTTP method, so a PUT URL cannot be used for GET — hence two URLs for the same storage object. Since upload-URL protocol v2 the PUT URL is signed for the exact size (and, where the backend can verify one, the SHA-256) of the body, and headers lists what the client must send with it.
| ATTRIBUTE | DESCRIPTION |
|---|---|
upload_url |
Pre-signed PUT URL for uploading data.
TYPE:
|
download_url |
Pre-signed GET URL for downloading the uploaded data.
TYPE:
|
expires_at |
Expiration time for the pre-signed URLs (UTC).
TYPE:
|
headers |
Headers the client MUST send on the PUT, verbatim (for
example
TYPE:
|
Validation¶
https_only_validator
¶
Reject URLs that do not use the https scheme.
This is the default url_validator for ExternalLocationConfig.
It prevents the client from issuing requests over plain HTTP or other
schemes (ftp, file, etc.) when resolving external-location
pointers.
| PARAMETER | DESCRIPTION |
|---|---|
url
|
The URL to validate.
TYPE:
|
| RAISES | DESCRIPTION |
|---|---|
ValueError
|
If the URL scheme is not |
Source code in vgi_rpc/external.py
S3 Backend¶
Requires pip install vgi-rpc[s3].
S3Storage
dataclass
¶
S3Storage(
bucket: str,
prefix: str = "vgi-rpc/",
presign_expiry_seconds: int = 3600,
region_name: str | None = None,
endpoint_url: str | None = None,
)
S3-backed ExternalStorage using boto3.
.. important:: Object lifecycle — uploaded objects persist
indefinitely. Configure an S3 Lifecycle Policy_ on the
bucket to expire objects under prefix (default
vgi-rpc/) after a suitable retention period.
See :mod:vgi_rpc.external for full details and examples.
| ATTRIBUTE | DESCRIPTION |
|---|---|
bucket |
S3 bucket name.
TYPE:
|
prefix |
Key prefix for uploaded objects.
TYPE:
|
presign_expiry_seconds |
Lifetime of pre-signed GET URLs.
TYPE:
|
region_name |
AWS region (
TYPE:
|
endpoint_url |
Custom endpoint for S3-compatible services (e.g. MinIO, LocalStack).
TYPE:
|
.. _S3 Lifecycle Policy: https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html
upload
¶
Upload IPC data to S3 and return a pre-signed GET URL.
| PARAMETER | DESCRIPTION |
|---|---|
data
|
Serialized Arrow IPC stream bytes.
TYPE:
|
schema
|
Schema of the data (unused but required by protocol).
TYPE:
|
content_encoding
|
Optional encoding applied to data
(e.g.
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
str
|
A pre-signed URL that can be used to download the data. |
Source code in vgi_rpc/s3.py
generate_upload_url
¶
generate_upload_url(
size: int, sha256_hex: str
) -> UploadUrl
Generate pre-signed PUT and GET URLs for one sized client upload.
The PUT URL signs Content-Length and x-amz-checksum-sha256
(SigV4), so S3 rejects any body that is not exactly size bytes
hashing to sha256_hex (BadDigest). The created S3 object is
not automatically deleted. Configure S3 Lifecycle Policies on the
bucket to expire objects after a suitable retention period.
| PARAMETER | DESCRIPTION |
|---|---|
size
|
Exact byte length the client will PUT.
TYPE:
|
sha256_hex
|
SHA-256 of those bytes, 64 lowercase hex.
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
UploadUrl
|
An |
UploadUrl
|
same S3 object and the |
UploadUrl
|
PUT must carry. |
Source code in vgi_rpc/s3.py
GCS Backend¶
Requires pip install vgi-rpc[gcs].
GCSStorage
dataclass
¶
GCSStorage(
bucket: str,
prefix: str = "vgi-rpc/",
presign_expiry_seconds: int = 3600,
project: str | None = None,
)
GCS-backed ExternalStorage using google-cloud-storage.
.. important:: Object lifecycle — uploaded objects persist
indefinitely. Configure Object Lifecycle Management_ on the
bucket to delete objects under prefix (default
vgi-rpc/) after a suitable retention period.
See :mod:vgi_rpc.external for full details and examples.
| ATTRIBUTE | DESCRIPTION |
|---|---|
bucket |
GCS bucket name.
TYPE:
|
prefix |
Key prefix for uploaded objects.
TYPE:
|
presign_expiry_seconds |
Lifetime of signed GET URLs.
TYPE:
|
project |
GCS project ID (
TYPE:
|
.. _Object Lifecycle Management: https://cloud.google.com/storage/docs/lifecycle
upload
¶
Upload IPC data to GCS and return a signed GET URL.
| PARAMETER | DESCRIPTION |
|---|---|
data
|
Serialized Arrow IPC stream bytes.
TYPE:
|
schema
|
Schema of the data (unused but required by protocol).
TYPE:
|
content_encoding
|
Optional encoding applied to data
(e.g.
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
str
|
A signed URL that can be used to download the data. |
Source code in vgi_rpc/gcs.py
generate_upload_url
¶
generate_upload_url(
size: int, sha256_hex: str
) -> UploadUrl
Generate signed PUT and GET URLs for one sized client upload.
GCS cannot verify a SHA-256 at storage, so the PUT URL signs the
length (x-goog-content-length-range: size,size) and the content
type; the server's vgi_rpc.location.sha256 check on the request
pointer catches a wrong body. The created GCS object is not
automatically deleted. Configure GCS Object Lifecycle Management on
the bucket to expire objects after a suitable retention period.
| PARAMETER | DESCRIPTION |
|---|---|
size
|
Exact byte length the client will PUT.
TYPE:
|
sha256_hex
|
SHA-256 of those bytes, 64 lowercase hex (validated; GCS cannot enforce it).
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
UploadUrl
|
An |
UploadUrl
|
object and the signed headers the PUT must carry. |
| RAISES | DESCRIPTION |
|---|---|
ValueError
|
If sha256_hex is not 64 lowercase hex characters. |
Source code in vgi_rpc/gcs.py
Blob Vendor¶
Requires pip install vgi-rpc[http,external].
VendorStorage
¶
VendorStorage(
url: str,
token: str,
*,
purpose: Purpose = "rpc",
timeout: float = _DEFAULT_TIMEOUT,
session: Client | None = None,
max_attempts: int = 3,
backoff_seconds: float = 0.25,
info_ttl_seconds: float = _DEFAULT_INFO_TTL,
pin_hosts: bool = True
)
Bases: _VendorClientBase
ExternalStorage and UploadUrlProvider backed by the blob vendor.
upload() (server-side externalization) asks the vendor for a sized,
checksummed grant, PUTs the bytes with exactly the grant's required
headers, and returns the grant's download URL. generate_upload_url()
(client uploads, upload-URL protocol v2) asks for the same kind of sized
grant for the client's declared size and digest.
Thread-safe. Objects are cleaned up by the vendor (registry alarm plus storage lifecycle rules); vgi-rpc never deletes them.
Create a vendor-backed storage.
| PARAMETER | DESCRIPTION |
|---|---|
url
|
Vendor base URL (
TYPE:
|
token
|
Bearer token.
TYPE:
|
purpose
|
Object purpose;
TYPE:
|
timeout
|
Per-request timeout in seconds.
TYPE:
|
session
|
Optional
TYPE:
|
max_attempts
|
Attempts per request (transport errors, 429 and 5xx are retried with exponential backoff).
TYPE:
|
backoff_seconds
|
Base backoff between attempts.
TYPE:
|
info_ttl_seconds
|
How long the
TYPE:
|
pin_hosts
|
Check vended upload/download URLs against the vendor's hosts before using them.
TYPE:
|
Source code in vgi_rpc/vendor.py
from_env
classmethod
¶
from_env(
*,
purpose: Purpose = "rpc",
env: Mapping[str, str] | None = None
) -> VendorStorage
Build from VGI_BLOB_VENDOR_URL and the vendor token variables.
| PARAMETER | DESCRIPTION |
|---|---|
purpose
|
Object purpose.
TYPE:
|
env
|
Environment mapping (defaults to
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
VendorStorage
|
A configured :class: |
| RAISES | DESCRIPTION |
|---|---|
ValueError
|
If the URL or token is missing. |
Source code in vgi_rpc/vendor.py
upload
¶
Upload data via a vendor grant and return its download URL.
| PARAMETER | DESCRIPTION |
|---|---|
data
|
Bytes to store (already compressed when content_encoding is set).
TYPE:
|
schema
|
Arrow schema of the stream (unused).
TYPE:
|
content_encoding
|
Encoding applied to data, stored with the object.
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
str
|
The pre-signed download URL. |
| RAISES | DESCRIPTION |
|---|---|
VendorError
|
If the vendor or the signed upload fails. |
Source code in vgi_rpc/vendor.py
generate_upload_url
¶
generate_upload_url(
size: int, sha256_hex: str
) -> UploadUrl
Ask the vendor for one sized, checksummed PUT/GET pair (purpose=rpc).
Uses the same /v1/blobs/put grant as :meth:upload, so the PUT
URL signs Content-Length and x-amz-checksum-sha256 and R2
rejects any other body. The request carries the advisory
origin: "client_upload" (:data:CLIENT_UPLOAD_ORIGIN): the vendor
ignores it, test fakes use it to tell client uploads from the
server's own writes. The grant's required_headers become
the URL's headers, minus Content-Length (the client sends the
body's own).
| PARAMETER | DESCRIPTION |
|---|---|
size
|
Exact byte length the client will PUT.
TYPE:
|
sha256_hex
|
SHA-256 of those bytes, 64 lowercase hex.
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
UploadUrl
|
The vendored :class: |
| RAISES | DESCRIPTION |
|---|---|
VendorError
|
If the vendor fails or returns no download URL. |
Source code in vgi_rpc/vendor.py
VendorBlobClient
¶
VendorBlobClient(
url: str,
token: str,
*,
purpose: Purpose = "fs",
timeout: float = _DEFAULT_TIMEOUT,
session: Client | None = None,
max_attempts: int = 3,
backoff_seconds: float = 0.25,
info_ttl_seconds: float = _DEFAULT_INFO_TTL,
fetch_config: FetchConfig | None = None
)
Bases: _VendorClientBase
Raw byte blobs through the vendor, referenced by :class:BlobRef.
For callers that store references themselves (e.g. a key-value store
that externalizes large values). Objects use purpose "fs" by
default: no download URL at put; get_bytes asks the vendor for a
fresh one.
Create a blob client.
| PARAMETER | DESCRIPTION |
|---|---|
url
|
Vendor base URL (
TYPE:
|
token
|
Bearer token.
TYPE:
|
purpose
|
Object purpose (default
TYPE:
|
timeout
|
Per-request timeout in seconds.
TYPE:
|
session
|
Optional
TYPE:
|
max_attempts
|
Attempts per vendor request.
TYPE:
|
backoff_seconds
|
Base backoff between attempts.
TYPE:
|
info_ttl_seconds
|
How long the
TYPE:
|
fetch_config
|
Download settings;
TYPE:
|
Source code in vgi_rpc/vendor.py
from_env
classmethod
¶
from_env(
*,
purpose: Purpose = "fs",
env: Mapping[str, str] | None = None
) -> VendorBlobClient
Build from VGI_BLOB_VENDOR_URL and the vendor token variables.
| PARAMETER | DESCRIPTION |
|---|---|
purpose
|
Object purpose.
TYPE:
|
env
|
Environment mapping (defaults to
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
VendorBlobClient
|
A configured :class: |
| RAISES | DESCRIPTION |
|---|---|
ValueError
|
If the URL or token is missing. |
Source code in vgi_rpc/vendor.py
put_bytes
¶
put_bytes(
data: bytes,
*,
scope: BlobScope | None = None,
idempotency_key: str | None = None
) -> BlobRef
Store data and return its reference.
| PARAMETER | DESCRIPTION |
|---|---|
data
|
The bytes to store.
TYPE:
|
scope
|
The execution the blob belongs to, for
TYPE:
|
idempotency_key
|
Makes a retried logical write reuse the same
object: the same key with the same size and SHA-256 returns
the same
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
A
|
class:
TYPE:
|
| RAISES | DESCRIPTION |
|---|---|
VendorError
|
If the vendor or the signed upload fails. |
Source code in vgi_rpc/vendor.py
get_bytes
¶
get_bytes(ref: BlobRef) -> bytes
Fetch and verify the blob ref names.
The download is pinned to the vendor's storage hosts, capped at the
fetch config's max_fetch_bytes, and checked against the size and
SHA-256 in ref (not the vendor's own report).
| PARAMETER | DESCRIPTION |
|---|---|
ref
|
The reference from :meth:
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
bytes
|
The blob bytes. |
| RAISES | DESCRIPTION |
|---|---|
VendorError
|
If the vendor fails, the object is missing, the download fails, or the bytes do not match ref. |
Source code in vgi_rpc/vendor.py
delete
¶
delete(refs: Iterable[BlobRef | str]) -> int
Delete objects by reference or object id (missing objects are ignored).
| PARAMETER | DESCRIPTION |
|---|---|
refs
|
References or object ids.
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
int
|
How many objects the vendor deleted. |
Source code in vgi_rpc/vendor.py
delete_scope
¶
Delete every object put under (shard, execution_id).
| PARAMETER | DESCRIPTION |
|---|---|
shard
|
The storage shard key.
TYPE:
|
execution_id
|
The execution id, encoded as in :class:
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
int
|
How many objects the vendor deleted. |
Source code in vgi_rpc/vendor.py
BlobRef
dataclass
¶
A reference to one vendored blob, small enough to store inline.
| ATTRIBUTE | DESCRIPTION |
|---|---|
object_id |
The vendor's object id.
TYPE:
|
size |
Exact byte length of the blob.
TYPE:
|
sha256 |
Lowercase hex SHA-256 of the blob bytes.
TYPE:
|
BlobScope
dataclass
¶
The execution a blob belongs to, for scoped clean-up.
The vendor accepts 1-128 characters of [A-Za-z0-9._:-] for each
field, so encode binary ids as hex (base64's +/= are refused).
| ATTRIBUTE | DESCRIPTION |
|---|---|
shard |
The storage shard key.
TYPE:
|
execution_id |
The execution identifier (e.g. hex of the raw id bytes).
TYPE:
|
VendorError
¶
Bases: RuntimeError
A blob-vendor or signed-URL request failed.
Messages never contain the bearer token or a signed query string.
| ATTRIBUTE | DESCRIPTION |
|---|---|
status |
The HTTP status of the failing response, or
TYPE:
|
code |
The vendor's machine-readable
TYPE:
|
Create the error.
| PARAMETER | DESCRIPTION |
|---|---|
message
|
Diagnostic text (already redacted).
TYPE:
|
status
|
HTTP status, when there was a response.
TYPE:
|
code
|
Machine-readable error code, when there was one.
TYPE:
|
Source code in vgi_rpc/vendor.py
pinned_hosts_validator
¶
pinned_hosts_validator(
hosts: Iterable[str],
*,
allow_http: bool = False,
prefixes: Iterable[str] = ()
) -> Callable[[str], None]
Build a url_validator that accepts only URLs on hosts.
A URL passes when its scheme is https (or http with
allow_http), it carries no userinfo, and its host and port are exactly
one of hosts. An entry is host or host:port ([v6]:port for
IPv6); an entry without a port matches the scheme's default port only.
Subdomains do not match. With prefixes, the URL (without its query)
must also start with one of them, e.g. a region's bucket path.
| PARAMETER | DESCRIPTION |
|---|---|
hosts
|
Allowed hosts.
TYPE:
|
allow_http
|
Also accept
TYPE:
|
prefixes
|
Optional URL prefixes the URL must start with.
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
Callable[[str], None]
|
A callable that raises |
| RAISES | DESCRIPTION |
|---|---|
ValueError
|
If hosts is empty or an entry will not parse. |
Source code in vgi_rpc/vendor.py
vendor_url_validator
¶
vendor_url_validator(
url: str,
token: str,
*,
ttl_seconds: float = _DEFAULT_INFO_TTL,
session: Client | None = None
) -> Callable[[str], None]
Build a url_validator pinned to the vendor's storage hosts.
The hosts come from the vendor's GET /v1/info, fetched on first use
and cached for ttl_seconds; a failed refresh keeps the last good list.
Only https URLs pass (http only when the vendor itself is a
loopback http URL, i.e. a local test vendor).
| PARAMETER | DESCRIPTION |
|---|---|
url
|
Vendor base URL.
TYPE:
|
token
|
Bearer token.
TYPE:
|
ttl_seconds
|
Cache lifetime of the host list.
TYPE:
|
session
|
Optional
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
Callable[[str], None]
|
A callable raising |
Source code in vgi_rpc/vendor.py
external_config_from_env
¶
Server external-location config from the blob-vendor environment.
Uses :class:VendorStorage for uploads, pins inbound pointer URLs to the
vendor's hosts, and reads VGI_RPC_EXTERNALIZE_THRESHOLD_BYTES
(default 1 MiB) and VGI_RPC_EXTERNAL_COMPRESSION (default none).
Pass config.storage as upload_url_provider to serve_http /
make_wsgi_app to also vend client upload URLs.
| PARAMETER | DESCRIPTION |
|---|---|
env
|
Environment mapping (defaults to
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
ServerExternalConfig | None
|
The config, or |
| RAISES | DESCRIPTION |
|---|---|
ValueError
|
If the vendor URL is set but the token is missing, or a variable will not parse. |
Source code in vgi_rpc/vendor.py
client_external_config_from_env
¶
client_external_config_from_env(
env: Mapping[str, str] | None = None,
) -> ClientExternalConfig | None
Client external-location config pinned to the blob vendor's hosts.
For clients that hold a vendor token (e.g. a worker calling another worker): pointer and upload URLs are accepted only on the vendor's storage hosts.
| PARAMETER | DESCRIPTION |
|---|---|
env
|
Environment mapping (defaults to
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
ClientExternalConfig | None
|
The config, or |
| RAISES | DESCRIPTION |
|---|---|
ValueError
|
If the vendor URL is set but the token is missing. |
Source code in vgi_rpc/vendor.py
max_upload_bytes_from_env
¶
Read VGI_RPC_MAX_UPLOAD_BYTES (advertised client-upload size).
| PARAMETER | DESCRIPTION |
|---|---|
env
|
Environment mapping (defaults to
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
int | None
|
The size, or |