Skip to content
potluckdb

Start here

↑ ↓ Select↵ OpenEsc Close

GitHub

Storage

Python: also in Node · Go · Rust · Swift · Dart · Java and Kotlin · .NET · C++ · Objective-C

Remotes using storage services through the shared native storage library.

Concepts explains the remote and local store.

Each constructor returns an ordinary remote. Pass it to the same open call and close it the same way. The other reference pages apply to these remotes without change.

potluckdb.storage.from_s3

Section titled “potluckdb.storage.from_s3”

A remote over an S3-compatible bucket.

potluckdb.storage.from_s3(
    bucket: str,
    namespace: str,
    *,
    region: str | None = None,
    endpoint: str | None = None,
    force_path_style: bool = False,
    access_key_id: str | None = None,
    secret_access_key: str | None = None,
    session_token: str | None = None,
    options: RemoteOptions | None = None,
) -> Remote

Parameters

  • bucket: The bucket holding the objects.
  • namespace: The database’s prefix. One bucket can hold many databases.
  • region: The signing region. If absent, the AWS chain selects it.
  • endpoint: The endpoint of a storage service other than AWS.
  • force_path_style: Address the bucket as a path segment instead of a subdomain.
  • access_key_id: An explicit access key. Supply it together with secret_access_key, or supply neither.
  • secret_access_key: The secret for access_key_id.
  • session_token: The session token for temporary credentials.
  • options: Remote concurrency and retention, as elsewhere in the SDK.

Returns

Remote. The reusable storage remote.

Errors

Other errors can occur. See Errors for every code.

Example

from potluckdb import storage
with storage.from_s3(
    "notes", "app", endpoint="http://127.0.0.1:43871", region="auto",
    force_path_style=True, access_key_id="demo", secret_access_key="demo",
) as remote:
    print("s3:", remote.status().closed)
Details

Credentials come from the AWS chain unless access_key_id and secret_access_key are supplied. aws-config resolves the chain itself:

  • Environment
  • Shared configuration and credentials files
  • SSO
  • Web identity
  • Container and instance metadata

A storage service other than AWS is named by endpoint. The AWS chain still supplies credentials unless access_key_id and secret_access_key are supplied, so supply the key pair that the service issues. Cloudflare R2 issues a pair with an account endpoint and requires region set to “auto”. MinIO and most self-hosted gateways require force_path_style. This addresses the bucket as a path segment instead of a subdomain.

Construction sends a HeadBucket request to check whether the bucket exists. If nothing listens at the endpoint, or it returns another error, construction proceeds; later requests can fail. If the endpoint accepts the connection but does not reply, construction waits without a time limit. If the endpoint reports a missing bucket, construction fails with bucket_not_found.

potluckdb.storage.from_gcs

Section titled “potluckdb.storage.from_gcs”

A remote over a Google Cloud Storage bucket.

potluckdb.storage.from_gcs(
    bucket: str,
    namespace: str,
    *,
    endpoint: str | None = None,
    service_account: dict[str, object] | str | None = None,
    options: RemoteOptions | None = None,
) -> Remote

Parameters

  • bucket: The bucket holding the objects.
  • namespace: The database’s prefix. One bucket can hold many databases.
  • endpoint: The endpoint of a storage service other than Google’s.
  • service_account: Google’s service account JSON, as a dict or text.
  • options: Remote concurrency and retention, as elsewhere in the SDK.

Returns

Remote. The reusable storage remote.

Errors

Other errors can occur. See Errors for every code.

Example

from potluckdb import storage
key = {"client_email": "demo@example.com", "private_key_id": "demo",
       "private_key": "demo", "project_id": "demo"}
with storage.from_gcs("notes", "app", endpoint="http://127.0.0.1:43871",
                      service_account=key) as remote:
    print("gcs:", remote.status().closed)
Details

Unless service_account is supplied, google-cloud-auth resolves Application Default Credentials in this order:

  • GOOGLE_APPLICATION_CREDENTIALS
  • The gcloud user credentials file
  • The metadata server

If the process has no such chain, pass Google’s service account JSON as a dict or text. A storage service other than Google’s is named by endpoint, including gateways and emulators.

potluckdb.storage.from_azure

Section titled “potluckdb.storage.from_azure”

A remote over an Azure Blob Storage container.

potluckdb.storage.from_azure(
    account: str,
    container: str,
    namespace: str,
    *,
    endpoint: str | None = None,
    account_key: str | None = None,
    options: RemoteOptions | None = None,
) -> Remote

Parameters

  • account: The storage account holding the container.
  • container: The container holding the objects.
  • namespace: The database’s prefix. One container can hold many databases.
  • endpoint: The endpoint of a storage service other than Azure’s.
  • account_key: An explicit account key.
  • options: Remote concurrency and retention, as elsewhere in the SDK.

Returns

Remote. The reusable storage remote.

Errors

Other errors can occur. See Errors for every code.

Example

from potluckdb import storage
with storage.from_azure(
    "demo", "notes", "app", endpoint="http://127.0.0.1:10000/demo",
    account_key="ZGVtbw==",
) as remote:
    print("azure:", remote.status().closed)
Details

Unless account_key is supplied, azure-identity resolves the Azure credential chain itself:

  • Environment
  • Workload identity
  • Managed identity
  • Azure CLI
  • Developer tools

Pass an account key for a storage service without such a chain. Azurite supplies the well-known devstoreaccount1 pair.

A storage service other than Azure’s is named by endpoint, passed to the library as endpoint_url. Azurite provides one account per port, so its endpoint includes the account as a path segment.