cedar.stores.gcs module

Helper utilities for using Google Cloud Storage

class cedar.stores.gcs.GCSStore(client, bucket)[source]

Bases: object

Store GEE “pre-ARD” images and metadata on Google Cloud Storage

Parameters

  • client (google.cloud.storage.client.Client) – GCS client

  • bucket (google.cloud.storage.bucket.Bucket) – GCS bucket

classmethod from_credentials(bucket_name, credentials=None, project=None)[source]

Load Google Cloud Storage credentials and create store

list(self, path=None, pattern=None)[source]

List stored images or metadata

Parameters

  • path (str, optional) – Prefix path to search within

  • pattern (str, optional) – Filename pattern

Returns

Names of stored data

Return type

list[str]

read_metadata(self, name, path=None)[source]

Read and parse JSON metadata into a dict

Parameters

  • name (str) – Filename of metadata blob to read

  • path (str, optional) – Parent directory for file/object stored

Returns

JSON metadata blob

Return type

dict

remove(self, name, path=None)[source]

Remove a file from GCS

Parameters

  • name (str) – Name of stored file/object

  • path (str, optional) – Parent directory for file/object stored

Returns

Name of file removed

Return type

str

retrieve_image(self, dest, name, path=None, overwrite=True)[source]

Retrieve (pieces of) an image from the GCS

Parameters

  • dest (str) – Destination folder to save image(s)

  • name (str) – Name of stored file/object

  • path (str, optional) – Parent directory for file/object stored on GCS

Yields

Sequence[str] – Filename(s) corresponding to retrieved data

retrieve_metadata(self, dest, name, path=None, overwrite=True)[source]

Retrieve image metadata from the GCS

Parameters

  • dest (str) – Destination folder to save metadata

  • name (str) – Name of stored file/object

  • path (str, optional) – Parent directory for file/object stored

Yields

pathlib.Path – Filename corresponding to retrieved data

store_image(self, image, name, path=None, **export_image_kwds)[source]

Create ee.batch.Task to create and store “pre-ARD”

Parameters

  • image (ee.Image) – Earth Engine image to compute & store

  • name (str) – Name of file/object to store

  • path (str, optional) – Parent directory for file/object stored

  • export_image_kwds (dict, optional) – Additional keyword arguments to pass onto ee.batch.Export.image.toCloudStorage() (hint: scale & crs)

Returns

Earth Engine Task

Return type

ee.Task

store_metadata(self, metadata, name, path=None)[source]

Store JSON metadata

Parameters

  • metadata (dict) – Metadata, to be saved as JSON

  • name (str) – Name of file/object to store

  • path (str, optional) – Parent directory for file/object stored. Otherwise assumed to be part of name

Returns

Path to uploaded object

Return type

str

cedar.stores.gcs.build_gcs_client(credentials=None, project=None)[source]

Return a Google Cloud Store API service client

Parameters

  • credentials (str, optional) – File name of Google Cloud Store credentials (typically from a service account)

  • project (str, optional) – Google Cloud Platform project to use

Returns

Client for the Google Cloud Storage client library

Return type

google.cloud.storage.Client

Notes

You might consider setting the envirnment variable GOOGLE_APPLICATION_CREDENTIALS with the path to your service account credentials file [1].

References

1

https://cloud.google.com/storage/docs/reference/libraries#setting_up_authentication

cedar.stores.gcs.delete_blob(bucket, path)[source]

Delete a GCS blob

Parameters

  • bucket (str or google.cloud.storage.bucket.Bucket) – Bucket or bucket name

  • path (str) – Path to file/folder

Returns

Name of deleted blob

Return type

path

cedar.stores.gcs.download_blob(blob, dest)[source]

Download a blob to a destination directory

Parameters

  • blob (google.cloud.storage.blob.Blob) – GCS blob to download

  • dest (str) – Local directory to download blob into

Returns

Filename written to

Return type

pathlib.Path

cedar.stores.gcs.exists(bucket, path)[source]

Check if file/folder exists

Parameters

  • bucket (str or google.cloud.storage.bucket.Bucket) – Bucket or bucket name

  • path (str) – Path to file/folder

Returns

True or False if exists

Return type

bool

cedar.stores.gcs.list_blobs(bucket, prefix=None, pattern=None)[source]

Return file/non-directory blobs within a on GCS

Parameters

  • bucket (str or google.cloud.storage.bucket.Bucket) – Bucket or bucket name

  • prefix (str) – List contents within this folder

  • pattern (str, optional) – Filter search by glob pattern (i.e., *.json)

Returns

List of blob files inside at prefix

Return type

list[google.cloud.storage.blob.Blob]

cedar.stores.gcs.list_dirs(bucket, prefix=None)[source]

Return “directory” blobs within a on GCS

Parameters

  • bucket (str or google.cloud.storage.bucket.Bucket) – Bucket or bucket name

  • prefix (str) – List contents within this folder

Returns

List of directories names inside at prefix

Return type

Sequence[str]

cedar.stores.gcs.mkdir(bucket, path)[source]

Make a directory, recursively

Parameters

  • bucket (str or google.cloud.storage.bucket.Bucket) – Bucket or bucket name

  • path (str) – Path to folder

Returns

GCS blob for directory created

Return type

google.cloud.storage.blob.Blob

cedar.stores.gcs.mkdir_p(bucket, path)[source]

Create a “directory” on GCS

Parameters

  • bucket (str or google.cloud.storage.bucket.Bucket) – Bucket or bucket name

  • path (str) – Path to folder

Returns

GCS blob for directory created

Return type

google.cloud.storage.blob.Blob

Notes

Directories don’t really exist on GCS but we can fake it [1].

References

1

https://cloud.google.com/storage/docs/gsutil/addlhelp/HowSubdirectoriesWork

cedar.stores.gcs.read_json(blob, encoding='utf-8')[source]

Read a blob of JSON string data into a dict

Parameters

  • blob (google.cloud.storage.blob.Blob) – Blob to read, decode, and load

  • encoding (str, optional) – Metadata encoding

Returns

Blob read, decoded, and loaded as a dict

Return type

dict

cedar.stores.gcs.upload_json(bucket, data, path, check=False, encoding='utf-8')[source]

Upload data as JSON to GCS

Parameters

  • bucket (google.cloud.storage.bucket.Bucket) – GCS bucket

  • data (str or dict) – JSON, either already dumped to str or as a dict

  • path (str) – Destination path on GCS for data

  • check (bool, optional) – Check to see if the file already exists first. If so, will overwrite (or “update” instead of “create”)

  • encoding (str, optional) – Metadata encoding

Returns

JSON as GCS blob

Return type

google.cloud.storage.blob.Blob