Storage

In order to create “Analysis Ready Data” (ARD) using the Google Earth Engine, we have to track image metadata, from the timestamp to information like the image registration fit, throughout the export process. This application tracks both “pre-ARD” imagery and metadata through the creation process by ensuring both the imagery and metadata are saved to the same export location. The Google Earth Engine can export images to either Google Drive (ee.batch.Export.image.toDrive`()) or Google Cloud Storage (ee.batch.Export.image.toCloudStorage()). Accordingly, we can either store and retrieve “pre-ARD” imagery and metadata from either Google Cloud Storage or Google Drive.

Google Drive

class cedar.stores.gdrive.GDriveStore(service)

Store GEE “pre-ARD” images and metadata on Google Drive

Parameters

service (googleapiclient.discovery.Resource) – Google Drive API service

classmethod from_credentials(client_secrets_file=None, credentials_file=None)

Create and/or load credentials and create the store

Parameters

• client_secrets_file (str or Path) – Filename of “client_secrets.[…].json” file

• credentials_file (str or Path) – Filename of user credentials to load, or to save to for future use. If not provided, will use default location.

list(self, path=None, pattern=None)

List stored images or metadata

Parameters

• path (str, optional) – Prefix path to search within

• pattern (str, optional) – Filename pattern

Returns

Names of stored data

Return type

list[str]

read_metadata(self, name, path=None)

Read and parse JSON metadata into a dict

Parameters

• name (str) – Filename of metadata to read

• path (str, optional) – Parent directory for file/object stored

Returns

JSON metadata

Return type

dict

remove(self, name, path=None)

Remove a file from Google Drive

Parameters

• name (str) – Name of stored file/object

• path (str, optional) – Parent directory for file/object stored

Returns

Name of file removed

Return type

str

retrieve_image(self, dest, name, path=None, overwrite=True)

Retrieve (pieces of) an image from the Google Drive

Parameters

• dest (str) – Destination folder to save image(s)

• name (str) – Name of stored file/object

• path (str, optional) – Parent directory for file/object stored on Google Drive

Yields

Sequence[str] – Filename(s) corresponding to retrieved data

retrieve_metadata(self, dest, name, path=None, overwrite=True)

Retrieve image metadata from the GCS

Parameters

• dest (str) – Destination folder to save metadata

• name (str) – Name of stored file/object

• path (str, optional) – Parent directory for file/object stored

Yields

pathlib.Path – Filename corresponding to retrieved data

store_image(self, image, name, path=None, **export_image_kwds)

Create ee.batch.Task to create and store “pre-ARD”

Parameters

• image (ee.Image) – Earth Engine image to compute & store

• name (str) – Name of file/object to store

• path (str, optional) – Parent directory for file/object stored

• export_image_kwds (dict, optional) – Additional keyword arguments to pass onto ee.batch.Export.image.toCloudStorage() (hint: scale & crs)

Returns

Earth Engine Task

Return type

ee.Task

store_metadata(self, metadata, name, path=None)

Store JSON metadata

Parameters

• metadata (dict) – Metadata, to be saved as JSON

• name (str) – Name of file/object to store

• path (str, optional) – Parent directory for file/object stored

Returns

ID of file uploaded

Return type

str

Google Cloud Storage

class cedar.stores.gcs.GCSStore(client, bucket)

Store GEE “pre-ARD” images and metadata on Google Cloud Storage

Parameters

• client (google.cloud.storage.client.Client) – GCS client

• bucket (google.cloud.storage.bucket.Bucket) – GCS bucket

classmethod from_credentials(bucket_name, credentials=None, project=None)

Load Google Cloud Storage credentials and create store

list(self, path=None, pattern=None)

List stored images or metadata

Parameters

• path (str, optional) – Prefix path to search within

• pattern (str, optional) – Filename pattern

Returns

Names of stored data

Return type

list[str]

read_metadata(self, name, path=None)

Read and parse JSON metadata into a dict

Parameters

• name (str) – Filename of metadata blob to read

• path (str, optional) – Parent directory for file/object stored

Returns

JSON metadata blob

Return type

dict

remove(self, name, path=None)

Remove a file from GCS

Parameters

• name (str) – Name of stored file/object

• path (str, optional) – Parent directory for file/object stored

Returns

Name of file removed

Return type

str

retrieve_image(self, dest, name, path=None, overwrite=True)

Retrieve (pieces of) an image from the GCS

Parameters

• dest (str) – Destination folder to save image(s)

• name (str) – Name of stored file/object

• path (str, optional) – Parent directory for file/object stored on GCS

Yields

Sequence[str] – Filename(s) corresponding to retrieved data

retrieve_metadata(self, dest, name, path=None, overwrite=True)

Retrieve image metadata from the GCS

Parameters

• dest (str) – Destination folder to save metadata

• name (str) – Name of stored file/object

• path (str, optional) – Parent directory for file/object stored

Yields

pathlib.Path – Filename corresponding to retrieved data

store_image(self, image, name, path=None, **export_image_kwds)

Create ee.batch.Task to create and store “pre-ARD”

Parameters

• image (ee.Image) – Earth Engine image to compute & store

• name (str) – Name of file/object to store

• path (str, optional) – Parent directory for file/object stored

• export_image_kwds (dict, optional) – Additional keyword arguments to pass onto ee.batch.Export.image.toCloudStorage() (hint: scale & crs)

Returns

Earth Engine Task

Return type

ee.Task

store_metadata(self, metadata, name, path=None)

Store JSON metadata

Parameters

• metadata (dict) – Metadata, to be saved as JSON

• name (str) – Name of file/object to store

• path (str, optional) – Parent directory for file/object stored. Otherwise assumed to be part of name

Returns

Path to uploaded object

Return type

str