cedar.tracker module

Tracker to submit and download GEE pre-ARD tasks

class cedar.tracker.Tracker(tile_grid, store, name_template='{collection}_h{tile.horizontal:03d}v{tile.vertical:03d}_{date_start}_{date_end}', prefix_template='CEDAR_PREARD', tracking_template='TRACKING_PERIOD{date_start}-{date_end}_TASK{today}', tracking_prefix='CEDAR_TRACKING', filters=None, export_image_kwds=None)[source]

Bases: object

CEDAR “pre-ARD” order tracker

Parameters

  • tile_grid (stems.gis.grids.TileGrid) – Tile Grid to use for ARD

  • store (cedar.stores.Store) – A Store that can be used to store images & metadata

  • name_template (str, optional) – Template for “pre-ARD” image and metadata name

  • prefix_template (str, optional) – Template for “pre-ARD” image and metadata prefix

  • tracking_template (str, optional) – Template for order tracking file name

  • tracking_prefix (str, optional) – Order tracking file prefix folder

  • filters (Dict[str, Sequence[dict or ee.Filter]]) – Earth Engine filters to apply, organized by image collection name. Values should either be ee.Filter objects or dictionaries that describe the filter (see cedar.utils.serialize_filter())

clean(self, tracking_info, tracking_name=None, callback=None)[source]

Clean “pre-ARD” imagery, metadata, and tracking metadata off GCS

Parameters

  • tracking_info (dict) – JSON tracking info data as a dict

  • tracking_name (str) – Name of tracking info file (will be deleted if provided)

  • callback (callable) – Callback function to execute after each file is deleted. Should take arguments “item” and “n_steps”. Use this for progress bars or other download status reporting

Returns

Mapping of GEE Task ID to filename(s) cleaned

Return type

dict[str, list[str]]

download(self, tracking_info, dest, overwrite=True, callback=None)[source]

Download “pre-ARD” and metadata to a directory

Parameters

  • tracking_info (dict) – JSON tracking info data as a dict

  • dest (str or pathlib.Path) – Destination download directory

  • overwrite (bool, optional) – Overwrite existing downloaded data

  • callback (callable) – Callback function to execute after each file is downloaded. Should take arguments “item” and “n_steps”. Use this for progress bars or other download status reporting

Returns

Key value pairs mapping GEE task IDs to the filenames of downloaded data. Wrap it in a dict to make it not lazy

Return type

tuple[str, list[str]]

property filters

Earth Engine filters to apply

Type

list[ee.Filter]

list(self, pattern=None)[source]

Return a list of all tracking metadata

Parameters

pattern (str, optional) – Search pattern for tracking info. Specify to subset to specific tracking info (e.g., from some date). If None provided, looks for tracking information matching tracking_template

Returns

Name of stored tracking information

Return type

list[str]

read(self, name)[source]

Returns stored tracking information as dict

Parameters

name (str) – Name of tracking metadata (e.g., taken from running list_tracking())

Returns

JSON tracking info data as a dict

Return type

dict

submit(self, collections, tile_indices, period_start, period_end, period_freq=None, save_empty_metadata=True, error_if_empty=False)[source]

Submit and track GEE pre-ARD tasks

Parameters

  • collections (str or Sequence[str]) – GEE image collection name(s)

  • tile_indices (Sequence[(int, int)]) – Tuple(s) of rows/columns in TileGrid to process

  • period_start (dt.datetime) – Starting period date

  • period_end (dt.datetime) – Ending period date

  • period_freq (str, optional) – If provided, period_start, period_end, and period_freq are interpeted as the range for pandas.date_range() and one or more Tasks will be submitted

  • save_empty_metadata (bool, optional) – If True, Pre-ARD image requests that have 0 results (e.g., because of spotty historical record) will store metadata, but will not start the task. If False, will not store this metadata

  • error_if_empty (bool, optional) – If True, raise an EmptyCollectionError if the image collection result has no images. The default behavior is to log and skip empty search results

Returns

  • str – Task tracking information name

  • str – Task tracking information identifier (an ID, path, etc)

update(self, name)[source]

Refresh and reupload tracking information by checking with the GEE

Parameters

name (str) – Name of tracking metadata (e.g., taken from running list_tracking())

Returns

JSON tracking info data as a dict

Return type

dict

cedar.tracker.clean_tracked(tracking_info, store)[source]

Delete stored “pre-ARD” and metadata described by tracking info

Parameters
Yields

  • id (str) – Order GEE Task ID

  • names (generator) – Generator that deletes files and returns their names

cedar.tracker.download_tracked(tracking_info, store, dest, overwrite=False)[source]

Download stored “pre-ARD” and metadata described by tracking info

Parameters

  • tracking_info (dict) – Tracking information

  • store (cedar.stores.Store) – cedar store class

  • dest (str or pathlib.Path) – Destination download directory

  • overwrite (bool, optional) – Overwrite previously downoaded data, or not

Yields

  • id (str) – Task ID

  • n_images (int) – Number of images to download if known, otherwise None

  • metadata (generator) – Generator that downloads metadata and yields filenames

  • images (generator) – Generator that downloads imagery and yields filenames