hyrax.download

Attributes

logger

Classes

Downloader

Class with primarily static methods to namespace downloader related constants and functions.

Module Contents

logger

class Downloader(config)

Class with primarily static methods to namespace downloader related constants and functions.

VARIABLE_FIELDS = ['tract', 'ra', 'dec']

RECT_COLUMN_NAMES = ['tract', 'ra', 'dec', 'ra', 'dec', 'sw', 'sh', 'filter', 'type', 'rerun', 'image', 'variance',...

MANIFEST_COLUMN_NAMES = ['tract', 'ra', 'dec', 'ra', 'dec', 'sw', 'sh', 'filter', 'type', 'rerun', 'image', 'variance',...

MANIFEST_FILE_NAME = 'manifest.fits'

config

cutout_path

manifest_file

run()

Main entrypoint for downloading cutouts from HSC for use with Hyrax

Parameters:

config (dict) – Runtime configuration as a nested dictionary

_prune_downloaded_rects()

Prunes already downloaded rects using the manifest in cutout_path. rects passed in is mutated by this operation

Returns:

Returns rects that was passed in. This is only to enable explicit style at the call site. ` rects` is mutated by this function.

Return type:

list[dC.Rect]

Raises:

RuntimeError – When there is an issue reading the manifest file, or the manifest file corresponds to a different set of cutouts than the current download being attempted

static _keep_rect(location_rect: hyrax.downloadCutout.downloadCutout.Rect, prior_manifest: dict[hyrax.downloadCutout.downloadCutout.Rect, str]) → bool

Private helper function to prune_downloaded_rects which operates the inner loop of the prune function, and allows it to be written as a list comprehension.

This function decides element-by-element for our rects that we want to download whether or not these rects have already been downloaded in a prior download, given the manifest from that prior download.

Parameters:

• location_rect (dC.Rect) – A rectangle on the sky that we are considering downloading.

• prior_manifest (dict[dC.Rect,str]) – The manifest of the prior download. This object is slowly consumed by repeated calls to this function. When the return value is False, all manifest entries corresponding to the passed in location_rect have been removed.

Returns:

Whether this sky location location_rect should be included in the download

Return type:

bool

_write_manifest()

Write out manifest fits file that is an inventory of the download. The manifest fits file should have columns object_id, ra, dec, tract, filter, filename

If filename is empty string (“”) that means a download attempt was made, but did not succeed If the object is not present in the manifest, no download was attempted. If the object is present in the manifest and the filename is not empty string that file exists and downloaded successfully.

This file respects the existence of other manifest files in the directory and operates additively. If a manifest file is present from an earlier download, this function will read that manifest in, and include the entire content of that manifest in addition to the manifests passed in.

The format of the manifest file has the following columns

object_id: The object ID from the original catalog filename: The file name where the file can be found OR the string “Attempted” indicating the download did not complete successfully. tract: The HSC tract ID number this either comes from the catalog or is the tract ID returned by the cutout server for downloaded files.

ra: Right ascension in degrees of the center of the cutout box dec: Declination in degrees of the center of the cutout box filter: The name of the filter requested sw: Semi-width of the cutout box in degrees sh: Semi-height of the cutout box in degrees rerun: The data release in use e.g. pdr3_wide type: coadd, warp, or other values allowed by the HSC docs dim: Tuple of integers with the dimensions of the image.

get_manifest()

Get the current downloader manifest, which is a list of files where download has been attempted The format of the table is outlined in _write_manifest()

Returns:

The entire download manifest

Return type:

astropy.table.Table

manifest_to_rects() → dict[hyrax.downloadCutout.downloadCutout.Rect, str]

Read the manifest.fits file from the given directory and return its contents as a dictionary with downloadCutout.Rectangles as keys and filenames as values.

If now manifest file is found, an empty dict is returned.

Returns:

A dictionary containing all the rects in the manifest and all the filenames, or empty dict if no manifest is found.

Return type:

dict[dC.Rect, str]

static _rect_hook(rect: hyrax.downloadCutout.downloadCutout.Rect, filename: pathlib.Path | str)

static download_thread(rects: list[hyrax.downloadCutout.downloadCutout.Rect], user: str, password: str, stats: hyrax.download_stats.DownloadStats, thread_num: int, manifest: dict[hyrax.downloadCutout.downloadCutout.Rect, str], **kwargs)

Download cutouts to the given directory. Called in its own thread with an id number.

Calls downloadCutout.download, so supports long lists of rects beyond the limits of the HSC web API

Parameters:

• rects (list[dC.Rect]) – The rects we would like to download

• user (string) – Username for HSC’s download service to use

• password (string) – Password for HSC’s download service to use

• stats (DownloadStats) – Instance of DownloadStats to use for stats tracking.

• thread_num (int,) – The ID number of thread we are, sequential from zero to num_threads-1

• manifest – A dictionary from dC.Rect to filename which we will fill in in as we download rects. This is the chief returned piece of data from each thread.

• **kwargs (dict) – Additonal arguments for downloadCutout.download. See downloadCutout.download for details

static filterfits(filename: pathlib.Path, column_names: list[str]) → astropy.table.Table

Read a fits file with the required column names for making cutouts

The easiest way to make such a fits file is to select from the main HSC catalog

Parameters:

• filename (str) – The fits file to read in

• column_names (list[str]) – The columns that are selected from the file and returned in the astropy Table.

Returns:

• Table

• Returns an astropy table containing only the fields specified in column_names

_rect_from_config() → hyrax.downloadCutout.downloadCutout.Rect

Takes our runtime config and loads cutout config common to all cutouts into a prototypical Rect for downloading

Returns:

A single rectangle with fields sw, sh, filter, rerun, and type populated from the config

Return type:

dC.Rect

static create_rects(locations: astropy.table.Table, path: pathlib.Path, offset: int = 0, default: hyrax.downloadCutout.downloadCutout.Rect | None = None, fields: list[str] | None = None) → list[hyrax.downloadCutout.downloadCutout.Rect]

Create the rects we will need to pass to the downloader. One Rect per location in our list of sky locations.

Rects are created with all fields in the default rect pre-filled

Offset here is to allow multiple downloads on different sections of the source list without file name clobbering during the download phase. The offset is intended to be the index of the start of the locations table within some larger fits file.

Parameters:

• locations (Table) – Table containing ra, dec locations in the sky

• path (Path) – Directory where the cutuout files ought live. Used to generate file names on the rect object.

• offset (int, optional) – Index to start the lineno field in the rects at, by default 0. The purpose of this is to allow multiple downloads on different sections of a larger source list without file name clobbering during the download phase. This is important because lineno in a rect can becomes a file name parameter The offset is intended to be the index of the start of the locations table within some larger fits file.

• default (dC.Rect, optional) – The default Rect that contains properties common to all sky locations, by default None

• fields (list[str], optional) – Default fields to pull from the locations table. If not provided, defaults to [“tract”, “ra”, “dec”]

Returns:

Rects populated with sky locations from the table

Return type:

list[dC.Rect]