hyrax.download ============== .. py:module:: hyrax.download Attributes ---------- .. autoapisummary:: hyrax.download.logger Classes ------- .. autoapisummary:: hyrax.download.Downloader Module Contents --------------- .. py:data:: logger .. py:class:: Downloader(config) Class with primarily static methods to namespace downloader related constants and functions. .. py:attribute:: VARIABLE_FIELDS :value: ['tract', 'ra', 'dec'] .. py:attribute:: RECT_COLUMN_NAMES :value: ['tract', 'ra', 'dec', 'ra', 'dec', 'sw', 'sh', 'filter', 'type', 'rerun', 'image', 'variance',... .. py:attribute:: MANIFEST_COLUMN_NAMES :value: ['tract', 'ra', 'dec', 'ra', 'dec', 'sw', 'sh', 'filter', 'type', 'rerun', 'image', 'variance',... .. py:attribute:: MANIFEST_FILE_NAME :value: 'manifest.fits' .. py:attribute:: config .. py:attribute:: cutout_path .. py:attribute:: manifest_file .. py:method:: run() Main entrypoint for downloading cutouts from HSC for use with Hyrax :param config: Runtime configuration as a nested dictionary :type config: dict .. py:method:: _prune_downloaded_rects() Prunes already downloaded rects using the manifest in `cutout_path`. `rects` passed in is mutated by this operation :returns: Returns `rects` that was passed in. This is only to enable explicit style at the call site. ` rects` is mutated by this function. :rtype: list[dC.Rect] :raises RuntimeError: When there is an issue reading the manifest file, or the manifest file corresponds to a different set of cutouts than the current download being attempted .. py:method:: _keep_rect(location_rect: hyrax.downloadCutout.downloadCutout.Rect, prior_manifest: dict[hyrax.downloadCutout.downloadCutout.Rect, str]) -> bool :staticmethod: Private helper function to prune_downloaded_rects which operates the inner loop of the prune function, and allows it to be written as a list comprehension. This function decides element-by-element for our rects that we want to download whether or not these rects have already been downloaded in a prior download, given the manifest from that prior download. :param location_rect: A rectangle on the sky that we are considering downloading. :type location_rect: dC.Rect :param prior_manifest: The manifest of the prior download. This object is slowly consumed by repeated calls to this function. When the return value is False, all manifest entries corresponding to the passed in location_rect have been removed. :type prior_manifest: dict[dC.Rect,str] :returns: Whether this sky location `location_rect` should be included in the download :rtype: bool .. py:method:: _write_manifest() Write out manifest fits file that is an inventory of the download. The manifest fits file should have columns object_id, ra, dec, tract, filter, filename If filename is empty string ("") that means a download attempt was made, but did not succeed If the object is not present in the manifest, no download was attempted. If the object is present in the manifest and the filename is not empty string that file exists and downloaded successfully. This file respects the existence of other manifest files in the directory and operates additively. If a manifest file is present from an earlier download, this function will read that manifest in, and include the entire content of that manifest in addition to the manifests passed in. The format of the manifest file has the following columns object_id: The object ID from the original catalog filename: The file name where the file can be found OR the string "Attempted" indicating the download did not complete successfully. tract: The HSC tract ID number this either comes from the catalog or is the tract ID returned by the cutout server for downloaded files. ra: Right ascension in degrees of the center of the cutout box dec: Declination in degrees of the center of the cutout box filter: The name of the filter requested sw: Semi-width of the cutout box in degrees sh: Semi-height of the cutout box in degrees rerun: The data release in use e.g. pdr3_wide type: coadd, warp, or other values allowed by the HSC docs dim: Tuple of integers with the dimensions of the image. .. py:method:: get_manifest() Get the current downloader manifest, which is a list of files where download has been attempted The format of the table is outlined in _write_manifest() :returns: The entire download manifest :rtype: astropy.table.Table .. py:method:: manifest_to_rects() -> dict[hyrax.downloadCutout.downloadCutout.Rect, str] Read the manifest.fits file from the given directory and return its contents as a dictionary with downloadCutout.Rectangles as keys and filenames as values. If now manifest file is found, an empty dict is returned. :returns: A dictionary containing all the rects in the manifest and all the filenames, or empty dict if no manifest is found. :rtype: dict[dC.Rect, str] .. py:method:: _rect_hook(rect: hyrax.downloadCutout.downloadCutout.Rect, filename: Union[pathlib.Path, str]) :staticmethod: .. py:method:: download_thread(rects: list[hyrax.downloadCutout.downloadCutout.Rect], user: str, password: str, stats: hyrax.download_stats.DownloadStats, thread_num: int, manifest: dict[hyrax.downloadCutout.downloadCutout.Rect, str], **kwargs) :staticmethod: Download cutouts to the given directory. Called in its own thread with an id number. Calls downloadCutout.download, so supports long lists of rects beyond the limits of the HSC web API :param rects: The rects we would like to download :type rects: list[dC.Rect] :param user: Username for HSC's download service to use :type user: string :param password: Password for HSC's download service to use :type password: string :param stats: Instance of DownloadStats to use for stats tracking. :type stats: DownloadStats :param thread_num: The ID number of thread we are, sequential from zero to num_threads-1 :type thread_num: int, :param manifest: A dictionary from dC.Rect to filename which we will fill in in as we download rects. This is the chief returned piece of data from each thread. :param \*\*kwargs: Additonal arguments for downloadCutout.download. See downloadCutout.download for details :type \*\*kwargs: dict .. py:method:: filterfits(filename: pathlib.Path, column_names: list[str]) -> astropy.table.Table :staticmethod: Read a fits file with the required column names for making cutouts The easiest way to make such a fits file is to select from the main HSC catalog :param filename: The fits file to read in :type filename: str :param column_names: The columns that are selected from the file and returned in the astropy Table. :type column_names: list[str] :returns: * *Table* * *Returns an astropy table containing only the fields specified in column_names* .. py:method:: _rect_from_config() -> hyrax.downloadCutout.downloadCutout.Rect Takes our runtime config and loads cutout config common to all cutouts into a prototypical Rect for downloading :returns: A single rectangle with fields `sw`, `sh`, `filter`, `rerun`, and `type` populated from the config :rtype: dC.Rect .. py:method:: create_rects(locations: astropy.table.Table, path: pathlib.Path, offset: int = 0, default: hyrax.downloadCutout.downloadCutout.Rect | None = None, fields: list[str] | None = None) -> list[hyrax.downloadCutout.downloadCutout.Rect] :staticmethod: Create the rects we will need to pass to the downloader. One Rect per location in our list of sky locations. Rects are created with all fields in the default rect pre-filled Offset here is to allow multiple downloads on different sections of the source list without file name clobbering during the download phase. The offset is intended to be the index of the start of the locations table within some larger fits file. :param locations: Table containing ra, dec locations in the sky :type locations: Table :param path: Directory where the cutuout files ought live. Used to generate file names on the rect object. :type path: Path :param offset: Index to start the `lineno` field in the rects at, by default 0. The purpose of this is to allow multiple downloads on different sections of a larger source list without file name clobbering during the download phase. This is important because `lineno` in a rect can becomes a file name parameter The offset is intended to be the index of the start of the locations table within some larger fits file. :type offset: int, optional :param default: The default Rect that contains properties common to all sky locations, by default None :type default: dC.Rect, optional :param fields: Default fields to pull from the locations table. If not provided, defaults to ["tract", "ra", "dec"] :type fields: list[str], optional :returns: Rects populated with sky locations from the table :rtype: list[dC.Rect]