hyrax.download_stats ==================== .. py:module:: hyrax.download_stats Attributes ---------- .. autoapisummary:: hyrax.download_stats.logger Classes ------- .. autoapisummary:: hyrax.download_stats.StatRecord hyrax.download_stats.DownloadStats Module Contents --------------- .. py:data:: logger .. py:class:: StatRecord(received_at: datetime.datetime, data_start: datetime.datetime, **kwargs) Recording object that represents one or many stats measurements reported by download request threads. This object supports creation with a full set of measurements from one thread. It also supports combining instances of the class to create aggregate stats from several measurements. .. py:attribute:: field_defaults .. py:attribute:: received_at .. py:attribute:: data_start .. py:attribute:: N :value: 1 .. py:method:: combine(obja: StatRecord, objb: Optional[StatRecord]) -> StatRecord :staticmethod: Combine two stats object into a new stats object representing the combination of the two. Stats objects can be the result of n processed data points, so when combining two objects with n and m data points, the new object will indicate it is composed of n+m data points. :param obja: The first stats record :type obja: StatRecord :param objb: The second stats record. If not provided a copy of obja will be returned. :type objb: StatRecord, optional :returns: A new object with the combination of the two stats objects. :rtype: StatRecord .. py:method:: _combine(obj: StatRecord) -> StatRecord Combine this object with another. Private helper for combine(). .. py:method:: _stat_update(key: str, value: Union[int, datetime.timedelta], m: int = 1) Update a stat within the record object. Automatically averages if the stat key ends with _avg :param key: The stat key to update. :type key: str :param value: The value to update it with. :type value: Union[int, datetime.timedelta] :param m: If we are averaging, we need to know how many data points the current average you are passing in represents, by default 1 :type m: int, optional .. py:method:: _div(num: float, denom: float, default: float = 0.0) -> float :staticmethod: .. py:method:: wall_clock_dur_s() -> int The wall clock duration that is represented by this stats object. How long between the beginning of when it was recording to the moment it was reported to the stats system in seconds? :returns: The number of seconds :rtype: int .. py:method:: total_dur_s() -> int Total duration of time that a thread was doing data transfer. In the case of multiple threads each thread's seconds are added. A stats object can represent an amalgamation of measurements from multiple worker threads. `total_dur_s` represents the total time data was being transferred added across all threads. :returns: The number of seconds :rtype: int .. py:method:: resp_s() -> int The time spent receiving responses from the server, added across all threads. :returns: A number of seconds :rtype: int .. py:method:: data_down_mb() -> float The amount of data downloaded by all threads together. :returns: A flooating point number of 1024-based megabytes :rtype: float .. py:method:: down_rate_mb_s() -> float The downstream data rate in megabytes per second experienced by the average thread. `data_down_mb`/`resp_s`. :returns: A floating point number of 1024-based megabytes per second :rtype: float .. py:method:: down_rate_mb_s_overall() -> float The downstream data rate in megabytes per second expereineced by an aggregate of threads. `data_down_mb`/`wall_clock_dur_s` :returns: A floating point number of 1024-based megabytes per second :rtype: float .. py:method:: req_s() -> int The time spent sending requests to the server, added across all threads. :returns: A number of seconds :rtype: int .. py:method:: data_up_mb() -> float The amount of data uploaded by all threads together. :returns: A flooating point number of 1024-based megabytes :rtype: float .. py:method:: up_rate_mb_s() -> float The upstream data rate in megabytes per second experienced by the average thread. `data_up_mb`/`resp_s`. :returns: A floating point number of 1024-based megabytes per second :rtype: float .. py:method:: snapshot_rate() -> float The rate of snapshots downloaded experienced by the average thread. (`snapshots`/`total_dur_s`) * 3600 * 24 :returns: A floating point number of snapshot images per day :rtype: float .. py:method:: snapshot_rate_overall() -> float The rate of snapshots downloaded by all threads together. ()`snapshots`/`wall_clock_dur_s`) * 3600 * 24 :returns: A floating point number of snapshot images per day :rtype: float .. py:method:: log_summary(log_level: int = logging.INFO, num_threads: int = 1, prefix: str = '') Log two lines of summary of this stats object at the given log level. The first line is an overall summary that treats all threads that gave the stats object data as a single unit The second line is a per-thread summary that produces thread-averaged statistics. If provided, prefix is appended to the beginning of the log lines emitted. :param log_level: The logging level to emit the log message at. :type log_level: int :param num_threads: Number of threads actively reporting into the stats object we want to print, by default 1 :type num_threads: int :param prefix: String to prefix all log messages with. "" by default :type prefix: str .. py:class:: DownloadStats(print_interval_s=60) Subsytem for keeping statistics on downloads. Used as a context manager in worker threads. .. py:attribute:: window_size .. py:attribute:: lock .. py:attribute:: cumulative_stats :value: None .. py:attribute:: stats_window :value: [] .. py:attribute:: active_threads :value: 0 .. py:attribute:: num_threads :value: 0 .. py:attribute:: print_stats :value: False .. py:attribute:: print_interval_s :value: 60 .. py:attribute:: watcher_thread .. py:method:: __enter__() .. py:method:: __exit__(exc_type, exc_value, traceback) .. py:method:: _watcher_thread(log_level) .. py:method:: _print_stats(log_level) .. py:method:: hook(request: urllib.request.Request, request_start: datetime.datetime, response_start: datetime.datetime, response_size: int, chunk_size: int) This hook is called on each chunk of snapshots downloaded. It is called immediately after the server has finished responding to the request, so datetime.datetime.now() is the end moment of the request :param request: The request object relevant to this call :type request: urllib.request.Request :param request_start: The moment the request was handed off to urllib.request.urlopen() :type request_start: datetime.datetime :param response_start: The moment there were bytes from the server to process :type response_start: datetime.datetime :param response_size: The size of the response from the server in bytes :type response_size: int :param chunk_size: The number of cutout files recieved in this request :type chunk_size: int