hyrax.config_utils

Attributes

`DEFAULT_CONFIG_FILEPATH`
`DEFAULT_USER_CONFIG_FILEPATH`
`KEYS_WITH_EXTERNAL_LIBS`
`logger`

Classes

ConfigManager

A class to manage the runtime configuration for a Hyrax object. This class

Functions

`config_help`(config, *args)	A simple config help function. It's a bit difficult to parse through
`parse_dotted_key`(→ list[str])	Parse a dotted key string, respecting quoted sections.
`find_keys`(config, key_name)	Recursively find all keys in a nested dictionary that match the given key name.
`create_results_dir`(→ pathlib.Path)	Creates a results directory for this run.
`find_most_recent_results_dir`(→ Optional[pathlib.Path])	Find the most recent results directory corresponding to a particular verb
`log_runtime_config`(runtime_config, output_path[, ...])	Log a runtime configuration.

Module Contents

DEFAULT_CONFIG_FILEPATH[source]

DEFAULT_USER_CONFIG_FILEPATH[source]

KEYS_WITH_EXTERNAL_LIBS = ['name', 'dataset_class'][source]

logger[source]

config_help(config: tomlkit.toml_document.TOMLDocument, *args)[source]

A simple config help function. It’s a bit difficult to parse through the Tomlkit Table to print just one item such that it would include the comments preceding it.

For now, we support the following cases, and generally print out the entire table for the given key.

Cases: - if no args, prints the whole config. - if args[0] is a table name, print the whole table - if args[0] is not a table, assume it’s a key and search – print each one of the tables that it is found in.

Parameters:

config (TOMLDocument) – A configuration dictionary that will be used to search for specified tables and keys.
args (str) – A variable number of string arguments that specify the table name or key to search for in the configuration dictionary.

parse_dotted_key(key: str) → list[str][source]

Parse a dotted key string, respecting quoted sections.

Quoted sections (using single or double quotes) are treated as a single key component, even if they contain dots. This allows for keys like ‘torch.optim.Adam’ to be used as a single table name in TOML configuration files.

Parameters:: key (str) – The dotted key to parse. Examples: - “model.name” -> [‘model’, ‘name’] - “‘torch.optim.Adam’.lr” -> [‘torch.optim.Adam’, ‘lr’] - ‘“torch.optim.Adam”.lr’ -> [‘torch.optim.Adam’, ‘lr’] - “optimizer.’torch.optim.Adam’.lr” -> [‘optimizer’, ‘torch.optim.Adam’, ‘lr’]
Returns:: A list of key components
Return type:: list[str]

find_keys(config: dict[str, Any], key_name: str)[source]

Recursively find all keys in a nested dictionary that match the given key name.

Parameters:

config (dict) – The nested dictionary to search.
key_name (str) – The name of the key to find.

Returns:

A list of matching keys.

Return type:

list

class ConfigManager(runtime_config_filepath: pathlib.Path | str | None = None, default_config_filepath: pathlib.Path | str = DEFAULT_CONFIG_FILEPATH)[source]

A class to manage the runtime configuration for a Hyrax object. This class will contain all the logic and methods for reading, merging, and validating the runtime configuration.

_called_from_test = False[source]: Hardcoded set of config keys which we know to contain paths, and we resolve to global paths during initialization in ConfigManager._resolve_config_paths().

PATH_CONFIG_KEYS = [['data_set', 'filter_catalog'], ['general', 'data_dir']][source]

hyrax_default_config: tomlkit.toml_document.TOMLDocument[source]

runtime_config_filepath = None[source]

config[source]

original_config[source]

static _render_config(user_specific_config: tomlkit.toml_document.TOMLDocument = None, hyrax_default_config: tomlkit.toml_document.TOMLDocument = None)[source]

set_config(key: str, value: Any)[source]

Set a config value at runtime. This modifies the in-memory config object. Once the configuration is updated, the entire config is re-rendered to ensure that any requested external library default configs are incorporated.

Parameters:

key (str) – The dotted key to set, e.g. “model.name” or “‘torch.optim.Adam’.lr” Quoted sections (using single or double quotes) are treated as single key components, allowing for table names like ‘torch.optim.Adam’.
value (Any) – The value to set the key to.

static read_runtime_config(config_filepath: pathlib.Path | str = DEFAULT_CONFIG_FILEPATH) → tomlkit.toml_document.TOMLDocument[source]

Read a single toml file and return a TOMLDocument

Parameters:: config_filepath (Union[Path, str], optional) – The path to the config file, by default DEFAULT_CONFIG_FILEPATH
Returns:: The contents of the toml file as a tomlkit.TOMLDocument
Return type:: TOMLDocument

static _find_external_library_default_config_paths(runtime_config: dict) → set[source]

Search for external libraries in the runtime configuration and gather the libpath specifications so that we can load the default configs for the libraries.

Parameters:: runtime_config (dict) – The runtime configuration as a tomlkit.TOMLDocument.
Returns:: A tuple containing the default configuration Paths for the external libraries that are requested in the users configuration file.
Return type:: set

static merge_external_default_configs(external_default_config_paths)[source]

Merge the default configurations from external libraries into the overall default configuration.

Parameters:: external_default_config_paths (set) – A set containing the default configuration Paths for the external libraries that are requested in the users configuration file.
Returns:: The merged overall default configuration including the external library defaults.
Return type:: dict

static merge_default_configs(hyrax_defaults, external_defaults)[source]

Merge the default configurations of external libraries on top of the Hyrax default configuration.

Parameters:

hyrax_defaults (dict) – The default configuration from hyrax.
external_defaults (dict) – The default configuration from external libraries.

Returns:

The merged overall default configuration including the external library defaults.

Return type:

dict

static merge_configs(base_config: dict, overriding_config: dict) → dict[source]

Merge two config dictionaries with the overriding_config values overriding the base_config values.

Parameters:

base_config (dict) – The base configuration with keys that may be overridden by the overriding_config.
overriding_config (dict) – The new configuration values that will override the values in base_config.

Returns:

The merged configuration.

Return type:

dict

static _validate_runtime_config(runtime_config: dict, default_config: dict)[source]

Recursive helper to check that all keys in runtime_config have a default in the merged default_config.

The two arguments passed in must represent the same nesting level of the runtime config and all default config parameters respectively.

Parameters:

runtime_config (dict) – Nested config dictionary representing the runtime config.
default_config (dict) – Nested config dictionary representing the defaults

Raises:

RuntimeError – Raised if any config that exists in the runtime config does not have a default defined in default_config

static _resolve_config_paths(runtime_config: dict) → None[source]

Convert all paths in a runtime config to global paths in the current environment. Uses the hardcoded list of paths in ConfigManager.PATH_CONFIG_KEYS

This mutates the config dictionary passed.

Parameters:: runtime_config (dict) – Current runtime config nested dictionary

static resolve_runtime_config(runtime_config_filepath: pathlib.Path | str | None = None) → pathlib.Path[source]

Resolve a user-supplied runtime config to where we will actually pull config from.

If a runtime config file is specified, we will use that file.
If no file is specified and there is a file named “hyrax_config.toml” in the cwd we will use it.
If no file is specified and there is no file named “hyrax_config.toml” in the cwd we will exclusively work off the configuration defaults in the packaged “hyrax_default_config.toml” file.

Parameters:: runtime_config_filepath (Union[Path, str, None], optional) – Location of the supplied config file, by default None
Returns:: Path to the configuration file ultimately used for config resolution. When we fall back to the package supplied default config file, the Path to that file is returned.
Return type:: Path
Raises:: FileNotFoundError – If a runtime config file is specified but does not exist.

create_results_dir(config: dict, postfix: str) → pathlib.Path[source]

Creates a results directory for this run.

Postfix is the verb name of the run e.g. (infer, train, etc)

The directory is created within the results dir (set with config results_dir) and follows the pattern <timestamp>-<postfix>

The resulting directory is returned.

Parameters:

config (dict) – The full runtime configuration for this run
postfix (str) – The verb name of the run.

Returns:

The path created by this function

Return type:

Path

find_most_recent_results_dir(config: dict, verb: str) → pathlib.Path | None[source]

Find the most recent results directory corresponding to a particular verb This is a best effort search in the currently configured results root.

If result directories are created within 1 second of one another this function will return one of the directories but it is undefined which one it will return.

This function may return None indicating it could not find a directory matching the query verb

log_runtime_config(runtime_config: dict, output_path: pathlib.Path, file_name: str = 'runtime_config.toml')[source]

Log a runtime configuration.

Parameters:

runtime_config (dict) – A dictionary object containing runtime configuration values.
output_path (str) – The path to put the config file
file_name (str, Optional) – Optional name for the config file, defaults to “runtime_config.toml”