Configuration

Hyrax ships with a complete default configuration file that can be used immediately to run the software, however, to make the most of Hyrax you’ll need to modify the configuration to suit your specific needs.

Using the configuration system

When creating an instance of a Hyrax object in a notebook or running hyrax from the command line, the configuration is the primary method for specifying the parameters.

If no configuration file is specified, the default will be used. To specify a different configuration file, use the -c | --runtime-config flag from the CLI or pass the path to the configuration file when creating a Hyrax object.

from hyrax import Hyrax

# Create an instance of the Hyrax object
f = Hyrax(config_file=<path_to_config_file.toml>)

# Train the model specified in the configuration file
f.train()

>> hyrax -c <path_to_config_file.toml> train

Your first custom configuration

You could create a copy of the entire default configuration file and modify it to suit your needs, however that’s typically not required because often there are only a few parameters that must be updated for any given Hyrax action.

If a specific configuration file is provided, Hyrax will combine it with the default configuration and overwrite the default values with the specific ones.

For example, if a file called my_config.toml had the following contents:

[general]
    log_level = "debug"

It could be used to override the default log_level configuration, while leaving the rest of the configuration unchanged.

from hyrax import Hyrax

# Create an instance of the Hyrax object
f = Hyrax(config_file=my_config.toml)

# Train the model specified in the configuration file
f.train()

>> hyrax -c my_config.toml train

Updating settings in a notebook

Additionally, Hyrax supports modification of the configuration interactively in a notebook.

from hyrax import Hyrax

# Create a Hyrax instance, implicitly using the default configuration
f = Hyrax()

# Set the log level for the Hyrax instance config
f.config['general']['log_level'] = 'debug'

# Train the model specified in the configuration file
f.train()

Immutable configurations

Once Hyrax begins running an action, the configuration becomes immutable. This means that the configuration cannot be changed during the execution of an action, and attempting to do so in code will raise an exception.

By making the configuration immutable during execution, we ensure that the state of all parameters can be accurately saved with the results of the action.

About the default configuration

The default configuration for Hyrax contains safe default values for all of the settings that Hyrax uses. A portion of the default configuration file is shown below.

Note

Only the first portion of the default configuration file is shown below. The entire file can be found at the bottom of the page here: Complete default configuration file.

[general]
# Set to `true` during development to skip checking for default config values
# in external libraries. Use `false` otherwise.
dev_mode = false

# Destination of log messages. Options: 'stderr', 'stdout' specify the console,
# "path/to/hyrax.log" specifies a file.
log_destination = "stderr"

# Lowest log level to emit. Options: "critical", "error", "warning", "info", "debug".
log_level = "info"

# Directory where data is stored.
data_dir = "./data"

#Top level directory for writing results.
results_dir = "./results"


[download]
# Cut out width in arcseconds.
sw = "22asec"

# Cut out height in arcseconds.
sh = "22asec"

There is a lot of information there, but don’t worry, we’ll break it down for you.

First, the file formatted using TOML for its easy readability and because it is one of the few markdown languages that natively support comments. TOML files are organized into “tables”, and each table contains one or more key/value pairs.

For instance the [general] table (the first table in the config) contains several keys including log_level and results_dir. Each of those keys has a value associated with it. e.g. log_level = "info".

Second, every key has an associated comment describing what the key does. We attempt to keep the comments as concise as possible.

Finally, the configuration file is organized into tables that roughly correspond to the different actions that Hyrax can take. For instance, the [train] table contains parameters needed when training a model such as epochs and weights_filename. While the [infer] table contains keys such as model_weights_file.

Complete default configuration file

[general]
# Set to `true` during development to skip checking for default config values
# in external libraries. Use `false` otherwise.
dev_mode = false

# Destination of log messages. Options: 'stderr', 'stdout' specify the console,
# "path/to/hyrax.log" specifies a file.
log_destination = "stderr"

# Lowest log level to emit. Options: "critical", "error", "warning", "info", "debug".
log_level = "info"

# Directory where data is stored.
data_dir = "./data"

#Top level directory for writing results.
results_dir = "./results"


[download]
# Cut out width in arcseconds.
sw = "22asec"

# Cut out height in arcseconds.
sh = "22asec"

# The filters to download.
filter = ["HSC-G", "HSC-R", "HSC-I", "HSC-Z", "HSC-Y"]

# The type of data to download.
type = "coadd"

# The data release to download from.
rerun = "pdr3_wide"

# Path to credentials.ini file for the downloader. File contents should be:
# username = "<your username>"
# password = "<your password>"
credentials_file = "./credentials.ini"

# Alternate way to pass credentials to the downloader. Users should prefer a
# credentials.ini file to avoid exposing credentials with source control.
username = false
password = false

# The number of sources to download from the catalog. Default is -1, which
# downloads all sources in the catalog.
num_sources = -1

# The number of concurrent connections to use when downloading data.
concurrent_connections = 4

# The number of seconds between printing download statistics.
stats_print_interval = 60

# The path to the catalog file that defines which cutouts to download.
fits_file = "./catalog.fits"

# The number of seconds to wait before retrying a failed HTTP request in seconds.
retry_wait = 30

# How many times to retry a failed HTTP request before moving on to the next one.
retries = 3

# Number of seconds to wait for a full HTTP response from the server.
timeout = 3600

# The number of sky location rectangles should we request in a single request.
chunk_size = 990

# Request the image layer from the cutout service
image = true

# Request the variance layer from the cutout service
variance = false

# Request the mask layer from the cutout service
mask = false


[model]
# NOTE: All parameters are NOT used by all models. Check the model code before training.

# The name of the model to use. Option are a built-in model class name or import path
# to an external model. e.g. "HyraxAutoencoder", "user_pkg.model.ExternalModel"
name = "HyraxAutoencoder"

# The number of output channels from the first layer.
base_channel_size = 32

# The length of the latent space vector. 
latent_dim = 64

# The activation function of the final layer.
final_layer = "tanh"


[model.HyraxCNN]
# The number of classes to predict as the output of the model. i.e. 2 would be a
# binary classifer, 10 would predict the 10 classes in the CiFAR dataset.
output_classes = 10


[criterion]
# The name of the built-in criterion to use or the import path to an external criterion
name = "torch.nn.CrossEntropyLoss"

# Whether to "sum" or "mean" loss across channels. Only used by HyraxAutoencoderV2
band_loss_reduction = "mean"


[optimizer]
# The name of the built-in optimizer to use or the import path to an external optimizer
name = "torch.optim.SGD"


["torch.optim.SGD"]
# learning rate for torch.optim.SGD optimizer.
lr = 0.01

# momentum for torch.optim.SGD optimizer.
momentum = 0.9

["torch.optim.Adam"]
# learning rate for torch.optim.SGD optimizer.
lr = 0.01


[train]
# The name of the file were the model weights will be saved after training.
weights_filename = "example_model.pth"

#The number of epochs to train for.
epochs = 10

# If resuming from a check point, set to the path of the checkpoint file.
# Otherwise set to `false` to start training from the beginning.
resume = false

# The data_set split to use when training a model.
split = "train"

# The name of the experiment when logging training results to mlflow
experiment_name = "notebook"

# The name of the run when logging training results to mlflow.
# If false, uses result directory string, <timestamp>-train-<uid>, as run name.
run_name = false

[onnx]

# The operator set version to use when exporting a model. See the following for info:
# https://onnxruntime.ai/docs/reference/compatibility.html#onnx-opset-support
opset_version = 20


[model_inputs]
# Note - the sub-table (model_inputs.<sub-table>) can be named anything you like.
[model_inputs.data]
# Name of the dataset class to use, or the import path to an external dataset class.
# e.g. "HyraxCifarDataSet", "user_pkg.data_set.ExternalDataset".
dataset_class = "HyraxCifarDataSet"

# Location (directory) where the data is stored.
data_location = "./data"

# The data field that represents a unique ID for each sample.
primary_id_field = "object_id"


[data_set]
# Warning - using data_set.name to define the dataset class will be deprecated.
# Please use [model_inputs.<friendly_name>.dataset_class] instead. See usage above.
name = "HyraxCifarDataSet"

# Crop pixel dimensions for images, e.g., [100, 100]. If false, scans for the
# smallest image size in [general].data_dir and uses it.
crop_to = false

# Used by HSCDataSet, LSSTDataset, and DownloadedLSSTDataset. 
# Limit to only particular filters. When `false`, use all filters. 
# Options: ["HSC-G", "HSC-R", "HSC-I", "HSC-Z", "HSC-Y"] for HSC
# Options: ["u", "g", "r", "i", "z" , "y"] for LSST
filters = false

# Path to a fits file that specifies object IDs to use from the data stored in
# [general].data_dir. Implementation is data_set class dependent. Use `false` for no filtering.
filter_catalog = false

# The transformation to be applied to images before being passed on to the model
# This must be a valid Numpy function. Passing false will result in no transformations
# (other than cropping) be applied to the images.  
transform = "tanh"

# train_size, validation_size, and test_size use these conventions:
# * A `float` between `0.0` and `1.0` is the proportion of the dataset to include in the split.
# * An `int`, represents the absolute number of samples in the particular split.
# * It is an error for these values to add to more than 1.0 as ratios or the size
#   of the dataset if expressed as integers.

# Size of the train split. If `false`, the value is automatically set to the
# complement of test_size plus validate_size (if any).
train_size = 0.6

# Size of the validation split. If `false`, and both train_size and test_size
# are defined, the value is set to the complement of the other two sizes summed.
# If `false`, and only one of the other sizes is defined, no validate split is created.
validate_size = 0.2

# Size of the test split. If `false`, the value is set to the complement of train_size plus
# the validate_size (if any). If `false` and `train_size = false`, test_size is set to `0.25`.
test_size = 0.2

# Number to seed with for generating a random split. Use `false` to seed from a
# system source at runtime.
seed = false

# If `true`, cache samples in memory during training to reduce runtime after the
# first epoch. Set to `false` when running inference or on memory-constrained systems.
use_cache = true

# If `true`, preload the in memory cache using many worker threads when the dataset is constructed 
# to reduce the effect of filesystem latency on first epoch runtime. 
# Warning: Only suitable for situations where the entire dataset fits in system memory
preload_cache = true

# Override the name of the object_id column for FitsImageDataset, HSCDataset and DownloadedLSSTDataset
object_id_column_name = false

# Override the name of the filter column for FitsImageDataset and HSCDataset
filter_column_name = false

# Override the name of the filename column for FitsImageDataset and HSCDataset
filename_column_name = false

# Replace NaN in input data with a value, modes are false for no replacement or "quantile" to replace with a 
# defined quantile of the non-NaN data, see nan_quantile.
nan_mode = false

# When replacing NaN values with a quantile, which quantile in the non-nan tensor should be used.
nan_quantile = 0.05

# The astropy table to use as a catalog in LSSTDataSet and friends
astropy_table = false

# Semi width in degrees of cutouts made from the butler (17 arcsec)
semi_width_deg = 0.00472

# Semi height in degrees of cutouts made from the butler (17 arcsec)
semi_height_deg = 0.00472



[data_set.HyraxRandomDataset]
# Total number of samples produced by the random dataset
size = 100

# The dimensions of the numpy arrays that will be produced for each sample represented
# as a list where each element is the size of dimension.
shape = [2,5,5]

# Seed to use for random number generation
seed = 42

# If a list is provided, the data will have randomly labeled with values from the list
# If set to false, no labels will be included with the data.
provided_labels = [0, 1, 2]

# Set this to a positive integer to randomly replace some values with an "invalid" value.
number_invalid_values = 0

# The value to use for invalid values in the data. Must be one of the following:
# "nan", "inf", "-inf", "none" or a float value.
invalid_value_type = "nan"


[data_loader]
# The number of data points to load at once.
batch_size = 512

# STRONG RECOMMENDATION: Leave this as `false`.
# Ensure that the data loader does no secondary shuffling of the data.
shuffle = false


[infer]
# The path to the model weights file to use for inference.
model_weights_file = false

# The data_set split to use for inference. Use `false` for entire dataset.
split = false


[vector_db]
# The type of vector db to use. Use "false" to disable vector database.
name = "chromadb"

# The directory where the vector database will be stored. Use "false" to create
# a new vector database in a timestamped directory. Otherwise set to a path.
vector_db_dir = false

# The path to inference results. Setting to "false" will use the most recent
# inference results.
infer_results_dir = false


[vector_db.chromadb]
# The approximate maximum size of a shard before creating a new one. A smaller
# value will decrease insert times while increasing search times.
shard_size_limit = 65536

# Inserting vectors with more than this many elements logs a warning message. ChromaDB
# performance degrades with vectors of this size. Set to "false" to disable warning.
vector_size_warning = 10000


[vector_db.qdrant]
# The number of elements in the vectors that will be stored in the vector database.
# This must be the same as the size of the vectors produced by the model.
vector_size = 64


[results]
# Path to inference results to use for visualization and lookups. Uses latest inference run if none provided.
inference_dir = false


[umap]
# Number of data points used to fit the umap transform.
fit_sample_size = 1024

# Save the fitted umap as a pickle file 
save_fit_umap = true

# Use multiprocessing during transforming to umap space (More memory intensive)
parallel = false

# Name of the umap implementation to use
name = "umap.UMAP"


[umap.UMAP]
# Specify any parameter accepted by https://umap-learn.readthedocs.io/en/latest/api.html#umap
# Dimension of the embedded space
n_components = 2

# Controls how UMAP balances local versus global structure in the data.
# See official documentation for details.
n_neighbors = 15

[visualize]

# List of metadata field names to use in visualizer. Must be available as metadata in your dataset
fields = []

# Whether to display a panel of randomly chosen images corresponding to the selected points
display_images = false

# Name of catalog column to use for coloring points in the scatter plot. Use false for no coloring.
color_column = false

# Colormap to use for coloring points in the scatter plot when color_column is specified
cmap = "viridis"

# Only valid for .pt tensor images. Which bands should be loaded for display
# [0,3,5] would map bands in that order to R,G,B. Single band will be grayscale.
torch_tensor_bands = [3]

# Whether to rasterize plot. Will break coloring (Haloviews Bug)
# Helpful to reduce lag in large datasets. 
rasterize_plot = false