ncalab ====== .. py:module:: ncalab Submodules ---------- .. toctree:: :maxdepth: 1 /autoapi/ncalab/autostepper/index /autoapi/ncalab/data/index /autoapi/ncalab/export/index /autoapi/ncalab/losses/index /autoapi/ncalab/models/index /autoapi/ncalab/paths/index /autoapi/ncalab/prediction/index /autoapi/ncalab/search/index /autoapi/ncalab/training/index /autoapi/ncalab/uncertainty/index /autoapi/ncalab/utils/index /autoapi/ncalab/visualization/index Attributes ---------- .. autoapisummary:: ncalab.__CURRENT_PATH ncalab.ROOT_PATH ncalab.WEIGHTS_PATH ncalab.TASK_PATH ncalab.DEFAULT_RANDOM_SEED ncalab.animator_style_dark ncalab.animator_styles Classes ------- .. autoapisummary:: ncalab.AutoStepper ncalab.GrowingNCADataset ncalab.DiceScore ncalab.DiceLoss ncalab.DiceBCELoss ncalab.FocalLoss ncalab.AbstractNCAModel ncalab.AbstractNCAHead ncalab.BasicNCAPerception ncalab.AbstractNCARule ncalab.MLPNCARule ncalab.CascadeNCA ncalab.ClassificationNCAModel ncalab.ClassificationNCAHead ncalab.DepthNCAModel ncalab.GrowingNCAModel ncalab.SegmentationNCAModel ncalab.Prediction ncalab.BasicNCATrainer ncalab.ParameterSet ncalab.ParameterSearch ncalab.Pool ncalab.BasicNCATrainer ncalab.TrainingHistory ncalab.EarlyStopping ncalab.TrainValRecord ncalab.SplitDefinition ncalab.KFoldCrossValidationTrainer ncalab.AbstractNCAModel ncalab.Prediction ncalab.UncertaintyEstimator ncalab.NQM ncalab.MCMC ncalab.AnimatorStyle ncalab.Animator ncalab.Prediction ncalab.Visual ncalab.VisualBinaryImageClassification ncalab.VisualRGBImageClassification ncalab.VisualMultiImageClassification ncalab.VisualBinaryImageSegmentation ncalab.VisualDepthEstimation ncalab.VisualGrowing Functions --------- .. autoapisummary:: ncalab.get_compute_device ncalab.pad_input ncalab.print_NCALab_banner ncalab.print_mascot ncalab.fix_random_seed ncalab.unwrap ncalab.intepret_range_parameter ncalab.draw_segmentation_overlay ncalab.unwrap ncalab.abbreviate_label ncalab.show_image_row Package Contents ---------------- .. py:class:: AutoStepper(min_steps: int = 10, max_steps: int = 100, plateau: int = 5, verbose: bool = False, threshold: float = 0.01) Helps selecting number of timesteps based on NCA activity. :param min_steps: Minimum number of timesteps to always execute, defaults to 10. :type min_steps: int, optional :param max_steps: Terminate after maximum number of steps, defaults to 100. :type max_steps: int, optional :param plateau: Number of steps that is considered a plateau, defaults to 5. :type plateau: int :param verbose: Whether to log interruption to stdout, defaults to False. :type verbose: bool :param threshold: Score threshold, defaults to 1e-2. :type threshold: float .. py:attribute:: min_steps :value: 10 .. py:attribute:: max_steps :value: 100 .. py:attribute:: plateau :value: 5 .. py:attribute:: verbose :value: False .. py:attribute:: threshold :value: 0.01 .. py:attribute:: cooldown :value: 0 .. py:attribute:: hidden_i :type: torch.Tensor | None :value: None .. py:attribute:: hidden_i_1 :type: torch.Tensor | None :value: None .. py:method:: _score() -> torch.Tensor Calculates activity score. Method check() uses this score to determine if the NCA is inactive. :return: Activity score estimate. :rtype: torch.Tensor .. py:method:: _check(step: int) -> bool Checks whether to interrupt inference after the current step. :param step: Current NCA inference step. :type step: int :return: Whether to interrupt inference after the current step. :rtype: bool .. py:method:: run(nca: ncalab.models.AbstractNCAModel, x) .. py:method:: __call__(*args: Any, **kwargs: Any) -> Any .. py:class:: GrowingNCADataset(image: numpy.ndarray, num_channels: int, batch_size: int = 8) Bases: :py:obj:`torch.utils.data.Dataset` An abstract class representing a :class:`Dataset`. All datasets that represent a map from keys to data samples should subclass it. All subclasses should overwrite :meth:`__getitem__`, supporting fetching a data sample for a given key. Subclasses could also optionally overwrite :meth:`__len__`, which is expected to return the size of the dataset by many :class:`~torch.utils.data.Sampler` implementations and the default options of :class:`~torch.utils.data.DataLoader`. Subclasses could also optionally implement :meth:`__getitems__`, for speedup batched samples loading. This method accepts list of indices of samples of batch and returns list of samples. .. note:: :class:`~torch.utils.data.DataLoader` by default constructs an index sampler that yields integral indices. To make it work with a map-style dataset with non-integral indices/keys, a custom sampler must be provided. Dedicated dataset for "growing" tasks, like growing emoji. The idea is to train a model solely for the purpose to generate ("grow") a fixed image. Hence, this Dataset class only stores multiple copies of the same image. :param image [np.ndarray]: Input image. :param num_channels [int]: Total number of image channels (including hidden) :param batch_size [int]: Output batch size. Defaults to 8. .. py:attribute:: batch_size :value: 8 .. py:attribute:: image .. py:attribute:: seed .. py:method:: __len__() .. py:method:: __getitem__(idx) .. py:class:: DiceScore Bases: :py:obj:`torch.nn.Module` Pytorch Module that computes the Dice overlap score between two images. Initialize internal Module state, shared by both nn.Module and ScriptModule. .. py:method:: forward(x: torch.Tensor, y: torch.Tensor, smooth: float = 1.0) -> torch.Tensor :param x: Reference Input :type x: torch.Tensor :param y: Other Input :type y: torch.Tensor :param smooth: Smooting factor, defaults to 1.0 :type smooth: float :returns: Dice score :rtype: torch.Tensor .. py:class:: DiceLoss Bases: :py:obj:`torch.nn.Module` Base class for all neural network modules. Your models should also subclass this class. Modules can also contain other Modules, allowing them to be nested in a tree structure. You can assign the submodules as regular attributes:: import torch.nn as nn import torch.nn.functional as F class Model(nn.Module): def __init__(self) -> None: super().__init__() self.conv1 = nn.Conv2d(1, 20, 5) self.conv2 = nn.Conv2d(20, 20, 5) def forward(self, x): x = F.relu(self.conv1(x)) return F.relu(self.conv2(x)) Submodules assigned in this way will be registered, and will also have their parameters converted when you call :meth:`to`, etc. .. note:: As per the example above, an ``__init__()`` call to the parent class must be made before assignment on the child. :ivar training: Boolean represents whether this module is in training or evaluation mode. :vartype training: bool Initialize internal Module state, shared by both nn.Module and ScriptModule. .. py:attribute:: dicescore .. py:method:: forward(x: torch.Tensor, y: torch.Tensor, smooth: float = 1.0) -> torch.Tensor .. py:class:: DiceBCELoss Bases: :py:obj:`torch.nn.Module` Combination of Dice and BCE Loss between two images. Initialize internal Module state, shared by both nn.Module and ScriptModule. .. py:attribute:: dicescore .. py:method:: forward(x: torch.Tensor, y: torch.Tensor, smooth: float = 1.0) -> torch.Tensor :param x: Reference Input :type x: torch.Tensor :param y: Other Input :type y: torch.Tensor :param smooth: Smooting factor, defaults to 1.0 :type smooth: float :returns: Dice score :rtype: torch.Tensor .. py:class:: FocalLoss(weight=None, gamma=2, device='cpu') Bases: :py:obj:`torch.nn.modules.loss._WeightedLoss` Base class for all neural network modules. Your models should also subclass this class. Modules can also contain other Modules, allowing them to be nested in a tree structure. You can assign the submodules as regular attributes:: import torch.nn as nn import torch.nn.functional as F class Model(nn.Module): def __init__(self) -> None: super().__init__() self.conv1 = nn.Conv2d(1, 20, 5) self.conv2 = nn.Conv2d(20, 20, 5) def forward(self, x): x = F.relu(self.conv1(x)) return F.relu(self.conv2(x)) Submodules assigned in this way will be registered, and will also have their parameters converted when you call :meth:`to`, etc. .. note:: As per the example above, an ``__init__()`` call to the parent class must be made before assignment on the child. :ivar training: Boolean represents whether this module is in training or evaluation mode. :vartype training: bool Initialize internal Module state, shared by both nn.Module and ScriptModule. .. py:attribute:: gamma :value: 2 .. py:attribute:: weight :value: None .. py:attribute:: device :value: 'cpu' .. py:attribute:: ce_loss .. py:method:: forward(_input, _target) .. py:class:: AbstractNCAModel(device: torch.device, num_image_channels: int, num_hidden_channels: int, num_output_channels: int, plot_function: Optional[ncalab.visualization.Visual] = None, validation_metric: Optional[str] = None, fire_rate: float = 0.5, hidden_size: int = 128, use_alive_mask: bool = False, immutable_image_channels: bool = True, num_learned_filters: int = 0, filter_padding: Literal['zero', 'reflect', 'replicate', 'circular'] = 'reflect', use_laplace: bool = False, kernel_size: int = 3, pad_noise: bool = False, use_temporal_encoding: bool = False, rule_type: type[ncalab.models.basicNCA.abstractNCArule.AbstractNCARule] = MLPNCARule, rule_args=None, training_timesteps: int | Tuple[int, int] = 100, inference_timesteps: int | Tuple[int, int] = 100) Bases: :py:obj:`torch.nn.Module`, :py:obj:`abc.ABC` Abstract base class for NCA models. BasicNCAModel is a composition of an NCA backbone model (called "rule"), and an (optional) head module for downstream tasks. :param device: Pytorch device descriptor. :param num_image_channels: Number of channels reserved for input image. :param num_hidden_channels: Number of hidden channels (communication channels). :param num_output_channels: Number of output channels. :param validation_metric: :param fire_rate: Fire rate for stochastic weight update. Defaults to 0.5. :param hidden_size: Number of neurons in hidden layer. Defaults to 128. :param use_alive_mask: Whether to use alive masking (channel 3) during training. Defaults to False. :param immutable_image_channels: If image channels should be fixed during inference, which is the case for most segmentation or classification problems. Defaults to True. :param num_learned_filters: Number of learned filters. If zero, use two sobel filters instead. Defaults to 2. :param filter_padding: Padding type to use. Might affect reliance on spatial cues. Defaults to "circular". :param use_laplace: Whether to use Laplace filter (only if num_learned_filters == 0) :param kernel_size: Filter kernel size (only for learned filters) :param pad_noise: Whether to pad input image tensor with noise in hidden / output channels :param use_temporal_encoding: :param rule_type: :param rule_args: :param training_timesteps: :param inference_timesteps: .. py:attribute:: device .. py:attribute:: num_image_channels .. py:attribute:: num_hidden_channels .. py:attribute:: num_output_channels .. py:attribute:: num_channels .. py:attribute:: fire_rate :value: 0.5 .. py:attribute:: hidden_size :value: 128 .. py:attribute:: use_alive_mask :value: False .. py:attribute:: immutable_image_channels :value: True .. py:attribute:: num_learned_filters :value: 0 .. py:attribute:: use_laplace :value: False .. py:attribute:: kernel_size :value: 3 .. py:attribute:: filter_padding :value: 'reflect' .. py:attribute:: pad_noise :value: False .. py:attribute:: use_temporal_encoding :value: False .. py:attribute:: plot_function :value: None .. py:attribute:: validation_metric :value: None .. py:attribute:: training_timesteps :value: 100 .. py:attribute:: inference_timesteps :value: 100 .. py:attribute:: perception .. py:attribute:: input_vector_size .. py:attribute:: rule_type .. py:attribute:: rule_args :value: None .. py:attribute:: rule .. py:attribute:: head :type: ncalab.models.basicNCA.abstractNCAhead.AbstractNCAHead | None :value: None .. py:attribute:: metrics :type: Dict[str, torchmetrics.Metric] .. py:method:: _define_rule() -> ncalab.models.basicNCA.abstractNCArule.AbstractNCARule .. py:method:: prepare_input(x: torch.Tensor) -> torch.Tensor Preprocess input. Intended to be overwritten by subclass, if preprocessing is necessary. :param x [torch.Tensor]: Input tensor to preprocess. :returns: Processed tensor. .. py:method:: _alive(x) .. py:method:: _update(x: torch.Tensor, step: int) -> torch.Tensor Compute residual cell update. :param x [torch.Tensor]: Input tensor, BCWH :param step [int]: Current timestep, required for computing temporal encoding. :returns: Residual cell update, BCWH. .. py:method:: _forward_step(x: torch.Tensor, step: int) .. py:method:: forward(x: torch.Tensor, steps: int = 1) -> ncalab.prediction.Prediction :param x [torch.Tensor]: Input image, padded along the channel dimension, BCWH. :param steps [int]: Time steps in forward pass. :returns [Prediction]: Prediction object. .. py:method:: _post_forward_step(x: torch.Tensor) -> torch.Tensor .. py:method:: loss(pred: ncalab.prediction.Prediction, label: torch.Tensor) -> Dict[str, torch.Tensor] Compute loss. Needs to be overloaded by any subclass. Please note that the returned dict needs to hold "total" key in which the total loss is stored, which is typically a weighted sum of other losses. The total loss is backpropagated, whereas the other losses are sent to tensorboard. :param image [torch.Tensor]: Input image, BCWH. :param label [torch.Tensor]: Ground truth, BCWH. :returns: Dictionary of identifiers mapped to computed losses. .. py:method:: finetune(freeze_head: bool = False) Prepare model for fine tuning by freezing everything except the final layer, and setting to "train" mode. :param: freeze_head .. py:method:: predict(image: torch.Tensor, steps: Optional[int | Tuple[int, int]] = None) -> ncalab.prediction.Prediction Make an NCA prediction, performing multiple forward passes to yield a final result. :param image: Input image, BCWH. :type image: torch.Tensor :param steps: Time steps :type steps: Optional[int] :returns: Prediction object. :rtype: Prediction .. py:method:: record(image: torch.Tensor, steps: Optional[int | Tuple[int, int]] = None) -> List[ncalab.prediction.Prediction] Record predictions for all time steps and return the resulting sequence of predictions. :param image: Input image, BCWH. :type image: torch.Tensor :returns: List of Prediction objects. :rtype: List[Prediction] .. py:method:: validate(dataloader: torch.utils.data.DataLoader, steps: Optional[int] = None) -> Tuple[Dict[str, float], List[ncalab.prediction.Prediction]] Make a prediction on an image of the validation set and return metrics computed with respect to a labelled validation image. :param dataloader [torch.utils.data.DataLoader]: Dataloader for validation images :param steps [int]: Inference steps :returns [Tuple[float, List[Prediction]]]: Validation metric, predicted image BCWH .. py:method:: _to_dict() -> Dict[str, Any] .. py:method:: to_dict() -> Dict[str, Any] .. py:method:: from_dict(d: Dict[str, Any]) :classmethod: .. py:method:: num_trainable_parameters() -> int Returns the number of trainable model parameters. :return: Number of trainable parameters. :rtype: int .. py:method:: save(path: str | os.PathLike) .. py:method:: load(model: AbstractNCAModel, path: str | os.PathLike) -> AbstractNCAModel :staticmethod: .. py:method:: post_prediction(prediction: ncalab.prediction.Prediction) -> ncalab.prediction.Prediction .. py:class:: AbstractNCAHead Bases: :py:obj:`torch.nn.Module`, :py:obj:`abc.ABC` Base class for all neural network modules. Your models should also subclass this class. Modules can also contain other Modules, allowing them to be nested in a tree structure. You can assign the submodules as regular attributes:: import torch.nn as nn import torch.nn.functional as F class Model(nn.Module): def __init__(self) -> None: super().__init__() self.conv1 = nn.Conv2d(1, 20, 5) self.conv2 = nn.Conv2d(20, 20, 5) def forward(self, x): x = F.relu(self.conv1(x)) return F.relu(self.conv2(x)) Submodules assigned in this way will be registered, and will also have their parameters converted when you call :meth:`to`, etc. .. note:: As per the example above, an ``__init__()`` call to the parent class must be made before assignment on the child. :ivar training: Boolean represents whether this module is in training or evaluation mode. :vartype training: bool Initialize internal Module state, shared by both nn.Module and ScriptModule. .. py:attribute:: optimizer :value: None .. py:method:: forward(x: torch.Tensor) -> torch.Tensor :param x: Input tensor :type x: torch.Tensor :returns: NotImplemented, subclasses are required to implement this method. .. py:method:: freeze(freeze_last: bool = True) -> None :abstractmethod: Freeze head weights. :param freeze_last: Whether to freeze the last layer (if applicable), defaults to True :type freeze_last: bool, optional :returns: NotImplemented, subclasses are required to implement this method. .. py:class:: BasicNCAPerception(nca: ncalab.models.basicNCA.AbstractNCAModel) .. py:attribute:: nca .. py:method:: _define_filters() Define list of perception filters, based on parameters passed in constructor. :param num_learned_filters [int]: Number of learned filters in perception filter bank. .. py:method:: perceive(x: torch.Tensor, step: int) -> torch.Tensor .. py:method:: freeze() .. py:class:: AbstractNCARule(device: torch.device, input_size: int, hidden_size: int, output_size: int) Bases: :py:obj:`torch.nn.Module`, :py:obj:`abc.ABC` Base class for all neural network modules. Your models should also subclass this class. Modules can also contain other Modules, allowing them to be nested in a tree structure. You can assign the submodules as regular attributes:: import torch.nn as nn import torch.nn.functional as F class Model(nn.Module): def __init__(self) -> None: super().__init__() self.conv1 = nn.Conv2d(1, 20, 5) self.conv2 = nn.Conv2d(20, 20, 5) def forward(self, x): x = F.relu(self.conv1(x)) return F.relu(self.conv2(x)) Submodules assigned in this way will be registered, and will also have their parameters converted when you call :meth:`to`, etc. .. note:: As per the example above, an ``__init__()`` call to the parent class must be made before assignment on the child. :ivar training: Boolean represents whether this module is in training or evaluation mode. :vartype training: bool Initialize internal Module state, shared by both nn.Module and ScriptModule. .. py:attribute:: device .. py:attribute:: input_size .. py:attribute:: hidden_size .. py:attribute:: output_size .. py:method:: freeze(freeze_last: bool = False) -> None :abstractmethod: .. py:class:: MLPNCARule(device: torch.device, input_size: int, hidden_size: int, output_size: int, nonlinearity: type[torch.nn.Module] = nn.ReLU) Bases: :py:obj:`ncalab.models.basicNCA.abstractNCArule.AbstractNCARule` NCA rule module based on a two-layer Multi-Layer-Perceptron (MLP). :param nn: _description_ :type nn: _type_ :param device: Compute device :type device: torch.device :param input_size: Input neurons :type input_size: int :param hidden_size: Hidden neurons :type hidden_size: int :param output_size: Output neurons :type output_size: int :param nonlinearity: Activation function, defaults to nn.ReLU :type nonlinearity: type[nn.Module], optional .. py:attribute:: nonlinearity .. py:method:: _build_network() .. py:method:: _initialize_network() Initialize network weights of the MLP. We assume that the default initialization of the first layer is good enough. Since the final layer is purely linear and unbiased, we initalize with 0. .. py:method:: forward(x: torch.Tensor) -> torch.Tensor :param x: BCWH perception vector :type x: torch.Tensor :return: BCWH residual update :rtype: torch.Tensor .. py:method:: freeze(freeze_last: bool = False) Freeze the first layer of the NCA rule network and, optionally, the final layer. :param freeze_last: _description_, defaults to False :type freeze_last: bool, optional .. py:class:: CascadeNCA(wrapped: ncalab.models.basicNCA.AbstractNCAModel, scales: List[int], steps: List[int], single_model: bool = True) Bases: :py:obj:`ncalab.models.basicNCA.AbstractNCAModel` Chain multiple instances of the same NCA model, operating at different image scales. The idea is to use this model as a wrapper and drop-in replacement for an existing model. For instance, if we created a model `nca = SegmentationNCA(...)` and all code to interface with it, we could instead write `cascade = CascadeNCA(SegmentationNCA(...), scales, steps)` without the need for adjusting any of the interfacing code. This is still highly experimental. In the future, we'll work on a cleaner interface for this. :param wrapped: Backbone model based on AbstractNCAModel. :type wrapped: ncalab.AbstractNCAModel :param scales: List of scales to operate at, e.g. [4, 2, 1]. :type scales: List[int] :param steps: List of number of NCA inference time steps. :type steps: List[int] :param single_model: Only train a single instance of the NCA model :type single_model: bool .. py:attribute:: loss Compute loss. Needs to be overloaded by any subclass. Please note that the returned dict needs to hold "total" key in which the total loss is stored, which is typically a weighted sum of other losses. The total loss is backpropagated, whereas the other losses are sent to tensorboard. :param image [torch.Tensor]: Input image, BCWH. :param label [torch.Tensor]: Ground truth, BCWH. :returns: Dictionary of identifiers mapped to computed losses. .. py:attribute:: finetune Prepare model for fine tuning by freezing everything except the final layer, and setting to "train" mode. :param: freeze_head .. py:attribute:: prepare_input Preprocess input. Intended to be overwritten by subclass, if preprocessing is necessary. :param x [torch.Tensor]: Input tensor to preprocess. :returns: Processed tensor. .. py:attribute:: head .. py:attribute:: metrics .. py:attribute:: wrapped .. py:attribute:: scales .. py:attribute:: steps .. py:attribute:: single_model :value: True .. py:attribute:: models :type: List[ncalab.models.basicNCA.AbstractNCAModel] .. py:method:: forward(x: torch.Tensor, *args, **kwargs) -> ncalab.prediction.Prediction :param x: Input image tensor, BCWH. :type x: torch.Tensor :param steps: Unused, as steps are defined in constructor. :type steps: torch.Tensor :return: Prediction object :rtype: Prediction .. py:method:: record(image: torch.Tensor, steps: Optional[int | Tuple[int, int]] = None) -> List[ncalab.prediction.Prediction] Records predictions for all time steps and returns the resulting sequence of predictions. Takes care of scaling the image in between steps. :param image: Input image, BCWH. :type image: torch.Tensor :returns: List of Prediction objects. :rtype: List[Prediction] .. py:class:: ClassificationNCAModel(device: torch.device, num_image_channels: int, num_hidden_channels: int, num_classes: int, fire_rate: float = 0.8, hidden_size: int = 128, use_alive_mask: bool = False, pixel_wise_loss: bool = False, num_learned_filters: int = 2, filter_padding: Literal['zero', 'reflect', 'replicate', 'circular'] = 'reflect', use_laplace: bool = False, kernel_size: int = 3, pad_noise: bool = False, use_temporal_encoding: bool = False, use_classifier: bool = True, class_names: Optional[List[str]] = None, avg_pool_size: int = 8, lambda_hidden: float = 0, **kwargs) Bases: :py:obj:`ncalab.models.basicNCA.AbstractNCAModel` Abstract base class for NCA models. BasicNCAModel is a composition of an NCA backbone model (called "rule"), and an (optional) head module for downstream tasks. :param device: Pytorch device descriptor. :param num_image_channels: _description_ :param num_hidden_channels: _description_ :param num_classes: _description_ :param fire_rate: Fire rate for stochastic weight update. Defaults to 0.8. :param hidden_size: Number of neurons in hidden layer. Defaults to 128. :param use_alive_mask: Whether to use alive masking (channel 3) during training. Defaults to False. :param pixel_wise_loss: Whether a prediction per pixel is desired, like in self-classifying MNIST. Defaults to False. :param num_learned_filters: Number of learned filters. If zero, use two sobel filters instead. Defaults to 2. :param filter_padding: Padding type to use. Might affect reliance on spatial cues. Defaults to "circular". :param pad_noise: Whether to pad input image tensor with noise in hidden / output channels .. py:attribute:: _num_classes .. py:attribute:: pixel_wise_loss :value: False .. py:attribute:: use_classifier :value: True .. py:attribute:: avg_pool_size :value: 8 .. py:attribute:: lambda_hidden :value: 0 .. py:attribute:: metrics .. py:attribute:: focal_loss .. py:property:: num_classes :type: int .. py:method:: classify(image: torch.Tensor, steps: int = 100, reduce: bool = False) -> torch.Tensor Predict classification for an input image. :param image: Input image. :param steps: Inference steps. Defaults to 100. :param reduce: Return a single softmax probability. Defaults to False. :returns: Single class index or vector of logits. .. py:method:: loss(pred: ncalab.prediction.Prediction, label: torch.Tensor) -> Dict[str, torch.Tensor] Return the classification loss. :param pred: Prediction. :param label: Ground truth. :returns: Dictionary of identifiers mapped to computed losses. .. py:method:: post_prediction(prediction: ncalab.prediction.Prediction) -> ncalab.prediction.Prediction .. py:class:: ClassificationNCAHead(num_hidden_channels: int, num_classes: int, device: torch.device, avg_pool_size: int, hidden_size: int = 32) Bases: :py:obj:`ncalab.models.basicNCA.abstractNCAhead.AbstractNCAHead` Base class for all neural network modules. Your models should also subclass this class. Modules can also contain other Modules, allowing them to be nested in a tree structure. You can assign the submodules as regular attributes:: import torch.nn as nn import torch.nn.functional as F class Model(nn.Module): def __init__(self) -> None: super().__init__() self.conv1 = nn.Conv2d(1, 20, 5) self.conv2 = nn.Conv2d(20, 20, 5) def forward(self, x): x = F.relu(self.conv1(x)) return F.relu(self.conv2(x)) Submodules assigned in this way will be registered, and will also have their parameters converted when you call :meth:`to`, etc. .. note:: As per the example above, an ``__init__()`` call to the parent class must be made before assignment on the child. :ivar training: Boolean represents whether this module is in training or evaluation mode. :vartype training: bool Initialize internal Module state, shared by both nn.Module and ScriptModule. .. py:attribute:: num_hidden_channels .. py:attribute:: num_classes .. py:attribute:: device .. py:attribute:: avg_pool_size .. py:attribute:: classifier .. py:method:: forward(x) :param x: Input tensor :type x: torch.Tensor :returns: NotImplemented, subclasses are required to implement this method. .. py:method:: freeze(freeze_last: bool = False) Freeze head weights. :param freeze_last: Whether to freeze the last layer (if applicable), defaults to True :type freeze_last: bool, optional :returns: NotImplemented, subclasses are required to implement this method. .. py:class:: DepthNCAModel(device: torch.device, num_image_channels: int = 3, num_hidden_channels: int = 18, fire_rate: float = 0.8, hidden_size: int = 128, num_learned_filters: int = 2, pad_noise: bool = False, **kwargs) Bases: :py:obj:`ncalab.models.basicNCA.AbstractNCAModel` NCA model for monocular depth estimation. :param device: Pytorch device descriptor. :param num_image_channels: Number of channels reserved for input image. :param num_hidden_channels: Number of hidden channels (communication channels). :param num_output_channels: Number of output channels. :param validation_metric: :param fire_rate: Fire rate for stochastic weight update. Defaults to 0.5. :param hidden_size: Number of neurons in hidden layer. Defaults to 128. :param use_alive_mask: Whether to use alive masking (channel 3) during training. Defaults to False. :param immutable_image_channels: If image channels should be fixed during inference, which is the case for most segmentation or classification problems. Defaults to True. :param num_learned_filters: Number of learned filters. If zero, use two sobel filters instead. Defaults to 2. :param filter_padding: Padding type to use. Might affect reliance on spatial cues. Defaults to "circular". :param use_laplace: Whether to use Laplace filter (only if num_learned_filters == 0) :param kernel_size: Filter kernel size (only for learned filters) :param pad_noise: Whether to pad input image tensor with noise in hidden / output channels :param use_temporal_encoding: :param rule_type: :param rule_args: :param training_timesteps: :param inference_timesteps: .. py:attribute:: vignette :value: None .. py:method:: loss(pred: ncalab.prediction.Prediction, label: torch.Tensor) -> Dict[str, torch.Tensor] :param image: Input image, BCWH. :param label: Ground truth. :returns: Dictionary of identifiers mapped to computed losses. .. py:class:: GrowingNCAModel(device: torch.device, num_image_channels: int = 4, num_hidden_channels: int = 16, fire_rate: float = 0.5, hidden_size: int = 128, use_alive_mask: bool = False, lambda_hidden: float = 0.0, **kwargs) Bases: :py:obj:`ncalab.models.basicNCA.AbstractNCAModel` NCA Model class for "growing" tasks, in which a structure is grown from a single seed pixel. This specialization of the BasicNCAModel has some interesting properties. For instance, it has no output channels, as the growing task directly manipulates the input image channels. :param device [torch.device]: Pytorch device descriptor. :param num_image_channels [int]: Number of channels reserved for input image. Defaults to 4. :param num_hidden_channels [int]: Number of hidden channels (communication channels). Defaults to 16. :param fire_rate [float]: Stochastic weight update. Defaults to 0.5. :param hidden_size [int]: Default number of nodes in hidden layer. Defaults to 128. :param use_alive_mask [bool]: Whether to use alive masking. Defaults to False. .. py:attribute:: lambda_hidden :value: 0.0 .. py:method:: loss(pred: ncalab.prediction.Prediction, label: torch.Tensor) -> Dict[str, torch.Tensor] Implements a simple MSE loss between target and prediction. :param pred: Prediction :param label: Target :returns [Tensor]: MSE Loss .. py:method:: make_seed(width: int, height: int) -> torch.Tensor .. py:method:: grow(seed: torch.Tensor, steps: int = 100) -> List[numpy.ndarray] Run the growth process and return the resulting output sequence. :param seed [torch.Tensor]: Seed image, can be generated through make_seed. :param steps [int]: Number of inference steps. Defaults to 100. :returns [List[np.ndarray]]: Sequence of output images. .. py:class:: SegmentationNCAModel(device: torch.device, num_image_channels: int = 3, num_hidden_channels: int = 16, num_classes: int = 1, fire_rate: float = 0.8, hidden_size: int = 128, num_learned_filters: int = 2, pad_noise: bool = False, filter_padding: Literal['zero', 'reflect', 'replicate', 'circular'] = 'circular', lambda_hidden: float = 0.001, **kwargs) Bases: :py:obj:`ncalab.models.basicNCA.AbstractNCAModel` Model used for image segmentation. Uses Dice score as the default validation metric. Currently, only binary segmentation masks are supported. :param device [torch.device]: Compute device. :param num_image_channels [int]: Number of image channels. Defaults to 3. :param num_hidden_channels [int]: Number of hidden channels. Defaults to 16. :param num_classes [int]: Number of classes. Defaults to 1. :param fire_rate [float]: NCA fire rate. Defaults to 0.8. :param hidden_size [int]: Number of neurons in hidden layer. Defaults to 128. :param learned_filters [int]: Number of learned filters. If 0, use sobel. Defaults to 2. :param pad_noise [bool]: Whether to pad input images with noise. Defaults to True. :param filter_padding [str]: Padding type to use. Might affect reliance on spatial cues. Defaults to "circular". .. py:attribute:: num_classes :value: 1 .. py:attribute:: metrics .. py:attribute:: lambda_hidden :value: 0.001 .. py:attribute:: bce_loss .. py:attribute:: dice_loss .. py:method:: loss(pred: ncalab.prediction.Prediction, label: torch.Tensor) -> Dict[str, torch.Tensor] Compute Dice loss. :param pred: Prediction. :param label: Ground truth. :returns: Dictionary of identifiers mapped to computed losses. .. py:method:: _post_forward_step(x: torch.Tensor) -> torch.Tensor .. py:method:: post_prediction(prediction: ncalab.prediction.Prediction) -> ncalab.prediction.Prediction .. py:data:: __CURRENT_PATH Project root directory. .. py:data:: ROOT_PATH Directory in which training weights are stored. .. py:data:: WEIGHTS_PATH Example task directory. .. py:data:: TASK_PATH .. py:class:: Prediction(model, steps: int, output_image: torch.Tensor, logits: torch.Tensor, head_prediction: Optional[torch.Tensor] = None) Stores the result of an NCA prediction, including the number of steps it took. Sequences are typically stored by BasicNCAModel's "record" function, and are returned as a list of Prediction objects. Constructor is typically not called explicitly. Rather, the forward pass of BasicNCAModel (and its subclasses) is responsible for filling its attributes. :param model: Reference to model used for prediction. :type model: ncalab.BasicNCAModel :param steps: Number of steps taken for the prediction. :type steps: int :param output_image: Output image tensor. :type output_image: torch.Tensor .. py:attribute:: model .. py:attribute:: steps .. py:attribute:: output_image .. py:attribute:: _output_array :type: Optional[numpy.ndarray] :value: None .. py:attribute:: logits .. py:attribute:: _logits_array :type: Optional[numpy.ndarray] :value: None .. py:attribute:: head_prediction :value: None .. py:attribute:: _head_prediction_array :type: Optional[numpy.ndarray] :value: None .. py:property:: image_channels :type: torch.Tensor Convenience property to access the image channels as a Tensor. :returns: BCWH Tensor :rtype: torch.Tensor .. py:property:: hidden_channels :type: torch.Tensor Convenience property to access the hidden channels as a Tensor. :returns: BCWH Tensor :rtype: torch.Tensor .. py:property:: output_channels :type: torch.Tensor Convenience property to access the output channels as a Tensor. :returns: BCWH Tensor :rtype: torch.Tensor .. py:property:: output_array :type: numpy.ndarray Convenience property to access the whole output image in the format of a numpy array. Brings the entire tensor to CPU on demand, and only at the first call. :returns: Numpy array in BCWH format :rtype: np.ndarray .. py:property:: image_channels_np :type: numpy.ndarray Convenience property to access the output image channels in the format of a numpy array. Brings the entire tensor to CPU on demand, and only at the first call. :returns: Numpy array in BCWH format :rtype: np.ndarray .. py:property:: hidden_channels_np :type: numpy.ndarray Convenience property to access the hidden image channels in the format of a numpy array. Brings the entire tensor to CPU on demand, and only at the first call. :returns: Numpy array in BCWH format :rtype: np.ndarray .. py:property:: output_channels_np :type: numpy.ndarray Convenience property to access the image's output channels in the format of a numpy array. Brings the entire tensor to CPU on demand, and only at the first call. :returns: Numpy array in BCWH format :rtype: np.ndarray .. py:property:: head_prediction_array :type: numpy.ndarray | None .. py:property:: logits_array :type: numpy.ndarray .. py:class:: BasicNCATrainer(nca: ncalab.models.basicNCA.AbstractNCAModel, model_path: Optional[pathlib.Path | pathlib.PosixPath], gradient_clipping: bool = False, lr: Optional[float] = None, lr_gamma: float = 0.99, adam_betas=(0.9, 0.95), batch_repeat: int = 2, max_epochs: int = 200, optimizer_method: str = 'adam', pool: Optional[ncalab.training.pool.Pool] = None, lr_scheduler: Optional[torch.optim.lr_scheduler.LRScheduler] = None) Trainer class for any model subclassing BasicNCA. :param nca: NCA model instance to train. :type nca: ncalab.AbstractNCAModel :param model_path: Path to saved models. If None, models are not saved, defaults to None. :type model_path: Path | PosixPath, optional :param gradient_clipping: Whether to clip gradients, defaults to False. :type gradient_clipping: bool, optional :param lr: Initial learning rate, defaults to 16e-4. :type lr: float, optional :param lr_gamma: Exponential learning rate decay, defaults to 0.9999. :type lr_gamma: float, optional :param adam_betas: Beta values for Adam optimizer, defaults to (0.9, 0.95). :type adam_betas: tuple, optional :param batch_repeat: How often each batch will be duplicated, dfaults to 2. :param max_epochs: Maximum number of epochs in training, defaults to 200. :param optimizer_method: Optimization method, defaults to 'adam'. :type optimizer_method: str, optional :param pool: Sample pool object. :type pool: ncalab.Pool .. py:attribute:: nca .. py:attribute:: model_path .. py:attribute:: gradient_clipping :value: False .. py:attribute:: lr_gamma :value: 0.99 .. py:attribute:: adam_betas :value: (0.9, 0.95) .. py:attribute:: batch_repeat :value: 2 .. py:attribute:: max_epochs :value: 200 .. py:attribute:: optimizer_method :value: 'adam' .. py:attribute:: pool :value: None .. py:attribute:: lr_scheduler :value: None .. py:method:: info() -> str Shows a markdown-formatted info string with training parameters. Useful for showing info on tensorboard to keep track of parameter changes. :returns [str]: Markdown-formatted info string. .. py:method:: _train_iteration(x: torch.Tensor, y: torch.Tensor, optimizer: torch.optim.Optimizer, head_optimizer: torch.optim.Optimizer | None, total_batch_iterations: int, summary_writer: Optional[torch.utils.tensorboard.SummaryWriter] = None) -> Tuple[ncalab.prediction.Prediction, Dict[str, torch.Tensor]] Run a single training iteration. :param x: Input training images. :param y: Input training labels. :param steps: Number of NCA inference time steps. :param optimizer: Optimizer. :param total_batch_iterations: Total training batch iterations :type total_batch_iterations: int :param summary_writer: Tensorboard SummaryWriter :type summary_writer: SummaryWriter, optional :returns: Predicted image. :rtype: Tuple[Prediction, Dict[str, torch.Tensor]] .. py:method:: train(dataloader_train: torch.utils.data.DataLoader, dataloader_val: Optional[torch.utils.data.DataLoader] = None, dataloader_test: Optional[torch.utils.data.DataLoader] = None, save_every: Optional[int] = None, summary_writer: Optional[torch.utils.tensorboard.SummaryWriter] = None, plot_function: Optional[ncalab.visualization.Visual] = None, earlystopping: Optional[ncalab.training.earlystopping.EarlyStopping] = None) -> ncalab.training.traininghistory.TrainingHistory Execute basic NCA training loop with a single function call. :param dataloader_train [DataLoader]: Training DataLoader :param dataloader_val [DataLoader]: Validation DataLoader :param save_every [int]: How often to save model state (in epochs). Useful for very small datasets, like growing lizard. :param summary_writer [SummaryWriter] Tensorboard SummaryWriter. Defaults to None. :param plot_function: Plot function override. If None, use model's default. Defaults to None. :param earlystopping (EarlyStopping, optional): EarlyStopping object. Defaults to None. :returns [TrainingHistory]: TrainingHistory object. .. py:class:: ParameterSet(**kwargs) .. py:attribute:: params .. py:attribute:: mutable .. py:attribute:: combinations :value: [] .. py:attribute:: index :value: 0 .. py:method:: is_mutable(key) .. py:method:: info() .. py:method:: next() -> Dict[str, Any] .. py:method:: num_combinations() .. py:method:: __len__() .. py:method:: __next__() .. py:method:: __iter__() .. py:class:: ParameterSearch(device, model_class, model_params: ParameterSet, trainer_params: ParameterSet) .. py:attribute:: device .. py:attribute:: model_class .. py:attribute:: model_params .. py:attribute:: trainer_params .. py:method:: info() -> str Generate information string with a summary of the search to run. .. py:method:: search(dataloader_train: torch.utils.data.DataLoader, dataloader_val: Optional[torch.utils.data.DataLoader] = None) Run search. :param dataloader_train [DataLoader]: Training DataLoader. :param dataloader_val [DataLoader]: Validation DataLoader. Defaults to None. .. py:method:: __call__(*args, **kwargs) Shorthand for running the search. .. py:class:: Pool(n_seed: int = 1, damage: bool = False, p_damage: float = 0.2) Sample pool that retains previous predictions. Also applies damaging patterns to images to increase the robustness of the trained NCA. :param n_seed: How many seed images to retain, defaults to 1 :type n_seed: int, optional :param damage: Whether to apply damaging patterns, defaults to False :type damage: bool, optional :param p_damage: Probability at which a damaging pattern is applied, defaults to 0.2 :type p_damage: float, optional .. py:attribute:: n_seed :value: 1 .. py:attribute:: damage :value: False .. py:attribute:: batch :type: torch.Tensor | None :value: None .. py:attribute:: p_damage :value: 0.2 .. py:method:: update(batch: torch.Tensor) :param batch: BCWH .. py:method:: sample(seed: torch.Tensor) -> torch.Tensor :param seed: BCWH :return: BCWH .. py:class:: BasicNCATrainer(nca: ncalab.models.basicNCA.AbstractNCAModel, model_path: Optional[pathlib.Path | pathlib.PosixPath], gradient_clipping: bool = False, lr: Optional[float] = None, lr_gamma: float = 0.99, adam_betas=(0.9, 0.95), batch_repeat: int = 2, max_epochs: int = 200, optimizer_method: str = 'adam', pool: Optional[ncalab.training.pool.Pool] = None, lr_scheduler: Optional[torch.optim.lr_scheduler.LRScheduler] = None) Trainer class for any model subclassing BasicNCA. :param nca: NCA model instance to train. :type nca: ncalab.AbstractNCAModel :param model_path: Path to saved models. If None, models are not saved, defaults to None. :type model_path: Path | PosixPath, optional :param gradient_clipping: Whether to clip gradients, defaults to False. :type gradient_clipping: bool, optional :param lr: Initial learning rate, defaults to 16e-4. :type lr: float, optional :param lr_gamma: Exponential learning rate decay, defaults to 0.9999. :type lr_gamma: float, optional :param adam_betas: Beta values for Adam optimizer, defaults to (0.9, 0.95). :type adam_betas: tuple, optional :param batch_repeat: How often each batch will be duplicated, dfaults to 2. :param max_epochs: Maximum number of epochs in training, defaults to 200. :param optimizer_method: Optimization method, defaults to 'adam'. :type optimizer_method: str, optional :param pool: Sample pool object. :type pool: ncalab.Pool .. py:attribute:: nca .. py:attribute:: model_path .. py:attribute:: gradient_clipping :value: False .. py:attribute:: lr_gamma :value: 0.99 .. py:attribute:: adam_betas :value: (0.9, 0.95) .. py:attribute:: batch_repeat :value: 2 .. py:attribute:: max_epochs :value: 200 .. py:attribute:: optimizer_method :value: 'adam' .. py:attribute:: pool :value: None .. py:attribute:: lr_scheduler :value: None .. py:method:: info() -> str Shows a markdown-formatted info string with training parameters. Useful for showing info on tensorboard to keep track of parameter changes. :returns [str]: Markdown-formatted info string. .. py:method:: _train_iteration(x: torch.Tensor, y: torch.Tensor, optimizer: torch.optim.Optimizer, head_optimizer: torch.optim.Optimizer | None, total_batch_iterations: int, summary_writer: Optional[torch.utils.tensorboard.SummaryWriter] = None) -> Tuple[ncalab.prediction.Prediction, Dict[str, torch.Tensor]] Run a single training iteration. :param x: Input training images. :param y: Input training labels. :param steps: Number of NCA inference time steps. :param optimizer: Optimizer. :param total_batch_iterations: Total training batch iterations :type total_batch_iterations: int :param summary_writer: Tensorboard SummaryWriter :type summary_writer: SummaryWriter, optional :returns: Predicted image. :rtype: Tuple[Prediction, Dict[str, torch.Tensor]] .. py:method:: train(dataloader_train: torch.utils.data.DataLoader, dataloader_val: Optional[torch.utils.data.DataLoader] = None, dataloader_test: Optional[torch.utils.data.DataLoader] = None, save_every: Optional[int] = None, summary_writer: Optional[torch.utils.tensorboard.SummaryWriter] = None, plot_function: Optional[ncalab.visualization.Visual] = None, earlystopping: Optional[ncalab.training.earlystopping.EarlyStopping] = None) -> ncalab.training.traininghistory.TrainingHistory Execute basic NCA training loop with a single function call. :param dataloader_train [DataLoader]: Training DataLoader :param dataloader_val [DataLoader]: Validation DataLoader :param save_every [int]: How often to save model state (in epochs). Useful for very small datasets, like growing lizard. :param summary_writer [SummaryWriter] Tensorboard SummaryWriter. Defaults to None. :param plot_function: Plot function override. If None, use model's default. Defaults to None. :param earlystopping (EarlyStopping, optional): EarlyStopping object. Defaults to None. :returns [TrainingHistory]: TrainingHistory object. .. py:class:: TrainingHistory(path: Optional[pathlib.Path | pathlib.PosixPath], metrics: Dict[str, float], current_epoch: int, current_model: ncalab.models.AbstractNCAModel, best_accuracy: float = 0, best_epoch: int = 0, best_model: Optional[ncalab.models.AbstractNCAModel] = None, verbose: bool = True) Stores data about the training progress. Populated during training with ncalab.training.BasicNCATrainer. :param path: Save and load path. :type path: Optional[Path | PosixPath] :param metrics: Dict of validation metrics :type metrics: Dict[str, float] :param current_epoch: Current training epoch. :type current_epoch: int :param current_model: Currently trained model. :type current_model: AbstractNCAModel :param best_accuracy: Best validation accuracy, defaults to 0 :type best_accuracy: float, optional :param best_epoch: Epoch of best validation accuracy, defaults to 0 :type best_epoch: int, optional :param best_model: Model with best validation accuracy, defaults to None :type best_model: Optional[AbstractNCAModel], optional :param verbose: Whether to print updates of validation accuracy, defaults to True :type verbose: bool, optional .. py:attribute:: path .. py:attribute:: metrics .. py:attribute:: current_epoch .. py:attribute:: current_model .. py:attribute:: best_accuracy :value: 0 .. py:attribute:: best_epoch :value: 0 .. py:attribute:: best_model :value: None .. py:attribute:: verbose :value: True .. py:attribute:: created_timestamp .. py:attribute:: modified_timestamp .. py:attribute:: loss :type: List[float] :value: [] .. py:method:: update(epoch: int, model: ncalab.models.AbstractNCAModel, accuracy: float, overwrite: bool = False) Populates history with current iteration's values. Automatically recognizes changes in accuracy. :param epoch: Current epoch :type epoch: int :param model: Current model :type model: AbstractNCAModel :param accuracy: Current accuracy, based on model's validation metric :type accuracy: float :param overwrite: Whether to overwrite best accuracy even with no improvement, defaults to False :type overwrite: bool, optional .. py:method:: save() Saves history and model checkpoint. .. py:method:: to_dict() -> Dict Return dict of recorded values :return: Dict of recorded values :rtype: Dict .. py:class:: EarlyStopping(patience: int, min_delta: float = 1e-06) Early stopping helper class. Helps to stop the training if no change in validation metrics is observed. :param patience: Steps to wait until stopping the training. :type patience: int :param min_delta: Minimum deviation until counter is reset, defaults to 1e-6. :type min_delta: float .. py:attribute:: patience .. py:attribute:: min_delta :value: 1e-06 .. py:attribute:: best_accuracy :value: 0.0 .. py:attribute:: counter :value: 0 .. py:method:: done() -> bool Checks whether the training can be stopped. Needs to be queried in training loop, once per epoch. :returns: Whether to stop the training or not. :rtype: bool .. py:method:: step(accuracy: float) Increases internal counter if accuracy doesn't improve, otherwise resets the counter. Needs to be called in training loop, once per epoch. :param accuracy: Validation accuracy. :type accuracy: float .. py:class:: TrainValRecord(train: List[str], val: List[str]) Helper class, storing a training / validation data split to generate respective DataLoader objects. :param train: List of training image file paths :type train: List[str] :param val: List of validation image file paths :type val: List[str] .. py:attribute:: train .. py:attribute:: val .. py:method:: dataloaders(DatasetType: Type, path: pathlib.Path | pathlib.PosixPath, transform=None, batch_sizes=None) Generate a pair of training and validation DataLoader objects, based on a given DataSet subtype. .. py:class:: SplitDefinition Stores a k-fold cross-validation split. .. py:attribute:: folds :value: [] .. py:attribute:: dataloader_test :value: None .. py:method:: read(path: pathlib.PosixPath) -> SplitDefinition :staticmethod: Reads json files with split definitions, similar to those created by nnUNet. Format is like .. highlight:: python .. code-block:: python [ { "train": [ "filename0", "filename1",... ] "val": [ "filename2", "filename3",... ] }, { ... } ] :param path: Path to JSON file containing split definition. :returns: SplitDefinition object :rtype: SplitDefinition .. py:method:: __len__() -> int .. py:method:: __getitem__(idx) -> TrainValRecord .. py:class:: KFoldCrossValidationTrainer(trainer: ncalab.training.trainer.BasicNCATrainer, split: SplitDefinition) :param trainer [BasicNCATrainer]: BasicNCATrainer, to train each individual fold. :param split [SplitDefinition]: Definition of the split used for k-fold cross-training. .. py:attribute:: trainer .. py:attribute:: model_prototype .. py:attribute:: model_name .. py:attribute:: split .. py:method:: train(DatasetType: Type, datapath: pathlib.Path | pathlib.PosixPath, transform, batch_sizes: None | Dict = None, save_every: int | None = None) -> List[ncalab.training.traininghistory.TrainingHistory] Run training loop with a single function call. :param DatasetType [Type]: Type of dataset class to use. :param datapath [Path]: _description_ :param transform: Data transform, e.g. initialized via Albumentations. :param batch_sizes: Dict of batch sizes per set, e.g. {"train": 8, "val": 16}. Defaults to None. :param save_every [int]: _description_. Defaults to None. :param plot_function: Plot function override. If None, use model's default. Defaults to None. :returns [List[TrainingHistory]]: List of TrainingHistory objects, one per fold. .. py:class:: AbstractNCAModel(device: torch.device, num_image_channels: int, num_hidden_channels: int, num_output_channels: int, plot_function: Optional[ncalab.visualization.Visual] = None, validation_metric: Optional[str] = None, fire_rate: float = 0.5, hidden_size: int = 128, use_alive_mask: bool = False, immutable_image_channels: bool = True, num_learned_filters: int = 0, filter_padding: Literal['zero', 'reflect', 'replicate', 'circular'] = 'reflect', use_laplace: bool = False, kernel_size: int = 3, pad_noise: bool = False, use_temporal_encoding: bool = False, rule_type: type[ncalab.models.basicNCA.abstractNCArule.AbstractNCARule] = MLPNCARule, rule_args=None, training_timesteps: int | Tuple[int, int] = 100, inference_timesteps: int | Tuple[int, int] = 100) Bases: :py:obj:`torch.nn.Module`, :py:obj:`abc.ABC` Abstract base class for NCA models. BasicNCAModel is a composition of an NCA backbone model (called "rule"), and an (optional) head module for downstream tasks. :param device: Pytorch device descriptor. :param num_image_channels: Number of channels reserved for input image. :param num_hidden_channels: Number of hidden channels (communication channels). :param num_output_channels: Number of output channels. :param validation_metric: :param fire_rate: Fire rate for stochastic weight update. Defaults to 0.5. :param hidden_size: Number of neurons in hidden layer. Defaults to 128. :param use_alive_mask: Whether to use alive masking (channel 3) during training. Defaults to False. :param immutable_image_channels: If image channels should be fixed during inference, which is the case for most segmentation or classification problems. Defaults to True. :param num_learned_filters: Number of learned filters. If zero, use two sobel filters instead. Defaults to 2. :param filter_padding: Padding type to use. Might affect reliance on spatial cues. Defaults to "circular". :param use_laplace: Whether to use Laplace filter (only if num_learned_filters == 0) :param kernel_size: Filter kernel size (only for learned filters) :param pad_noise: Whether to pad input image tensor with noise in hidden / output channels :param use_temporal_encoding: :param rule_type: :param rule_args: :param training_timesteps: :param inference_timesteps: .. py:attribute:: device .. py:attribute:: num_image_channels .. py:attribute:: num_hidden_channels .. py:attribute:: num_output_channels .. py:attribute:: num_channels .. py:attribute:: fire_rate :value: 0.5 .. py:attribute:: hidden_size :value: 128 .. py:attribute:: use_alive_mask :value: False .. py:attribute:: immutable_image_channels :value: True .. py:attribute:: num_learned_filters :value: 0 .. py:attribute:: use_laplace :value: False .. py:attribute:: kernel_size :value: 3 .. py:attribute:: filter_padding :value: 'reflect' .. py:attribute:: pad_noise :value: False .. py:attribute:: use_temporal_encoding :value: False .. py:attribute:: plot_function :value: None .. py:attribute:: validation_metric :value: None .. py:attribute:: training_timesteps :value: 100 .. py:attribute:: inference_timesteps :value: 100 .. py:attribute:: perception .. py:attribute:: input_vector_size .. py:attribute:: rule_type .. py:attribute:: rule_args :value: None .. py:attribute:: rule .. py:attribute:: head :type: ncalab.models.basicNCA.abstractNCAhead.AbstractNCAHead | None :value: None .. py:attribute:: metrics :type: Dict[str, torchmetrics.Metric] .. py:method:: _define_rule() -> ncalab.models.basicNCA.abstractNCArule.AbstractNCARule .. py:method:: prepare_input(x: torch.Tensor) -> torch.Tensor Preprocess input. Intended to be overwritten by subclass, if preprocessing is necessary. :param x [torch.Tensor]: Input tensor to preprocess. :returns: Processed tensor. .. py:method:: _alive(x) .. py:method:: _update(x: torch.Tensor, step: int) -> torch.Tensor Compute residual cell update. :param x [torch.Tensor]: Input tensor, BCWH :param step [int]: Current timestep, required for computing temporal encoding. :returns: Residual cell update, BCWH. .. py:method:: _forward_step(x: torch.Tensor, step: int) .. py:method:: forward(x: torch.Tensor, steps: int = 1) -> ncalab.prediction.Prediction :param x [torch.Tensor]: Input image, padded along the channel dimension, BCWH. :param steps [int]: Time steps in forward pass. :returns [Prediction]: Prediction object. .. py:method:: _post_forward_step(x: torch.Tensor) -> torch.Tensor .. py:method:: loss(pred: ncalab.prediction.Prediction, label: torch.Tensor) -> Dict[str, torch.Tensor] Compute loss. Needs to be overloaded by any subclass. Please note that the returned dict needs to hold "total" key in which the total loss is stored, which is typically a weighted sum of other losses. The total loss is backpropagated, whereas the other losses are sent to tensorboard. :param image [torch.Tensor]: Input image, BCWH. :param label [torch.Tensor]: Ground truth, BCWH. :returns: Dictionary of identifiers mapped to computed losses. .. py:method:: finetune(freeze_head: bool = False) Prepare model for fine tuning by freezing everything except the final layer, and setting to "train" mode. :param: freeze_head .. py:method:: predict(image: torch.Tensor, steps: Optional[int | Tuple[int, int]] = None) -> ncalab.prediction.Prediction Make an NCA prediction, performing multiple forward passes to yield a final result. :param image: Input image, BCWH. :type image: torch.Tensor :param steps: Time steps :type steps: Optional[int] :returns: Prediction object. :rtype: Prediction .. py:method:: record(image: torch.Tensor, steps: Optional[int | Tuple[int, int]] = None) -> List[ncalab.prediction.Prediction] Record predictions for all time steps and return the resulting sequence of predictions. :param image: Input image, BCWH. :type image: torch.Tensor :returns: List of Prediction objects. :rtype: List[Prediction] .. py:method:: validate(dataloader: torch.utils.data.DataLoader, steps: Optional[int] = None) -> Tuple[Dict[str, float], List[ncalab.prediction.Prediction]] Make a prediction on an image of the validation set and return metrics computed with respect to a labelled validation image. :param dataloader [torch.utils.data.DataLoader]: Dataloader for validation images :param steps [int]: Inference steps :returns [Tuple[float, List[Prediction]]]: Validation metric, predicted image BCWH .. py:method:: _to_dict() -> Dict[str, Any] .. py:method:: to_dict() -> Dict[str, Any] .. py:method:: from_dict(d: Dict[str, Any]) :classmethod: .. py:method:: num_trainable_parameters() -> int Returns the number of trainable model parameters. :return: Number of trainable parameters. :rtype: int .. py:method:: save(path: str | os.PathLike) .. py:method:: load(model: AbstractNCAModel, path: str | os.PathLike) -> AbstractNCAModel :staticmethod: .. py:method:: post_prediction(prediction: ncalab.prediction.Prediction) -> ncalab.prediction.Prediction .. py:class:: Prediction(model, steps: int, output_image: torch.Tensor, logits: torch.Tensor, head_prediction: Optional[torch.Tensor] = None) Stores the result of an NCA prediction, including the number of steps it took. Sequences are typically stored by BasicNCAModel's "record" function, and are returned as a list of Prediction objects. Constructor is typically not called explicitly. Rather, the forward pass of BasicNCAModel (and its subclasses) is responsible for filling its attributes. :param model: Reference to model used for prediction. :type model: ncalab.BasicNCAModel :param steps: Number of steps taken for the prediction. :type steps: int :param output_image: Output image tensor. :type output_image: torch.Tensor .. py:attribute:: model .. py:attribute:: steps .. py:attribute:: output_image .. py:attribute:: _output_array :type: Optional[numpy.ndarray] :value: None .. py:attribute:: logits .. py:attribute:: _logits_array :type: Optional[numpy.ndarray] :value: None .. py:attribute:: head_prediction :value: None .. py:attribute:: _head_prediction_array :type: Optional[numpy.ndarray] :value: None .. py:property:: image_channels :type: torch.Tensor Convenience property to access the image channels as a Tensor. :returns: BCWH Tensor :rtype: torch.Tensor .. py:property:: hidden_channels :type: torch.Tensor Convenience property to access the hidden channels as a Tensor. :returns: BCWH Tensor :rtype: torch.Tensor .. py:property:: output_channels :type: torch.Tensor Convenience property to access the output channels as a Tensor. :returns: BCWH Tensor :rtype: torch.Tensor .. py:property:: output_array :type: numpy.ndarray Convenience property to access the whole output image in the format of a numpy array. Brings the entire tensor to CPU on demand, and only at the first call. :returns: Numpy array in BCWH format :rtype: np.ndarray .. py:property:: image_channels_np :type: numpy.ndarray Convenience property to access the output image channels in the format of a numpy array. Brings the entire tensor to CPU on demand, and only at the first call. :returns: Numpy array in BCWH format :rtype: np.ndarray .. py:property:: hidden_channels_np :type: numpy.ndarray Convenience property to access the hidden image channels in the format of a numpy array. Brings the entire tensor to CPU on demand, and only at the first call. :returns: Numpy array in BCWH format :rtype: np.ndarray .. py:property:: output_channels_np :type: numpy.ndarray Convenience property to access the image's output channels in the format of a numpy array. Brings the entire tensor to CPU on demand, and only at the first call. :returns: Numpy array in BCWH format :rtype: np.ndarray .. py:property:: head_prediction_array :type: numpy.ndarray | None .. py:property:: logits_array :type: numpy.ndarray .. py:class:: UncertaintyEstimator(nca: ncalab.models.AbstractNCAModel) :param nca: Trained NCA model :type nca: AbstractNCAModel .. py:attribute:: nca .. py:method:: _estimate(image: torch.Tensor) -> Tuple[torch.Tensor, List[ncalab.prediction.Prediction]] Internal uncertainty estimation method. :param image: Input image sample :type image: torch.Tensor :return: Float tensor of uncertainty heatmap (BCWH), predictions, reduced uncertainty score :rtype: Tuple[torch.Tensor, List[Prediction], float] .. py:method:: estimate(image: torch.Tensor, reduce: str = 'mean') -> Tuple[torch.Tensor, List[ncalab.prediction.Prediction], torch.Tensor] Estimate predictive uncertainty. :param image: Input image sample :type image: torch.Tensor :param reduce: Reduction strategy, defaults to "mean". :type reduce: str :return: Float tensor of uncertainty heatmap (BCWH), final prediction, reduced uncertainty score for batch (BC) :rtype: Tuple[torch.Tensor, List[Prediction], torch.Tensor] .. py:method:: __call__(*args, **kwargs) .. py:class:: NQM(nca: ncalab.models.AbstractNCAModel, N: int = 10, normalize=False) Bases: :py:obj:`UncertaintyEstimator` Variance over multiple predictions. :param nca: Trained NCA model :type nca: AbstractNCAModel .. py:attribute:: N :value: 10 .. py:attribute:: normalize :value: False .. py:method:: _estimate(image: torch.Tensor) -> Tuple[torch.Tensor, List[ncalab.prediction.Prediction]] Internal uncertainty estimation method. :param image: Input image sample :type image: torch.Tensor :return: Float tensor of uncertainty heatmap (BCWH), predictions, reduced uncertainty score :rtype: Tuple[torch.Tensor, List[Prediction], float] .. py:class:: MCMC(nca: ncalab.models.AbstractNCAModel, N_last: int = 10, normalize=False) Bases: :py:obj:`UncertaintyEstimator` Markov-Chain Monte Carlo :param nca: Trained NCA model :type nca: AbstractNCAModel .. py:attribute:: N_last :value: 10 .. py:attribute:: normalize :value: False .. py:method:: _estimate(image: torch.Tensor) -> Tuple[torch.Tensor, List[ncalab.prediction.Prediction]] Internal uncertainty estimation method. :param image: Input image sample :type image: torch.Tensor :return: Float tensor of uncertainty heatmap (BCWH), predictions, reduced uncertainty score :rtype: Tuple[torch.Tensor, List[Prediction], float] .. py:function:: get_compute_device(device: str = 'cuda:0') -> torch.device Obtain a pytorch compute device handle based on input string. If user tries to get a CUDA device, but none is available, defaults to CPU. :param device: Device string, defaults to "cuda:0". :type device: str :returns: Pytorch compute device. :rtype: torch.device .. py:function:: pad_input(x: torch.Tensor, nca: ncalab.models.AbstractNCAModel, noise: bool = True, mean: float = 0.5, std: float = 0.225) -> torch.Tensor Pads the BCWH input tensor along its channel dimension to match the expected number of channels required by the NCA model. Pads with either Gaussian noise (parameterized by mean and std) or zeros, depending on the "noise" parameter. :param x: Input image tensor, BCWH. :type x: torch.Tensor :param nca: NCA model definition. :type nca: ncalab.BasicNCAModel :param noise: Whether to pad with noise. Otherwise zeros, defaults to True. :type noise: bool, optional :param mean: Mean (mu) of Gaussian noise distribution, defaults to 0.5. :type mean: float, optional :param std: Standard deviation (sigma) of Gaussian noise distribution, defaults to 0.225. :type std: float, optional :returns: Input tensor, BCWH, padded along the channel dimension. :rtype: torch.Tensor .. py:function:: print_NCALab_banner() Show NCALab banner on terminal. .. py:function:: print_mascot(message: str) Show help text in a speech bubble. :param message: Message to display. :type message: str .. py:data:: DEFAULT_RANDOM_SEED :value: 1337 .. py:function:: fix_random_seed(seed: int = DEFAULT_RANDOM_SEED) Fixes the random seed for all pseudo-random number generators, including Python-native, Numpy and Pytorch. :param seed: Random seed, defaults to DEFAULT_RANDOM_SEED. :type seed: int, optional .. py:function:: unwrap(x: Any) Panics if x is None, otherwise returns x. This is a useful shorthand for cases such as ``x = unwrap(some_object).do_something()`` in which we are 99% certain that some_object is not None and want to avoid a mypy complaint. :param x: Any kind of object. :type x: Any :raises RuntimeError: If x is None. :return: Just passes through the input x if it is not None. .. py:function:: intepret_range_parameter(x: int | Tuple[int, int]) -> int Interpret a range parameter that is passed for NCA timesteps. If the parameter is a single int, just return it as is. If the parameter is a two-valued tuple, interpret it as a [min,max) and randomly sample from that range. :param x: _description_ :type x: int | Tuple[int, int] :raises TypeError: If something else than an int or a tuple was passed. :return: _description_ :rtype: int .. py:class:: AnimatorStyle(color_background, color_overlay, color_title, color_progress, underline: bool = True, progress_h: int = 3) .. py:attribute:: color_background .. py:attribute:: color_overlay .. py:attribute:: color_title .. py:attribute:: color_progress .. py:attribute:: underline :value: True .. py:attribute:: progress_h :value: 3 .. py:method:: apply(fig, ax) .. py:data:: animator_style_dark .. py:data:: animator_styles .. py:function:: draw_segmentation_overlay(image, mask, style) .. py:class:: Animator(nca: ncalab.models.AbstractNCAModel, seed: torch.Tensor, steps: Optional[int] = None, interval: int = 100, repeat: bool = True, repeat_delay: int = 10000, overlay: bool = False, show_timestep: bool = True, hidden: bool = False, show_input: bool = False, style: str | AnimatorStyle = 'dark') Responsible for rendering NCA predictions as GIFs. :param nca: NCA model instance :type nca: ncalab.AbstractNCAModel :param seed: Input image for the NCA model :type seed: torch.Tensor :param steps: Number of NCA prediction steps per sample, defaults to 100 :type steps: int, optional :param interval: Time of each frame (milliseconds), defaults to 100 :type interval: int, optional :param repeat: Whether to loop the animation, defaults to True :type repeat: bool, optional :param repeat_delay: Time after which the animation is repeated (milliseconds), defaults to 10000 :type repeat_delay: int, optional :param overlay: Whether to overlay output channel (segmentation mask), defaults to False :type overlay: bool, optional :param show_timestep: Whether to display timestep in caption, defaults to True :type show_timestep: bool, optional .. py:attribute:: animation_fig .. py:method:: save(path: str | pathlib.Path) Save generated figure as GIF :param path: Output path :type path: str | Path .. py:class:: Prediction(model, steps: int, output_image: torch.Tensor, logits: torch.Tensor, head_prediction: Optional[torch.Tensor] = None) Stores the result of an NCA prediction, including the number of steps it took. Sequences are typically stored by BasicNCAModel's "record" function, and are returned as a list of Prediction objects. Constructor is typically not called explicitly. Rather, the forward pass of BasicNCAModel (and its subclasses) is responsible for filling its attributes. :param model: Reference to model used for prediction. :type model: ncalab.BasicNCAModel :param steps: Number of steps taken for the prediction. :type steps: int :param output_image: Output image tensor. :type output_image: torch.Tensor .. py:attribute:: model .. py:attribute:: steps .. py:attribute:: output_image .. py:attribute:: _output_array :type: Optional[numpy.ndarray] :value: None .. py:attribute:: logits .. py:attribute:: _logits_array :type: Optional[numpy.ndarray] :value: None .. py:attribute:: head_prediction :value: None .. py:attribute:: _head_prediction_array :type: Optional[numpy.ndarray] :value: None .. py:property:: image_channels :type: torch.Tensor Convenience property to access the image channels as a Tensor. :returns: BCWH Tensor :rtype: torch.Tensor .. py:property:: hidden_channels :type: torch.Tensor Convenience property to access the hidden channels as a Tensor. :returns: BCWH Tensor :rtype: torch.Tensor .. py:property:: output_channels :type: torch.Tensor Convenience property to access the output channels as a Tensor. :returns: BCWH Tensor :rtype: torch.Tensor .. py:property:: output_array :type: numpy.ndarray Convenience property to access the whole output image in the format of a numpy array. Brings the entire tensor to CPU on demand, and only at the first call. :returns: Numpy array in BCWH format :rtype: np.ndarray .. py:property:: image_channels_np :type: numpy.ndarray Convenience property to access the output image channels in the format of a numpy array. Brings the entire tensor to CPU on demand, and only at the first call. :returns: Numpy array in BCWH format :rtype: np.ndarray .. py:property:: hidden_channels_np :type: numpy.ndarray Convenience property to access the hidden image channels in the format of a numpy array. Brings the entire tensor to CPU on demand, and only at the first call. :returns: Numpy array in BCWH format :rtype: np.ndarray .. py:property:: output_channels_np :type: numpy.ndarray Convenience property to access the image's output channels in the format of a numpy array. Brings the entire tensor to CPU on demand, and only at the first call. :returns: Numpy array in BCWH format :rtype: np.ndarray .. py:property:: head_prediction_array :type: numpy.ndarray | None .. py:property:: logits_array :type: numpy.ndarray .. py:function:: unwrap(x: Any) Panics if x is None, otherwise returns x. This is a useful shorthand for cases such as ``x = unwrap(some_object).do_something()`` in which we are 99% certain that some_object is not None and want to avoid a mypy complaint. :param x: Any kind of object. :type x: Any :raises RuntimeError: If x is None. :return: Just passes through the input x if it is not None. .. py:function:: abbreviate_label(L, max_len=8) .. py:function:: show_image_row(ax, images, vmin=None, vmax=None, cmap=None, overlays=None, overlay_vmin=None, overlay_vmax=None, overlay_cmap=None, label: str = '', colorbar: bool = False, x_index: bool = False, normalize: bool = False) Shows a row of images next to each other. :param ax: Axis object. :param images: List of grayscale, RGB or RGBA images, can be CWH or WHC. :param vmin: Minimum value to clip channel values, defaults to None :param vmax: Maximum value to clip channel values, defaults to None :param cmap: matplotlib colormap to apply, defaults to None :param overlays: _description_, defaults to None :param overlay_vmin: _description_, defaults to None :param overlay_vmax: _description_, defaults to None :param overlay_cmap: _description_, defaults to None :param label: y-axis label next to first image, defaults to "" :param colorbar: Whether to display a colorbar next to the last image, defaults to False :param x_index: Whether to show the batch index below each image, defaults to False :param normalize: Whether to normalize images across batch .. py:class:: Visual Base class for tensorboard visuals. .. py:method:: show(model, image: numpy.ndarray, prediction: ncalab.prediction.Prediction, label: numpy.ndarray) -> matplotlib.figure.Figure .. py:class:: VisualBinaryImageClassification Bases: :py:obj:`Visual` Base class for tensorboard visuals. .. py:method:: show(model, image: numpy.ndarray, prediction: ncalab.prediction.Prediction, label: numpy.ndarray) -> matplotlib.figure.Figure .. py:class:: VisualRGBImageClassification Bases: :py:obj:`Visual` Base class for tensorboard visuals. .. py:method:: show(model, image: numpy.ndarray, prediction: ncalab.prediction.Prediction, label: numpy.ndarray) -> matplotlib.figure.Figure .. py:class:: VisualMultiImageClassification Bases: :py:obj:`Visual` Base class for tensorboard visuals. .. py:attribute:: new_instance .. py:class:: VisualBinaryImageSegmentation Bases: :py:obj:`Visual` Base class for tensorboard visuals. .. py:method:: show(model, image: numpy.ndarray, prediction: ncalab.prediction.Prediction, label: numpy.ndarray) -> matplotlib.figure.Figure .. py:class:: VisualDepthEstimation Bases: :py:obj:`Visual` Base class for tensorboard visuals. .. py:method:: show(model, image: numpy.ndarray, prediction: ncalab.prediction.Prediction, label: numpy.ndarray) -> matplotlib.figure.Figure .. py:class:: VisualGrowing Bases: :py:obj:`Visual` Base class for tensorboard visuals. .. py:method:: show(model, image: numpy.ndarray, prediction: ncalab.prediction.Prediction, label: numpy.ndarray) -> matplotlib.figure.Figure