deeptrees.dataloading

deeptrees.dataloading.datamodule

class deeptrees.dataloading.datamodule.TreeCrownDelineationDataModule(rasters, masks, outlines, distance_transforms, training_split=0.7, batch_size=16, val_batch_size=2, num_workers=8, augment_train={}, augment_eval={}, ndvi_config={'concatenate': False}, gci_config={'concatenate': False}, hue_config={'concatenate': False}, divide_by=1, dilate_outlines=0, shuffle=True, train_indices=None, val_indices=None, test_indices=None, ground_truth_config={'labels': None}, dim_ordering='CHW')

Bases: LightningDataModule

This class is responsible for managing the datasets, applying preprocessing steps, and providing DataLoaders for training, validation, testing, and prediction.

rasters

List of file paths, or path to folder containing the training raster files (TIF).

Type:

Union[str, list]

masks

List of file paths, or path to folder containing the masks.

Type:

Union[str, list]

outlines

List of file paths, or path to folder containing the outlines.

Type:

Union[str, list]

distance_transforms

List of file paths, or path to folder containing the distance transforms.

Type:

Union[str, list]

training_split

Training data split. Defaults to 0.7.

Type:

float

batch_size

Training batch size. Defaults to 16.

Type:

int

val_batch_size

Validation batch size. Defaults to 2.

Type:

int

num_workers

Number of workers in DataLoader. Defaults to 8.

Type:

int

augment_train

Dictionary defining torchvision augmentations to be used during training. Defaults to {}.

Type:

Dict[str, Any]

augment_eval

Dictionary defining torchvision augmentations to be used during validation/testing. Defaults to {}.

Type:

Dict[str, Any]

ndvi_config

Dictionary defining the NDVI concatenation settings. Defaults to {‘concatenate’: False}.

Type:

Dict[str, Any]

divide_by

Scalar used to normalize rasters. Defaults to 1.

Type:

float

dilate_outlines

If present (>0), dilate outlines by the given number of pixels. Defaults to 0.

Type:

int

shuffle

If True, shuffle data before applying split. Defaults to True.

Type:

bool

train_indices

List of indices of files to be used for training. Cannot be used with shuffle. Defaults to None.

Type:

list[int]

val_indices

List of indices of files to be used for validation. Cannot be used with shuffle. Defaults to None.

Type:

list[int]

test_indices

List of indices of files to be used for testing. Cannot be used with shuffle. Defaults to None.

Type:

list[int

ground_truth_config

Dictionary defining the ground truth preprocessing settings. Defaults to {‘labels’: None}.

Type:

Dict[str, Any]

__init__(rasters, masks, outlines, distance_transforms, training_split=0.7, batch_size=16, val_batch_size=2, num_workers=8, augment_train={}, augment_eval={}, ndvi_config={'concatenate': False}, gci_config={'concatenate': False}, hue_config={'concatenate': False}, divide_by=1, dilate_outlines=0, shuffle=True, train_indices=None, val_indices=None, test_indices=None, ground_truth_config={'labels': None}, dim_ordering='CHW')

TreeCrownDelineationDataModule

Datamodule to hold the different datasets, apply preprocessing, and return DataLoaders.

Parameters:

• rasters (Union[str, list]) – List of file paths, or path to folder containing the training raster files (TIF).

• masks (Union[str, list]) – List of file paths, or path to folder containing the masks.

• outlines (Union[str, list]) – List of file paths, or path to folder containing the outlines.

• distance_transforms (Union[str, list]) – List of file paths, or path to folder containing the distance transforms.

• training_split (float, optional) – Training data split. Defaults to 0.7.

• batch_size (int, optional) – Training batch size. Defaults to 16.

• val_batch_size (int, optional) – Validation batch size. Defaults to 2.

• num_workers (int, optional) – Number of workers in DataLoader. Defaults to 8.

• augment_train (Dict[str, Any], optional) – Dictionary defining torchvision augmentations to be used during training. Defaults to {}.

• augment_eval (Dict[str, Any], optional) – Dictionary defining torchvision augmentations to be used during validation/testing. Defaults to {}.

• ndvi_config (_type_, optional) – Dictionary defining the NDVI concatenation settings. Defaults to {‘concatenate’: False}.

• divide_by (float, optional) – Scalar used to normalize rasters. Defaults to 1.

• dilate_outlines (int, optional) – If present (>0), dilate outlines be given number of pixels. Defaults to False (=0).

• shuffle (bool, optional) – If True, shuffle data before applying split. Defaults to True.

• train_indices (list[int], optional) – List of indices of files to be used for training. Cannot be used with shuffle. Defaults to None.

• val_indices (list[int], optional) – List of indices of files to be used for validation. Cannot be used with shuffle. Defaults to None.

• test_indices (list[int], optional) – List of indices of files to be used for testing. Cannot be used with shuffle. Defaults to None.

• ground_truth_config (Dict[str, Any], optional) – Dictionary defining the ground truth preprocessing settings. Defaults to {‘labels’: None}.

predict_dataloader()

Return the dataloader for the predict dataset.

Returns:

Pytorch dataloader for the predict dataset.

Return type:

DataLoader

prepare_data()

Prepare the ground truth masks, outlines, and distance transforms from ground truth labels.

Return type:

None

setup(stage='fit')

Setup the dataset.

Parameters:

stage (str, optional) – Current stage (fit/test). Defaults to fit.

Raises:

ValueError – If shuffled dataset is passed together with fixed indices.

test_dataloader()

Return the dataloader for the test dataset.

Returns:

Pytorch dataloader for the test dataset.

Return type:

DataLoader

train_dataloader()

Return the dataloader for the training dataset.

Returns:

Pytorch dataloader for the training dataset.

Return type:

DataLoader

val_dataloader()

Return the dataloader for the validation dataset.

Returns:

Pytorch dataloader for the validation dataset.

Return type:

DataLoader

deeptrees.dataloading.datasets

class deeptrees.dataloading.datasets.TreeCrownDelineationBaseDataset(raster_files, target_files, augmentation, ndvi_config={'concatenate': False}, gci_config={'concatenate': False}, hue_config={'concatenate': False}, dilate_outlines=0, overwrite_nan_with_zeros=True, in_memory=False, dim_ordering='CHW', dtype='float32', divide_by=1)

Bases: ABC

In-memory remote sensing dataset for image segmentation.

This base dataset class handles the loading and preprocessing of raster and target files for tree crown delineation, and image segmentation task.

Parameters: - raster_files (list[str]): List of file paths to the raster images. - target_files (list[str]): List of file paths to the target images. - augmentation (Dict[str, Any]): Dictionary containing augmentation configurations. - ndvi_config (Dict[str, Any], optional): Configuration for NDVI, default is {‘concatenate’: False}. - dilate_outlines (int, optional): Number of pixels to dilate the outlines, default is 0. - overwrite_nan_with_zeros (bool, optional): Whether to overwrite NaN values with zeros, default is True. - in_memory (bool, optional): Whether to load the data into memory, default is False. - dim_ordering (str, optional): Dimension ordering, default is “CHW”. - dtype (str, optional): Data type of the raster images, default is “float32”. - divide_by (int, optional): Value to divide the raster data by, default is 1.

__init__(raster_files, target_files, augmentation, ndvi_config={'concatenate': False}, gci_config={'concatenate': False}, hue_config={'concatenate': False}, dilate_outlines=0, overwrite_nan_with_zeros=True, in_memory=False, dim_ordering='CHW', dtype='float32', divide_by=1)

Creates a dataset containing images and targets (masks, outlines, and distance_transforms).

Parameters:

• raster_files (list[str]) – List of file paths to source rasters. File names must be of the form ‘…/the_name_i.tif’ where i is some index

• target_files (list[str]) – mask_files: A tuple containing lists of file paths to different sorts of ‘masks’, e.g. mask, outline, distance transform. The mask and raster file names must have the same index ending.

• augmentation (Dict[str, Any]) – Dictionary defining augmentations. Keys correspond to torchvision transforms, values to their kwargs.

• ndvi_config (_type_, optional) – Dictionary defining NDVI concatenation. Defaults to {‘concatenate’: False}.

• dilate_outlines (int, optional) – If present, dilate outlines by give amount of pixels. Defaults to 0.

• overwrite_nan_with_zeros (bool, optional) – If True, fill missing values in targets with 0. Defaults to True.

• in_memory (bool, optional) – If True, load all rasters and targets into memory (works for small datasets, beware of OOM error). Defaults to True.

• dim_ordering (str, optional) – Order of dimensions. Defaults to “CHW”.

• dtype (str, optional) – torch Datatype. Defaults to “float32”.

• divide_by (int, optional) – Scalar to divide the raster pixel values by. Defaults to 1.

_abc_impl = <_abc._abc_data object>

static concatenate_gci_to_raster(raster, red=0, green=1, nir=3, dim_ordering='CHW', rescale=False)

Concatenate GCI to the raster.

Parameters:

• raster (xr.Dataset) – loaded raster tile

• red (int, optional) – Index of red channel in raster bands. Defaults to 0.

• green (int, optional) – Index of green channel in raster bands. Defaults to 1.

• nir (int, optional) – Index of NIR channel in raster bands. Defaults to 3.

• rescale (bool, optional) – Rescale GCI to [0, 1]. Defaults to False.

Returns:

_description_

Return type:

xr.Dataset

static concatenate_hue_to_raster(raster, red=0, green=1, blue=2, dim_ordering='CHW', rescale=False)

Concatenate hue to the raster.

Parameters:

• raster (xr.Dataset) – loaded raster tile

• red (int, optional) – Index of red channel in raster bands. Defaults to 0.

• green (int, optional) – Index of green channel in raster bands. Defaults to 1.

• blue (int, optional) – Index of blue channel in raster bands. Defaults to 2.

• rescale (bool, optional) – Rescale hue to [0, 1]. Defaults to False.

Returns:

_description_

Return type:

xr.Dataset

static concatenate_ndvi_to_raster(raster, red=0, nir=3, dim_ordering='CHW', rescale=False)

Concatenate NDVI to the raster.

Parameters:

• raster (xr.Dataset) – loaded raster tile

• red (int, optional) – Index of red channel in raster bands. Defaults to 0.

• nir (int, optional) – Index of NIR channel in raster bands. Defaults to 3.

• rescale (bool, optional) – Rescale NDVI to [0, 1]. Defaults to False.

Returns:

_description_

Return type:

xr.Dataset

load_data()

Load and preprocess data from files into memory.

load_raster(file, used_bands=None)

Loads a raster from disk.

Parameters:

• file (str) – file to load

• used_bands (list) – bands to use, indexing starts from 0, default ‘None’ loads all bands

Returns:

raster

load_target(file)

Load target raster from disk.

Parameters:

file (str) – path to file.

Returns:

Raster.

Return type:

xr.Dataset or xr.DataArray

class deeptrees.dataloading.datasets.TreeCrownDelineationDataset(raster_files, target_files, augmentation, ndvi_config={'concatenate': False}, gci_config={'concatenate': False}, hue_config={'concatenate': False}, dilate_outlines=0, overwrite_nan_with_zeros=True, in_memory=True, dim_ordering='CHW', dtype='float32', divide_by=1)

Bases: TreeCrownDelineationBaseDataset, IterableDataset

Iterable TreeCrownDelineation dataset.

Yields samples from all rasters until, statistically, each pixel has been covered once. If the raster edge length exceeds the sample edge length, this implies that multiple samples can be sampled from one raster.

__init__(raster_files, target_files, augmentation, ndvi_config={'concatenate': False}, gci_config={'concatenate': False}, hue_config={'concatenate': False}, dilate_outlines=0, overwrite_nan_with_zeros=True, in_memory=True, dim_ordering='CHW', dtype='float32', divide_by=1)

Creates a dataset containing images and targets (masks, outlines, and distance_transforms).

Parameters:

• raster_files (list[str]) – List of file paths to source rasters. File names must be of the form ‘…/the_name_i.tif’ where i is some index

• target_files (list[str]) – mask_files: A tuple containing lists of file paths to different sorts of ‘masks’, e.g. mask, outline, distance transform. The mask and raster file names must have the same index ending.

• augmentation (Dict[str, Any]) – Dictionary defining augmentations. Keys correspond to torchvision transforms, values to their kwargs.

• ndvi_config (_type_, optional) – Dictionary defining NDVI concatenation. Defaults to {‘concatenate’: False}.

• dilate_outlines (int, optional) – If present, dilate outlines by give amount of pixels. Defaults to 0.

• overwrite_nan_with_zeros (bool, optional) – If True, fill missing values in targets with 0. Defaults to True.

• in_memory (bool, optional) – If True, load all rasters and targets into memory (works for small datasets, beware of OOM error). Defaults to True.

• dim_ordering (str, optional) – Order of dimensions. Defaults to “CHW”.

• dtype (str, optional) – torch Datatype. Defaults to “float32”.

• divide_by (int, optional) – Scalar to divide the raster pixel values by. Defaults to 1.

_abc_impl = <_abc._abc_data object>

class deeptrees.dataloading.datasets.TreeCrownDelineationInferenceDataset(raster_files, augmentation, ndvi_config={'concatenate': False}, gci_config={'concatenate': False}, hue_config={'concatenate': False}, dilate_outlines=0, overwrite_nan_with_zeros=True, in_memory=False, dim_ordering='CHW', dtype='float32', divide_by=1)

Bases: TreeCrownDelineationBaseDataset, Dataset

Map-style (“standard”) dataset for the TreeCrownDelineation data.

This dataset is used for inference and returns complete rasters along with selected metadata.

Parameters: - raster_files (list[str]): List of file paths to the raster images. - augmentation (Dict[str, Any]): Dictionary containing augmentation configurations. - ndvi_config (Dict[str, Any], optional): Configuration for NDVI, default is {‘concatenate’: False}. - dilate_outlines (int, optional): Number of pixels to dilate the outlines, default is 0. - overwrite_nan_with_zeros (bool, optional): Whether to overwrite NaN values with zeros, default is True. - in_memory (bool, optional): Whether to load the data into memory, default is False. - dim_ordering (str, optional): Dimension ordering, default is “CHW”. - dtype (str, optional): Data type of the raster images, default is “float32”. - divide_by (int, optional): Value to divide the raster data by, default is 1.

__init__(raster_files, augmentation, ndvi_config={'concatenate': False}, gci_config={'concatenate': False}, hue_config={'concatenate': False}, dilate_outlines=0, overwrite_nan_with_zeros=True, in_memory=False, dim_ordering='CHW', dtype='float32', divide_by=1)

Creates a dataset containing images and targets (masks, outlines, and distance_transforms).

Parameters:

• raster_files (list[str]) – List of file paths to source rasters. File names must be of the form ‘…/the_name_i.tif’ where i is some index

• target_files (list[str]) – mask_files: A tuple containing lists of file paths to different sorts of ‘masks’, e.g. mask, outline, distance transform. The mask and raster file names must have the same index ending.

• augmentation (Dict[str, Any]) – Dictionary defining augmentations. Keys correspond to torchvision transforms, values to their kwargs.

• ndvi_config (_type_, optional) – Dictionary defining NDVI concatenation. Defaults to {‘concatenate’: False}.

• dilate_outlines (int, optional) – If present, dilate outlines by give amount of pixels. Defaults to 0.

• overwrite_nan_with_zeros (bool, optional) – If True, fill missing values in targets with 0. Defaults to True.

• in_memory (bool, optional) – If True, load all rasters and targets into memory (works for small datasets, beware of OOM error). Defaults to True.

• dim_ordering (str, optional) – Order of dimensions. Defaults to “CHW”.

• dtype (str, optional) – torch Datatype. Defaults to “float32”.

• divide_by (int, optional) – Scalar to divide the raster pixel values by. Defaults to 1.

_abc_impl = <_abc._abc_data object>

deeptrees.dataloading.preprocessing

Classes to be used in preprocessing labels -> ground truth rasters

Based on scripts/rasterize.py and scripts/rasterize_to_distance_transform.py

class deeptrees.dataloading.preprocessing.DistanceTransformGenerator(rasters, output_path, output_file_prefix, ground_truth_labels, valid_class_ids='all', class_column_name='class', crs='EPSG:25832', nproc=1, area_max=None, area_min=3)

Bases: GroundTruthGenerator

Generate distance transforms from tiles and ground truth labels.

__init__(rasters, output_path, output_file_prefix, ground_truth_labels, valid_class_ids='all', class_column_name='class', crs='EPSG:25832', nproc=1, area_max=None, area_min=3)

Initialize the MaskOutlinesGenerator instance.

Parameters:

• rasters (Union[str, list]) – (List of) file path(s) to the raster files

• output_path (str) – Output directory

• output_file_prefix (str) – Output file prefix. Suffix is infered from raster files.

• ground_truth_labels (Union[str, gpd.GeoDataFrame]) – Path to ground truth labels or frame with labels

• nproc (int, optional) – Number of parallel processes to use. Defaults to 1.

• valid_class_ids (Union[str, list]) – Valid class IDs in ground_truth_labels. Defaults to ‘all’ (use all classes).

• class_column_name (str) – Column name of class ID in ground_truth_labels.

• crs (str) – Coordinate reference system. Defaults to EPSG:25832.

• area_max (int) – Maximum area of polygons to consider. Defaults to None.

• area_min (int) – Minimum area of polygons to consider. Defaults to 3.

_abc_impl = <_abc._abc_data object>

process(input_file)

process Create raster distance_transform from input tile

The distance transform is normalized per polygon (better for instance segmentation).

Parameters:

input_file (str) – input raster file

class deeptrees.dataloading.preprocessing.GroundTruthGenerator(rasters, output_path, output_file_prefix, ground_truth_labels, valid_class_ids='all', class_column_name='class', crs='EPSG:25832', nproc=1)

Bases: ABC

Base class to generate ground truth (masks, outlines, distance transforms)

Loads a raster and a vector file, then rasterizes the vector file within the extent of the raster with the same resolution. Uses gdal_rasterize under the hood, but provides some more features like specifying which classes to rasterize into which layer of the output. If you want to infer the output file names, the input file name suffixes have to be delimited by an ‘_’.”,

Based on scripts/rasterize.py in Freudenberg 2022.

__init__(rasters, output_path, output_file_prefix, ground_truth_labels, valid_class_ids='all', class_column_name='class', crs='EPSG:25832', nproc=1)

Parameters:

• rasters (Union[str, list]) – (List of) file path(s) to the raster files

• output_path (str) – Output directory

• output_file_prefix (str) – Output file prefix. Suffix is infered from raster files.

• ground_truth_labels (Union[str, gpd.GeoDataFrame]) – Path to ground truth labels or frame with labels

• nproc (int, optional) – Number of parallel processes to use. Defaults to 1.

• valid_class_ids (Union[str, list]) – Valid class IDs in ground_truth_labels. Defaults to ‘all’ (use all classes).

• class_column_name (str) – Column name of class ID in ground_truth_labels.

• crs (str) – Coordinate reference system. Defaults to EPSG:25832.

_abc_impl = <_abc._abc_data object>

apply_process()

Apply the processing function in parallel.

constrain_geometry_to_tile(input_file)

constrain_geometry_to_raster

Filter the ground truth labels that fall within the bounding box of the given raster image.

Parameters:

input_file (str) – path to input raster file

Returns:

list of polygons within tile bounding box

Return type:

list[Polygon]

output_filename(input_file)

setup_process

• Construct output file name from input file name

Parameters:

input_file (str) – input file

Returns:

output file

Return type:

str

process(input_file)

Process function that works on one tile. Needs to be defined in subclass.

Parameters:

input_file (str) – Path to input input_file file.

class deeptrees.dataloading.preprocessing.MaskOutlinesGenerator(rasters, output_path, output_file_prefix, ground_truth_labels, valid_class_ids='all', class_column_name='class', crs='EPSG:25832', nproc=1, generate_outlines=False)

Bases: GroundTruthGenerator

Generate masks and outlines from tiles and ground truth labels.

__init__(rasters, output_path, output_file_prefix, ground_truth_labels, valid_class_ids='all', class_column_name='class', crs='EPSG:25832', nproc=1, generate_outlines=False)

Initialize the MaskOutlinesGenerator instance.

Parameters:

• rasters (Union[str, list]) – (List of) file path(s) to the raster files

• output_path (str) – Output directory

• output_file_prefix (str) – Output file prefix. Suffix is infered from raster files.

• ground_truth_labels (Union[str, gpd.GeoDataFrame]) – Path to ground truth labels or frame with labels

• nproc (int, optional) – Number of parallel processes to use. Defaults to 1.

• valid_class_ids (Union[str, list]) – Valid class IDs in ground_truth_labels. Defaults to ‘all’ (use all classes).

• class_column_name (str) – Column name of class ID in ground_truth_labels.

• crs (str) – Coordinate reference system. Defaults to EPSG:25832.

• generate_outlines (bool) – If True, generate outlines. If False, generate masks. Defaults to False.

_abc_impl = <_abc._abc_data object>

process(input_file)

process Create raster mask/outline from input tile

Parameters:

input_file (str) – input raster file

Contents