fatf.utils.models.processing.batch_data

fatf.utils.models.processing.batch_data(data: numpy.ndarray, batch_size: int = 50, transformation_fn: Callable = None) → numpy.ndarray

Slices data into batches and returns then sequentially.

New in version 0.1.1.

Since some data may be too large to fit into memory as whole, this function slices them into batches and yields them sequentially. If desired, each batch can be processed by transformation_fn prior to returning it.

Parameters

datanumpy.ndarray

A two dimensional numpy array (either classic or structured) to be sliced into batches.

batch_sizeinteger, optional (default=50)

The size (number of rows) of each batch.

transformation_fncallable, optional (default=None)

A callable object to apply to each batch before returning it. It must have exactly one required parameter.

Yields

slicenumpy.ndarray

A slice of data.

Raises

IncorrectShapeError

The data array is not 2-dimensional.

RuntimeError

The transformation function does not have exactly one required parameter.

TypeError

The batch_size is not an integer or the transformation_fn is not a callable object.

ValueError

The batch_size is smaller than 1.