API Reference (0.0.2)

This is the class and function reference of fat-forensics. Please refer to the full user guide for further details, as the class and function raw specifications may not be enough to give full guidelines on their uses.

Note

The package is designed to work with both classic and structured numpy arrays. The latter is introduced to help manage numpy arrays holding vanila categorical features. Please see the Measuring Fairness of a Data Set and Measuring Fairness of a Predictive Model – Disparate Impact examples to see how the package can be used with a structured numpy array.

FAT-Forensics

FAT-Forensics is a Python module integrating a variety of fairness, accountability (security, privacy) and transparency (explainability, interpretability) approaches to assess social impact of artificial intelligence systems.


fatf.fairness: Fairness

The fatf.fairness module implements a variety of fairness algorithms.

This module holds a variety of techniques that can be used to assess fairness of artificial intelligence pipelines and the machine learning process: data (fatf.fairness.data), models (fatf.fairness.models) and predictions (fatf.fairness.predictions).

fatf.fairness.data: Fairness for Data

The fatf.fairness.data module implements fairness algorithms for data.

measures.systemic_bias

Checks for systemic bias in a dataset.

measures.systemic_bias_check

Indicates whether a dataset has a systemic bias.

fatf.fairness.models: Fairness for Models

The fatf.fairness.models module holds fairness algorithms for models.

measures.disparate_impact

Calculates selected disparate impact grid for a data set.

measures.disparate_impact_indexed

Calculates selected disparate impact grid for indexed data.

measures.disparate_impact_check

Checks if any sub-population pair violates chosen disparate impact measure.

measures.demographic_parity

Checks for demographic parity between all of the sub-populations.

measures.equal_opportunity

Checks for equal opportunity between all of the sub-populations.

measures.equal_accuracy

Checks if accuracy difference of all grouping pairs is within tolerance.

fatf.fairness.predictions: Fairness for Predictions

The fatf.fairness.predictions module houses fairness for predictions.

measures.counterfactual_fairness

Checks counterfactual fairness of a prediction given a model.

measures.counterfactual_fairness_check

Checks for counterfactual fairness using a counterfactual fairness arrays.

fatf.accountability: Accountability

The fatf.accountability module holds a range of accountability methods.

This module holds a variety of techniques that can be used to assess privacy, security and robustness of artificial intelligence pipelines and the machine learning process: data, models and predictions.

fatf.accountability.data: Accountability for Data

The fatf.accountability.data module implements accountability algorithms for data.

measures.sampling_bias

Computes information needed for evaluating and remedying sampling bias.

measures.sampling_bias_indexed

Computes information needed for evaluating and remedying sampling bias.

measures.sampling_bias_grid_check

Checks for a pairwise sampling bias based on the provided threshold.

measures.sampling_bias_check

Checks for a pairwise sampling bias based on the provided threshold.

fatf.accountability.models: Accountability for Models

The fatf.accountability.models module holds accountability algorithms for models.

measures.systematic_performance_bias

Checks for a systematic bias in provided predictive performance values.

measures.systematic_performance_bias_grid

Checks for pairwise systematic bias in group-wise predictive performance.

fatf.transparency: Transparency

The fatf.transparency module implements various transparency algorithms.

This module holds a variety of techniques that can be used to assess explainability and interpretability of artificial intelligence pipelines and the machine learning process: data, models and predictions.

fatf.transparency.data: Transparency for Data

The fatf.transparency.data module holds data transparency algorithms.

describe_functions.describe_array

Describes categorical (textual) and numerical columns in the input array.

describe_functions.describe_numerical_array

Describes a numerical numpy array with basic statistics.

describe_functions.describe_categorical_array

Describes a categorical numpy array with basic statistics.

fatf.transparency.models: Transparency for Models

The fatf.transparency.models module holds models transparency functions.

fatf.transparency.models.feature_influence: Feature Influence

The fatf.transparency.models.feature_influence module holds functions for calculating feature influence for predictive models.

This module implements Partial Dependence (PD) and Individual Conditional Expectation (ICE) – model agnostic feature influence measurements.

feature_influence.individual_conditional_expectation

Calculates Individual Conditional Expectation for a selected feature.

feature_influence.merge_ice_arrays

Merges multiple Individual Conditional Expectation arrays.

feature_influence.partial_dependence_ice

Calculates Partial Dependence based on Individual Conditional Expectations.

feature_influence.partial_dependence

Calculates Partial Dependence for a selected feature.

fatf.transparency.models.lime: LIME for Models

Deprecated since version 0.0.3: This module will be deprecated in FAT Forensics version 0.0.3. Instead of wrapping the lime package a full (modular) version of the LIME surrogate explainer has been implemented – see the fatf.transparency.predictions.surrogate_explainers.TabularBlimeyLime class and the How to build LIME yourself (bLIMEy) – Surrogate Tabular Explainers how-to guide for more details.

The fatf.transparency.models.lime module prepares the LIME explainer for explaining models.

This module instantiates fatf.transparency.lime.Lime class with selected arguments to create a Lime class explainer for a whole model.

lime.Lime

A sub-class of the Lime class designated for explaining a whole model.

fatf.transparency.predictions: Transparency for Predictions

The fatf.transparency.predictions module holds predictions transparency functions.

fatf.transparency.predictions.counterfactuals: Counterfactual Explainers

The fatf.transparency.predictions.counterfactuals module implements counterfactual explainers for predictions.

counterfactuals.CounterfactualExplainer

Generates counterfactual explanations of black-box classifier predictions.

counterfactuals.textualise_counterfactuals

Translates a counterfactuals array into a textual description.

fatf.transparency.predictions.surrogate_explainers: Surrogate Explainers

New in version 0.0.2.

The fatf.transparency.predictions.surrogate_explainers module implements example surrogate explainers.

Guidelines and tips for building custom surrogate explainers for various types of data (tabular, image and text) can be found in the How-to Guides part of the documentation:

The scikit-learn package is required for the surrogate tree and linear model (LIME) explainers to work. When importing this module with scikit-learn missing, the user will only be warned about disabling this functionality not being available and the two explainers will be disabled to allow using the abstract SurrogateTabularExplainer without the scikit-learn package installed.

surrogate_explainers.SurrogateTabularExplainer

An abstract parent class for implementing surrogate explainers.

surrogate_explainers.TabularBlimeyLime

A tabular LIME explainer – a surrogate explainer based on a linear model.

surrogate_explainers.TabularBlimeyTree

A surrogate explainer based on a decision tree.

fatf.transparency.predictions.lime: LIME for Predictions

Deprecated since version 0.0.3: This module will be deprecated in FAT Forensics version 0.0.3. Instead of wrapping the lime package a full (modular) version of the LIME surrogate explainer has been implemented – see the fatf.transparency.predictions.surrogate_explainers.TabularBlimeyLime class and the How to build LIME yourself (bLIMEy) – Surrogate Tabular Explainers how-to guide for more details.

The fatf.transparency.predictions.lime module prepares the LIME explainer for explaining predictions.

This module instantiates fatf.transparency.lime.Lime class with selected arguments to create a Lime class explainer for a single prediction.

lime.Lime

A sub-class of the Lime class designated for explaining a prediction.

fatf.transparency.lime: LIME Wrapper

Deprecated since version 0.0.3: This module will be deprecated in FAT Forensics version 0.0.3. Instead of wrapping the lime package a full (modular) version of the LIME surrogate explainer has been implemented – see the fatf.transparency.predictions.surrogate_explainers.TabularBlimeyLime class and the How to build LIME yourself (bLIMEy) – Surrogate Tabular Explainers how-to guide for more details.

The fatf.transparency.lime module wraps the LIME explainer.

This module implements a generic tabular data LIME explainer that can be customised to either explain models (fatf.transparency.models.lime module) or predictions (fatf.transparency.predictions.lime module).

This module requires the lime package to be installed.

(At the moment, only the tabular data explainer is wrapped. Since LIME is slowly being reimplemented withing this package, wrapping anything else would be a waste of time.)

lime.Lime

Wraps LIME package’s lime.lime_tabular.LimeTabularExplainer class.

fatf.transparency.sklearn: Scikit-learn Explainers

The fatf.transparency.sklearn module implements scikit-learn explainers.

This module requires the scikit-learn package to be installed.

fatf.transparency.sklearn.tools: Scikit-learn Explainer Tools

New in version 0.0.2.

The fatf.transparency.sklearn.tools module implements a base scikit-learn explainer.

tools.SKLearnExplainer

Implements a base scikit-learn model explainer class.

tools.is_sklearn_model

Checks whether a class instance or a class is a scikit-learn predictor.

tools.is_sklearn_model_instance

Checks whether a class instance (object) is a scikit-learn predictor.

fatf.transparency.sklearn.linear_model: Linear Model Explainers

New in version 0.0.2.

The fatf.transparency.sklearn.linear_model module implements linear scikit-learn model explainers.

linear_model.SKLearnLinearModelExplainer

A scikit-learn linear model explainer class.

linear_model.linear_classifier_coefficients

Extracts coefficients (feature importances) of a linear scikit-learn model.

fatf.vis: Visualisations

The fatf.vis module implements visualisations of various FAT methods.

This module requires the matplotlib package to be installed.

feature_influence.plot_individual_conditional_expectation

Plots Individual Conditional Expectation for a selected class.

feature_influence.plot_partial_dependence

Plots Partial Dependence for a selected class.

lime.plot_lime

Plots a LIME explanation.

fatf.exceptions: Exceptions, Errors and Warnings

The fatf.exceptions module holds custom exceptions, errors and warnings.

FATFException

Base class for FAT-Forensics exceptions (inherits from Exception).

IncorrectShapeError

Exception raised when the shape of an array is not what is expected.

IncompatibleExplainerError

Exception raised when an explainer lacks desired functionality.

IncompatibleModelError

Exception raised when a model lacks desired functionality.

UnfittedModelError

Exception raised when a model is unfitted and a fitted one is expected.

PrefittedModelError

Exception raised when a model is fitted and an unfitted one is expected.

fatf.utils: Utilities

The fatf.utils module includes various utilities.

Developer guide: See the Developers Guide page for further details.

Base classes

data.augmentation.Augmentation

An abstract class for implementing data augmentation methods.

models.models.Model

An abstract class used to implement predictive models.

fatf.utils.array: Array Utilities

The fatf.utils.array module holds array tools and validation functions.

tools.indices_by_type

Identifies indices of columns with numerical and non-numerical values.

tools.get_invalid_indices

Returns a numpy array with column indices that the input array is missing.

tools.are_indices_valid

Checks whether all the input indices are valid for the input array.

tools.generalise_dtype

Finds the more general type of the two given.

tools.structured_to_unstructured_row

Calls either local or numpy’s structured_to_unstructured function.

tools.structured_to_unstructured

Calls either local or numpy’s structured_to_unstructured function.

tools.as_unstructured

Converts an array like object into an unstructured array.

validation.is_numerical_dtype

Determines whether a numpy dtype object is of numerical type.

validation.is_textual_dtype

Determines whether a numpy dtype object is of textual type.

validation.is_base_dtype

Determines whether a numpy dtype object is one of base types.

validation.is_flat_dtype

Determines whether a numpy dtype object is flat.

validation.are_similar_dtypes

Checks whether two numpy dtypes are similar.

validation.are_similar_dtype_arrays

Determines whether two numpy array-like object have a similar data type.

validation.is_numerical_array

Determines whether a numpy array-like object has a numerical data type.

validation.is_textual_array

Determines whether a numpy array-like object has a textual data type.

validation.is_base_array

Determines whether a numpy array-like object holds base data types.

validation.is_1d_array

Determines whether a numpy array-like object is 1-dimensional.

validation.is_2d_array

Determines whether a numpy array-like object has 2 dimensions.

validation.is_structured_row

Determines whether the input is a structured numpy array’s row object.

validation.is_1d_like

Checks if the input is either a 1D numpy array or a structured numpy row.

validation.is_structured_array

Determines whether a numpy array-like object is a structured array.

fatf.utils.data: Data Utilities

The fatf.utils.data module holds data tools and data sets.

fatf.utils.data.datasets: Data Sets

The fatf.utils.data.datasets module holds examples of data sets.

The iris data set is returned as a classic numpy array, whereas the health records data set is a structured numpy array.

datasets.load_data

Loads a dataset from a file.

datasets.load_health_records

Loads in a fake health records dataset.

datasets.load_iris

Loads the IRIS dataset [R47dce3bbc9de-FISHER1936].

fatf.utils.data.tools: Data Set Tools

The fatf.utils.data.tools module implements tools for data set handling.

tools.group_by_column

Groups row indices of an array based on value grouping of a chosen column.

tools.apply_to_column_grouping

Applies a function to the specified groups of labels and predictions.

tools.validate_indices_per_bin

Validates a list of binned indices.

tools.validate_binary_matrix

Validates a binary, square and symmetric numpy array.

fatf.utils.data.augmentation: Data Set Augmentation

The fatf.utils.data.augmentation module implements data set augmenters.

augmentation.Augmentation

An abstract class for implementing data augmentation methods.

augmentation.NormalSampling

Sampling data from a normal distribution.

augmentation.TruncatedNormalSampling

Sampling data from a truncated normal distribution.

augmentation.Mixup

Sampling data with the Mixup method.

augmentation.NormalClassDiscovery

Sampling data to discover instances spanning all the possible classes.

augmentation.DecisionBoundarySphere

Sampling data in a hyper-sphere around the closest decision boundary.

augmentation.LocalSphere

Sampling data in a hyper-sphere around the selected data point.

fatf.utils.data.instance_augmentation: Data Point Augmentation

New in version 0.0.2.

The fatf.utils.data.instance_augmentation module implements various augmentation function for 1-dimensional numpy array-like objects.

instance_augmentation.binary_sampler

Samples non-zero elements of the binary data_row array uniformly.

fatf.utils.data.transformation: Data Set Transformation

New in version 0.0.2.

The fatf.utils.data.transformation module holds data transformation functions.

transformation.dataset_row_masking

Creates a binary representation of the dataset by masking its rows.

fatf.utils.data.density: Data Set Density Checking and Estimation

The fatf.utils.data.density module implements data density estimators.

density.DensityCheck

Checks and scores density in the data_set and for new data points.

fatf.utils.data.discretisation: Data Set Discretisation

New in version 0.0.2.

The fatf.utils.data.discretisation module implements data discretisation approaches.

discretisation.Discretiser

An abstract class that all discretiser implementations should inherit from.

discretisation.QuartileDiscretiser

Discretises selected numerical features of the dataset into quartiles.

fatf.utils.data.feature_selection: Data Set Feature Selection

New in version 0.0.2.

The fatf.utils.data.feature_selection module holds feature selection tools.

fatf.utils.data.feature_selection.sklearn: sklearn Feature Selection

New in version 0.0.2.

The fatf.utils.data.feature_selection.sklearn module implements scikit-learn-based feature selection approaches.

sklearn.lasso_path

Selects the specified number of features based on Lasso path coefficients.

fatf.utils.models: Models Utilities

The fatf.utils.models module holds models tools and utilities.

fatf.utils.models.models: Example Models

The fatf.utils.models.models module holds custom models.

The models implemented in this module are mainly used for used for FAT-Forensics package testing and the examples in the documentation.

models.KNN

A K-Nearest Neighbours model based on Euclidean distance.

fatf.utils.models.validation: Model Validation Tools

The fatf.utils.models.validation module validates models functionality.

This module holds functions responsible for validating models functionality across the FAT-Forensics package.

validation.check_model_functionality

Checks whether a model object has all the required functionality.

fatf.utils.distances: Distances and Distance Utilities

The fatf.utils.distances module holds a variety of distance metrics.

The distance metrics and tools implemented in this module are mainly used for the fatf.utils.models.models.KNN model implementation, to measure distance (and similarity) of data points for various functions in this package as well as for documentation examples and testing.

get_distance_matrix

Computes a distance matrix (2-D) between all rows of the data_array.

get_point_distance

Computes the distance between a data point and an array of data.

euclidean_distance

Calculates the Euclidean distance between two 1-dimensional numpy “arrays”.

euclidean_point_distance

Calculates the Euclidean distance between y and every row of X.

euclidean_array_distance

Calculates the Euclidean distance matrix between rows in X and Y.

hamming_distance_base

Calculates the Hamming distance between two strings x and y.

hamming_distance

Computes the Hamming distance between 1-dimensional non-numerical arrays.

hamming_point_distance

Calculates the Hamming distance between y and every row of X.

hamming_array_distance

Calculates the Hamming distance matrix between rows in X and Y.

binary_distance

Computes the binary distance between two 1-dimensional arrays.

binary_point_distance

Calculates the binary distance between y and every row of X.

binary_array_distance

Calculates the binary distance matrix between rows in X and Y.

check_distance_functionality

Checks whether a distance function takes exactly 2 required parameters.

fatf.utils.kernels: Kernel Functions and Kernel Utilities

New in version 0.0.2.

The fatf.utils.kernels module holds distance transformation kernels.

The kernel functions implemented by this module are mainly used to transform a distance into a similarity measure. One of their applications is to weight training data samples when training a predictive model based on their similarity (closeness) to a selected data point.

exponential_kernel

Applies an exponential kernel to an array of distances.

check_kernel_functionality

Checks whether a kernel function has exactly one required parameter.

fatf.utils.metrics: Performance Metrics and Utilities

The fatf.utils.metrics module holds predictive performance metrics.

This module implements confusion matrix calculation and manipulation, predictive performance measures and per-population performance metrics.

fatf.utils.metrics.metrics: Basic Metrics

The fatf.utils.metrics.metrics module holds common performance metrics.

metrics.multiclass_true_positive_rate

Calculates the “true positive rate” for a multi-class confusion matrix.

metrics.multiclass_true_negative_rate

Calculates the “true negative rate” for a multi-class confusion matrix.

metrics.multiclass_false_positive_rate

Calculates the “false positive rate” for a multi-class confusion matrix.

metrics.multiclass_false_negative_rate

Calculates the “false negative rate” for a multi-class confusion matrix.

metrics.true_positive_rate

Calculates the true positive rate for a binary confusion matrix.

metrics.true_negative_rate

Calculates the true negative rate for a binary confusion matrix.

metrics.false_positive_rate

Calculates the false positive rate for a binary confusion matrix.

metrics.false_negative_rate

Calculates the false negative rate for a binary confusion matrix.

metrics.multiclass_positive_predictive_value

Gets the “positive predictive value” for a multi-class confusion matrix.

metrics.multiclass_negative_predictive_value

Gets the “negative predictive value” for a multi-class confusion matrix.

metrics.positive_predictive_value

Calculates the positive predictive value for a binary confusion matrix.

metrics.negative_predictive_value

Calculates the negative predictive value for a binary confusion matrix.

metrics.accuracy

Computes the accuracy for an arbitrary confusion matrix.

metrics.multiclass_treatment

Computes the “treatment” metric for a multi-class confusion matrix.

metrics.treatment

Computes the “treatment” metric for a binary confusion matrix.

fatf.utils.metrics.subgroup_metrics: Metrics for Sub-Populations

The fatf.utils.metrics.subgroup_metrics module holds sub-group metrics.

These functions are mainly used to compute a given performance metric for every sub population in a data set defined by a grouping on a selected feature.

subgroup_metrics.apply_metric_function

Applies the provided performance metric to every confusion matrix.

subgroup_metrics.apply_metric

Applies one of the predefined performance metric to all confusion matrices.

subgroup_metrics.performance_per_subgroup

Computes a chosen metric per sub-population for a data set.

subgroup_metrics.performance_per_subgroup_indexed

Computes a chosen metric per sub-population for index-based grouping.

fatf.utils.metrics.tools: Metric Tools

The fatf.utils.metrics.tools module holds tools for performance metrics.

These tools – mainly confusion matrix computation and manipulation – are useful for constructing new performance metrics. Examples of how they are used can be found in fatf.utils.metrics.metrics module.

tools.get_confusion_matrix

Computes a confusion matrix based on predictions and ground truth vectors.

tools.confusion_matrix_per_subgroup

Computes confusion matrices for every defined sub-population.

tools.confusion_matrix_per_subgroup_indexed

Computes confusion matrices for every defined sub-population.

tools.validate_confusion_matrix

Validates a confusion matrix.

tools.validate_confusion_matrix_size

Validates the exact shape of the confusion matrix.

fatf.utils.transparency: Transparency Utilities

New in version 0.0.2.

The fatf.utils.transparency module holds transparency utilities.

fatf.utils.transparency.explainers: Explainer Utilities

New in version 0.0.2.

The fatf.utils.transparency.explainers module holds utilities for building custom explainer objects.

explainers.Explainer

A base class for any explainer object implemented in the package.

explainers.check_instance_explainer_functionality

Checks whether an explainer object can explain a data point (instance).

fatf.utils.transparency.surrogate_evaluation: Surrogates Evaluation

New in version 0.0.2.

The fatf.utils.transparency.surrogate_evaluation module implements various surrogate model evaluation measures.

surrogate_evaluation.local_fidelity_score

Computes local fidelity between a global and a local (surrogate) model.

fatf.utils.tools: FAT-Forensics Tools

The fatf.utils.tools module implements general tools for the package.

at_least_verion

Checks if the package_version satisfies the minimum_requirement.

fatf.utils.validation: FAT-Forensics Validation Functions

New in version 0.0.2.

The fatf.utils.validation module validates functions and objects.

This module holds functions responsible for validating generic functions and objects implemented across the FAT-Forensics package.

get_required_parameters_number

Checks if a callable object has the correct number of required parameters.

check_object_functionality

Checks if an object has specified methods with given number of parameters.

fatf.utils.testing: Testing Utilities

The fatf.utils.testing module holds functions used across unit tests.

Classes

transparency.InvalidModel

An invalid model class – it does not implement a predict_proba method.

transparency.NonProbabilisticModel

A model that is not probabilistic – no predict_proba function.

Functions

imports.module_import_tester

Provides a context for testing imports of installed and missing modules.

transparency.is_explanation_equal_dict

Tests if the two dictionaries of a given structure are equal.

transparency.is_explanation_equal_list

Tests if the two dictionaries of a given structure are equal.

vis.get_plot_data

Extracts plot’s title, x-axis name and range and y-axis name and range.

vis.get_line_data

Extracts line’s data array, colour, alpha channel, label and width.

vis.get_bar_data

Extracts plot’s title, x-axis name and range and y-axis name and range.

warnings.handle_warnings_filter_pattern

Converts a patter within a warning filter into a regex pattern.

warnings.set_default_warning_filters

Sets the warning filters to default values (as of Python 3.7).

warnings.is_warning_class_displayed

Checks whether a warning of a given class will be shown to the user.

fatf: FAT Forensics Initialisation Functions

This API documentation section describes set-up functionality of the FAT Forensics package.

setup_warning_filters

Sets up desired warning filters.

setup_random_seed

Sets up Python’s and numpy’s random seed.