Developers Guide¶

In this guide you will find all the information you need to contribute high quality code and documentation to the package.

Communication¶

We use a range of platforms for communication in the project:

for issues with the source code or the documentation please open an issue on our GitHub issue tracker;
the code-related development discussion should happen on our gitter channel;
the discussion about the project’s future and the direction of the development happens on our slack channel and mailing list.

Contributing Code¶

There is a number of steps that any new code or documentation patch has to go through before being merged into the package. All of these are described below.

Installing Development Requirements¶

The python tools and packages used for the development and testing of the code need to be installed before proceeding. In order to run all of the tests all: the required package dependencies, the auxiliary package dependencies and the development package dependencies need to be installed.

Files with these requirements can be found in the root folder of the package repository under the following names:

requirements.txt – the dependencies required by the package;
requirements-aux.txt – the soft dependencies of the package (cf. Soft Dependencies); and
requirements-dev.txt – the development dependencies of the package.

All of these can be installed directly from these files with

$ pip install -r <dependencies file name>

or alongside the package with

$ pip install -e fat-forensics[all,dev]

given that your current directory is the root directory of the package. (See the section below for more information.)

Installing the Package¶

When developing code for the package we advise to install it as an editable copy directly from sources placed in the dev branch. To achieve that you need to first clone our git repository and then install FAT Forensics as an editable package alongside all the required dependencies. To this end, please execute the four commands shown below.

$ git clone https://github.com/fat-forensics/fat-forensics.git
$ cd fat-forensics
$ git checkout dev
$ pip install -e '.[all,dev]'

Note

Consider using pyenv and pyenv-virtualenv plugin to manage your Python versions and virtual environments. It makes the development so much easier.

If you want to install the latest stable version from sources instead, please use the master branch instead.

Testing Environment¶

We develop the package so that it is compatible with Python versions 3.5 and up. The specific versions of the package dependencies are listed in the three requirement files listed in the Installing Development Requirements section.

All of the test and build commands are held in the Makefile, which is placed in the root directory of the package. Please consult this file for the specific commands that are executed when building and testing the package.

Our Continuous Integration (CI) is based on GitHub Actions, with the configuration files held in the .github/workflows directory.

Note

Random number generator

Since some of the package’s tests sample random numbers or are influenced by “randomness” of its dependencies, the package allows to fix the random seed of both Python’s random and numpy’s numpy.random random number generation modules. This can be achieved by setting the FATF_SEED system variable to a selected number. (For tests this is set to 42 in the Makefile.) If set, the fatf.setup_random_seed function, which is called upon importing the FAT Forensics package, will set both of the aforementioned random seeds to the desired value.

If anywhere in the tests you wish to restore it to the desired value, calling the fatf.setup_random_seed function should suffice.

Code Formatting¶

When writing the code we try to follow the Google Python Style Guide. Code formatting adherence to this guideline can be checked with Yet Another Python Formater (YAPF) by executing:

$ make linting-yapf

which will highlight what needs to be changed rather than reformat the code automatically. The configuration of YAPF for the package can be found in the .style.yapf file in the root directory of the package.

Code formatting is also checked with Pylint and Flake8. These can be executed with the following two commands:

$ make linting-pylint
$ make linting-flake8

The configuration of both these linters can be found in .pylintrc and .flake8 files respectively, both placed in the root directory of the package.

To help the contributors adhere to the formatting style of the code, documentation and configuration files we use EditorConfig. By installing the EditorConfig plugin for your code editor the style of the new content that you author will automatically adhere to some of our coding style. You can find the configuration file of the EditorConfig – .editorconfig – in the root directory of the package.

Type Hints¶

We try to annotate the code in the package with type hints whenever possible. The typing of the code is checked statically with mypy. Our mypy configuration file – .mypy.ini – is placed in the root directory of the package and the type checking is performed by executing the following line of code:

$ make check-types

Testing¶

We run tests on the package itself as well as on the code snippets spread throughout the documentation. To this end, we use pytest configured with the pytest.ini file kept in the root directory of the package.

To gather code coverage statistics we use pytest-cov plugin with its partial configuration placed in the .coveragerc file kept in the root directory of the package.

Code¶

The unit tests for the package are held in directories named tests created separately for each module. To test the code you can execute:

$ make test

and to get the code coverage:

$ make code-coverage

However, we recommend to execute both these steps at once to save time by using:

$ make test-with-code-coverage

Note

The fatf.utils.testing module holds a range of functions that are useful for the unit tests. If you find yourself reusing a piece of code in multiple places in the unit tests, please consider making it a part of this module.

Documentation¶

There are three different tests run on the documentation. The first one checks validity of links in the documentation and is run with:

$ make doc-linkcheck

The second one checks which Python objects, methods and functions are not documented (documentation coverage) and can be run with:

$ make doc-coverage

Finally, the code snippets spread throughout the documentation are run to test whether their output agrees with the one provided in the documentation. These tests can be run with:

$ make test-doc

Note

Because of incompatibility of vanila (pytest) doctest and sphinx doctest we are using pure doctest syntax, i.e. no group annotations are possible.

Step by Step Guide¶

To summarise, the following commands should be executed to fully test the package (cf. .github/workflows/tests.yml for more details):

$ make linting-pylint
$ make linting-flake8
$ make linting-yapf

$ make check-types

$ make test-with-code-coverage

$ make doc-linkcheck
$ make doc-coverage
$ make test-doc

Contributing Documentation¶

To understand the documentation structure and the intention of each section in the documentation please familiarise yourself with the Getting Started part of the documentation.

Code Documentation¶

For building the documentation we use Sphinx with a custom theme. The API is documented using numpydoc docstring syntax and structure. The code example galleries are generated with the sphinx-gallery extension.

To build the documentation you can execute:

$ make doc-html

In addition to this command being available in the main Makefile of the project, there is a documentation-specific Makefile in the doc directory that supports the following documentation build command:

$ make html

Since some of the code snippets (in particular the ones placed in the tutorials) produce plots and figures that are later included in the documentation, these need to be executed first. To this end, the documentation tests (make test-doc) has to be executed before building the documentation.

Warning

Since the .rst files describing the API documentation are generated automatically with sphinx’s autosummary extension and placed in the doc/generated directory some of the changes that are made to the API template or documentation may not trigger the automatic rebuilding of the generated files. In such cases the doc/generated directory has to be cleaned. This can be achieved with the documentation-specific Makefile via the following command:

$ make doc-clean

User Guide¶

In addition to the FAT Forensics package-oriented documentation we also maintain a User Guide that describes Fairness, Accountability and Transparency approaches on a more theoretical level. Entries in the User Guide should try to follow a specific fact-oriented pattern. When contributing please try to adhere to the style of the entries that are already in the User Guide as much as possible. At a minimum please provide the following fields in the method description placed in the User Guide:

Name.
Literature reference (BibTeX).
List of implementations (both standalone implementations and implementations in packages and libraries).
- Programming language.
- Implementation URL (possibly GitHub).
- Licence.
- Fairness, Accountability, Transparency or * (other and related) field.
Is is a Metric (measure) or an Application (mitigation) technique.
Is it Model Dependent (what are the applicable models) or Model Agnostic.
Is it Post-Hoc or Ante-Hoc.

Pull Requests and Issues¶

When making a pull request on GitHub please use the provided pull request template and make sure that you comply with all the requirements listed therein. All of the pull requests should be made against the dev branch (please note that by default they will be created against the master branch). Furthermore, please have a browse through other pull requests and issues to locate all the problems/solutions similar to yours.

Similarly, we have issue templates. Please use them (whenever possible) while opening a new issue.

Package Structure¶

All the details of the package structure can be learnt from the API documentation. However, we include a short summary below for completeness.

Fairness¶

`fatf.fairness`	The `fatf.fairness` module implements a variety of fairness algorithms.
`fatf.fairness.data`	The `fatf.fairness.data` module implements fairness algorithms for data.
`fatf.fairness.models`	The `fatf.fairness.models` module holds fairness algorithms for models.
`fatf.fairness.predictions`	The `fatf.fairness.predictions` module houses fairness for predictions.

Accountability¶

`fatf.accountability`	The `fatf.accountability` module holds a range of accountability methods.
`fatf.accountability.data`	The `fatf.accountability.data` module implements accountability algorithms for data.
`fatf.accountability.models`	The `fatf.accountability.models` module holds accountability algorithms for models.

Transparency¶

`fatf.transparency`	The `fatf.transparency` module implements various transparency algorithms.
`fatf.transparency.data`	The `fatf.transparency.data` module holds data transparency algorithms.
`fatf.transparency.models`	The `fatf.transparency.models` module holds models transparency functions.
`fatf.transparency.predictions`	The `fatf.transparency.predictions` module holds predictions transparency functions.

Visualisations¶

fatf.vis

The fatf.vis module implements visualisations of various FAT methods.

Utilities¶

`fatf.utils`	The `fatf.utils` module includes various utilities.
`fatf.utils.data`	The `fatf.utils.data` module holds data tools and data sets.
`fatf.utils.models`	The `fatf.utils.models` module holds models tools and utilities.
`fatf.utils.transparency`	New in version 0.0.2.
`fatf.utils.array`	The `fatf.utils.array` module holds array tools and validation functions.
`fatf.utils.distances`	The `fatf.utils.distances` module holds a variety of distance metrics.
`fatf.utils.kernels`	New in version 0.0.2.
`fatf.utils.metrics`	The `fatf.utils.metrics` module holds predictive performance metrics.
`fatf.utils.tools`	The `fatf.utils.tools` module implements general tools for the package.
`fatf.utils.testing`	The `fatf.utils.testing` module holds functions used across unit tests.
`fatf.utils.validation`	New in version 0.0.2.

Package Resources¶

The documentation is build on top of the Bootstrap (v4.3.1) and jQuery (3.4.1) libraries. The custom theme is based on Sphinx’s nature theme (commit hash 1b1ebd2; 2nd January 2019).

The “Fork me on GitHub” ribbon is based on the CSS solution written by codepo8.

The package icons were created with Amazon Alexa Icon Builder:

FAT Forensics icon.
- Style.
  - Size: Maximal size.
  - Type: Gradient.
  - RGB: 40, 40, 40.
  - Angle: 0.
- Background.
  - Type: Gradient.
  - RGB: 185, 186, 70.
  - Angle: 0.
- Border.
  - Type: Solid.
  - RGB: 166, 153, 134.
Fairness, Accountability and Transparency icons.
- Style.
  - Size: Maximal size.
  - Type: Gradient.
  - RGB: 40, 40, 40.
  - Angle: 0.
- Background.
  - Type: Gradient.
  - RGB: 185, 186, 70.
  - Angle: 135.
- Border.
  - Type: Solid.
  - RGB: 166, 153, 134.