How can you determine if a neural network is fair to all groups?

1 Fairness definitions

There is no universal agreement on what constitutes fairness in machine learning, but there are some common definitions that capture different aspects of the problem. For example, one definition is demographic parity, which means that the outcomes of a neural network are independent of the protected attributes of the groups, such as race, gender, or age. Another definition is equal opportunity, which means that the true positive rates of a neural network are equal across the groups, or that the groups have equal chances of getting favorable outcomes. A third definition is equalized odds, which means that both the true positive and false positive rates of a neural network are equal across the groups, or that the groups have equal chances of getting accurate outcomes.

2 Fairness metrics

To measure how well a neural network satisfies a fairness definition, you need to use some fairness metrics. These are numerical indicators that compare the outcomes of a neural network for different groups, and quantify the degree of disparity or discrimination. For example, one fairness metric is the statistical parity difference, which measures the difference in the probabilities of positive outcomes between the groups. Another fairness metric is the equal opportunity difference, which measures the difference in the true positive rates between the groups. A third fairness metric is the average odds difference, which measures the average of the differences in the true positive and false positive rates between the groups.

3 Fairness techniques

To achieve fairness in a neural network, you need to use some fairness techniques. These are methods that modify the data, the model, or the predictions of a neural network to reduce or eliminate bias. For example, one fairness technique is pre-processing, which means that you transform the data before feeding it to the neural network, such as by removing, masking, or balancing the protected attributes. Another fairness technique is in-processing, which means that you incorporate fairness constraints or objectives into the neural network during training, such as by penalizing or optimizing for fairness metrics. A third fairness technique is post-processing, which means that you adjust the predictions of the neural network after training, such as by changing the thresholds or applying calibration.