CP-Mark: A conformal prediction benchmark for measuring the performance of fraud controls

点击翻译按钮获取中文摘要

During COVID-19, the financial behaviour of people changed. Crime evolved to a new level. We are starting to witness rising financial crime rates where there is no presence of the cardholder and abuse of government support for the crisis. These problems require new ways for control systems to rapidly adapt to this reality, which will demand a corresponding benchmark that can appropriately measure the performance of transaction monitoring systems, which is one of the biggest challenges financial institutions face today. The performance measurement of machine learning algorithms is usually done with metrics like precision, recall and others that derive from these values, such as F-score. Precision allows us to identify, from the whole set of criminal behaviour detected in a given dataset, how much of it is actually crime, whereas recall gives the number of real crimes detected in proportion from the total amount of crime present in the dataset. The biggest issue of relying on these metrics as benchmarks in fincrime analytics is that the exact amount of hidden crime present in the real dataset is unknown, effectively invalidating the reliability of these conventional metrics for fraud analytics. This project introduces CP-Mark, a benchmark for evaluating controls in financial institutions. Financial institutions tune their control systems according to applicable regulations, which carries two clear objectives: detect and prevent as much criminal activity (increasing true positives), and reduce the number of innocent people wrongfully accused (reducing false positives). Financial institutions' efforts to achieve these goals are hindered, mainly because of their inability to adequately assess the actual amount of hidden crime present in their datasets, rendering conventional metrics with little benefit. Criminals leave fingerprints of their activities in financial institution's records. Unfortunately, the use of this data is very restricted under privacy regulations such as GDPR, significantly reducing the possibility of collaboration between different stakeholders to improve fraud control tools and prevent financial crime. A crucial part of the solution to these problems will require a combined response of enriched synthetic dataset generation and proper metrics that can adequately benchmark performance and effectiveness of machine learning algorithms that operate as part of transaction monitoring systems. PaySim is a payment simulation software that creates digital synthetic data enriched for advanced solutions based on machine learning techniques to understand the patterns in data that lead to financial misbehaviours. These patterns are extracted from real data sources preserving its privacy constraints, capturing the dynamics of fraud and combining them into tailor-made scenarios of diverse crime typologies. This project will allow us to evaluate the use of Conformal Prediction (CP-Mark) as a reliable benchmark tool to test the effectiveness of our software and other Machine Learning algorithms used as part of transaction monitoring systems. We will then perform benchmarks on several controls using these datasets with a state-of-the-art machine learning framework called conformal prediction to build predictive models capable of detecting known and currently undiscovered patterns of fraud.