CP-Mark: A conformal prediction benchmark for measuring the performance of fraud controls

在新冠肺炎期间，人们的金融行为发生了变化。犯罪发展到了一个新的水平。我们开始目睹金融犯罪率上升，因为没有持卡人的存在，以及滥用政府对危机的支持。这些问题需要控制系统迅速适应这一现实的新方法，这将需要一个相应的基准来适当地衡量交易监控系统的性能，这是当今金融机构面临的最大挑战之一。机器学习算法的性能测量通常使用精度、召回率等指标，以及从这些值派生出的其他指标，如F-Score。精确度使我们能够从给定数据集中发现的整个犯罪行为中识别出其中有多少实际上是犯罪，而Recall则提供了在数据集中存在的犯罪总量中按比例检测到的真实犯罪数量。在金融犯罪分析中依赖这些指标作为基准的最大问题是，真实数据集中存在的隐藏犯罪的确切数量是未知的，这实际上使这些传统指标用于欺诈分析的可靠性无效。该项目引入了CP-Mark，这是一种评估金融机构控制的基准。金融机构根据适用的法规调整其控制系统，这有两个明确的目标：尽可能多地发现和预防犯罪活动(增加真阳性)，以及减少被错误指控的无辜人数(减少假阳性)。金融机构实现这些目标的努力受到阻碍，主要是因为它们无法充分评估其数据集中存在的隐藏犯罪的实际数量，使得传统指标几乎没有好处。犯罪分子在金融机构的记录中留下了他们活动的指纹。不幸的是，在GDPR等隐私法规下，这些数据的使用受到非常严格的限制，大大降低了不同利益攸关方合作改进欺诈控制工具和预防金融犯罪的可能性。这些问题的解决方案的一个关键部分将需要丰富的合成数据集生成和适当的指标的组合响应，这些指标可以充分衡量作为交易监控系统一部分运行的机器学习算法的性能和有效性。PaySim是一款支付模拟软件，它基于机器学习技术创建数字合成数据，为高级解决方案提供丰富的数据，以了解导致金融不当行为的数据模式。这些模式是从真实数据源中提取的，保留了其隐私限制，捕获了欺诈的动态，并将它们组合成各种犯罪类型的定制场景。该项目将使我们能够评估共形预测(CP-Mark)作为一种可靠的基准工具的使用，以测试我们的软件和用作交易监控系统一部分的其他机器学习算法的有效性。然后，我们将使用这些数据集和称为共形预测的最先进的机器学习框架对几个控制进行基准测试，以建立能够检测已知和当前未发现的欺诈模式的预测模型。

During COVID-19, the financial behaviour of people changed. Crime evolved to a new level. We are starting to witness rising financial crime rates where there is no presence of the cardholder and abuse of government support for the crisis. These problems require new ways for control systems to rapidly adapt to this reality, which will demand a corresponding benchmark that can appropriately measure the performance of transaction monitoring systems, which is one of the biggest challenges financial institutions face today. The performance measurement of machine learning algorithms is usually done with metrics like precision, recall and others that derive from these values, such as F-score. Precision allows us to identify, from the whole set of criminal behaviour detected in a given dataset, how much of it is actually crime, whereas recall gives the number of real crimes detected in proportion from the total amount of crime present in the dataset. The biggest issue of relying on these metrics as benchmarks in fincrime analytics is that the exact amount of hidden crime present in the real dataset is unknown, effectively invalidating the reliability of these conventional metrics for fraud analytics. This project introduces CP-Mark, a benchmark for evaluating controls in financial institutions. Financial institutions tune their control systems according to applicable regulations, which carries two clear objectives: detect and prevent as much criminal activity (increasing true positives), and reduce the number of innocent people wrongfully accused (reducing false positives). Financial institutions' efforts to achieve these goals are hindered, mainly because of their inability to adequately assess the actual amount of hidden crime present in their datasets, rendering conventional metrics with little benefit. Criminals leave fingerprints of their activities in financial institution's records. Unfortunately, the use of this data is very restricted under privacy regulations such as GDPR, significantly reducing the possibility of collaboration between different stakeholders to improve fraud control tools and prevent financial crime. A crucial part of the solution to these problems will require a combined response of enriched synthetic dataset generation and proper metrics that can adequately benchmark performance and effectiveness of machine learning algorithms that operate as part of transaction monitoring systems. PaySim is a payment simulation software that creates digital synthetic data enriched for advanced solutions based on machine learning techniques to understand the patterns in data that lead to financial misbehaviours. These patterns are extracted from real data sources preserving its privacy constraints, capturing the dynamics of fraud and combining them into tailor-made scenarios of diverse crime typologies. This project will allow us to evaluate the use of Conformal Prediction (CP-Mark) as a reliable benchmark tool to test the effectiveness of our software and other Machine Learning algorithms used as part of transaction monitoring systems. We will then perform benchmarks on several controls using these datasets with a state-of-the-art machine learning framework called conformal prediction to build predictive models capable of detecting known and currently undiscovered patterns of fraud.