CP-Mark: A conformal prediction benchmark for measuring the performance of fraud controls

CP-Mark：用于衡量欺诈控制性能的共形预测基准

• 批准号: 89039
• 金额: $ 8.38万
• 依托单位: FINCRIME DYNAMICS LTD
• 依托单位国家: 英国
• 项目类别: Collaborative R&D
• 财政年份: 2021
• 资助国家: 英国
• 起止时间: 2021 至 无数据
• 项目状态: 已结题
• 来源: https://gtr.ukri.org/projects?ref=89039
• 关键词: CP; Mark; conformal; prediction; benchmark

During COVID-19, the financial behaviour of people changed. Crime evolved to a new level. We are starting to witness rising financial crime rates where there is no presence of the cardholder and abuse of government support for the crisis. These problems require new ways for control systems to rapidly adapt to this reality, which will demand a corresponding benchmark that can appropriately measure the performance of transaction monitoring systems, which is one of the biggest challenges financial institutions face today. The performance measurement of machine learning algorithms is usually done with metrics like precision, recall and others that derive from these values, such as F-score. Precision allows us to identify, from the whole set of criminal behaviour detected in a given dataset, how much of it is actually crime, whereas recall gives the number of real crimes detected in proportion from the total amount of crime present in the dataset. The biggest issue of relying on these metrics as benchmarks in fincrime analytics is that the exact amount of hidden crime present in the real dataset is unknown, effectively invalidating the reliability of these conventional metrics for fraud analytics. This project introduces CP-Mark, a benchmark for evaluating controls in financial institutions. Financial institutions tune their control systems according to applicable regulations, which carries two clear objectives: detect and prevent as much criminal activity (increasing true positives), and reduce the number of innocent people wrongfully accused (reducing false positives). Financial institutions' efforts to achieve these goals are hindered, mainly because of their inability to adequately assess the actual amount of hidden crime present in their datasets, rendering conventional metrics with little benefit. Criminals leave fingerprints of their activities in financial institution's records. Unfortunately, the use of this data is very restricted under privacy regulations such as GDPR, significantly reducing the possibility of collaboration between different stakeholders to improve fraud control tools and prevent financial crime. A crucial part of the solution to these problems will require a combined response of enriched synthetic dataset generation and proper metrics that can adequately benchmark performance and effectiveness of machine learning algorithms that operate as part of transaction monitoring systems. PaySim is a payment simulation software that creates digital synthetic data enriched for advanced solutions based on machine learning techniques to understand the patterns in data that lead to financial misbehaviours. These patterns are extracted from real data sources preserving its privacy constraints, capturing the dynamics of fraud and combining them into tailor-made scenarios of diverse crime typologies. This project will allow us to evaluate the use of Conformal Prediction (CP-Mark) as a reliable benchmark tool to test the effectiveness of our software and other Machine Learning algorithms used as part of transaction monitoring systems. We will then perform benchmarks on several controls using these datasets with a state-of-the-art machine learning framework called conformal prediction to build predictive models capable of detecting known and currently undiscovered patterns of fraud.

在COVID-19期间，人们的财务行为发生了变化。犯罪发展到一个新的水平。我们开始目睹金融犯罪率上升，没有持卡人和滥用政府对危机的支持。这些问题需要新的控制系统来快速适应这一现实，这将需要一个相应的基准，可以适当地衡量交易监控系统的性能，这是金融机构今天面临的最大挑战之一。 机器学习算法的性能测量通常是用精确度、召回率和其他来自这些值的指标来完成的，比如F分数。精确度使我们能够从给定数据集中检测到的全部犯罪行为中识别出其中有多少是真正的犯罪，而召回率则给出了检测到的真实的犯罪数量与数据集中存在的犯罪总量的比例。在Fincrime分析中依赖这些指标作为基准的最大问题是，真实的数据集中存在的隐藏犯罪的确切数量是未知的，这实际上使这些传统指标的可靠性失效。该项目采用了CP-Mark，这是一个评估金融机构控制措施的基准。 金融机构根据适用的法规调整其控制系统，这有两个明确的目标：检测和预防尽可能多的犯罪活动（增加真阳性），并减少无辜者被错误指控的数量（减少误报）。金融机构实现这些目标的努力受到阻碍，主要是因为它们无法充分评估其数据集中存在的隐藏犯罪的实际数量，从而使传统的衡量标准没有什么好处。 犯罪分子在金融机构的记录中留下了他们活动的指纹。不幸的是，根据GDPR等隐私法规，这些数据的使用受到非常严格的限制，这大大降低了不同利益相关者之间合作改进欺诈控制工具和预防金融犯罪的可能性。解决这些问题的关键部分将需要丰富的合成数据集生成和适当的指标的组合响应，这些指标可以充分衡量作为事务监控系统一部分的机器学习算法的性能和有效性。 PaySim是一款支付模拟软件，可创建数字合成数据，以丰富基于机器学习技术的高级解决方案，从而了解导致金融不当行为的数据模式。这些模式是从真实的数据源中提取的，保留了其隐私约束，捕捉了欺诈的动态，并将其结合到各种犯罪类型的量身定制的场景中。该项目将使我们能够评估共形预测（CP标记）作为可靠的基准工具的使用，以测试我们的软件和其他机器学习算法作为交易监控系统的一部分的有效性。 然后，我们将使用这些数据集和最先进的机器学习框架（称为共形预测）对几个控件进行基准测试，以构建能够检测已知和当前未发现的欺诈模式的预测模型。