Effective Computational Optimization in Data Mining and Financial Applications

点击翻译按钮获取中文摘要

In the past five years, two phrases dominate discussion in the global news: market instability and big data. The main focus of this proposal is to develop computationally efficient and effective methods to optimally utilize available data, in order to improve health care, business and finance, including detection and prevention of systematic risk in financial markets.**After the 2008 financial market collapse, many have asked the question: why had there been no warnings of the signs of troubles? If these signs exist, why were they not detected? How can we achieve better detection and prevention of such a future market demise?**The Dodd Frank Wall Street Reform and Consumer Protection Act has resulted in a societal mandate to identify risks to financial stability from the events of financial firms. Although there is no clear definition of systemic risk measure, Biasias et al (2012) recently propose that a robust framework incorporating a diverse collection of perspectives and processes be adopted to dynamically adapt systemic risk measures to changes in financial market structures. Khandani et al (2010) have applied machine learning techniques to bank transactions and credit-bureau data of customers in order to predict consumer credit risk. In particular, it is suggested that the proportion of predicted delinquencies is a signal to systemic risk indicator in consumer lending. Hardle et al (2007) use scores from support vector machines to estimate default probabilities of financial firms.**With increasing accumulation of data, efficient and effective data mining methods stand to potentially offer solutions to challenging problems faced in finance, business, and our lives in general. The "2011 McKinsey Report on Big Data" estimates that data mining could potentially bring $300 billion annual value to US health care, 250 billion annual value to the European public administration sector, and a $600 billion potential consumer surplus. While there have been major advances in information gathering, to turn these estimates into realities, we need a commensurate advance in data analytics. **The urgency of solving challenging data analysis problems is illustrated by the recent Heritage Provider Network (HPN) sponsored global incentivized competition. In an effort to identify at-risk individuals earlier and ensure they receive prompt treatment, the objective of the competition was to create algorithms that use patient data to predict hospitalizations. The competition ran for two years with a grand prize of $3 million, attracting nearly 2000 participants from various disciplines around the world. Together with my PhD student Aditya Tayal* and colleague Thomas Coleman, we investigated and developed several computational optimization algorithms that ultimately led us to securing a fourth place ranking in the competition.**A data mining method has three components: minimizing training error, maximizing stability and a mechanism to balance the trade off between the two objectives. The remaining challenging optimization problems in data mining are typically nonconvex and large scale. For example, in many real data analysis problems, only very limited labels are available. How do we learn a predictive model, using partial label information? How do we optimally select features from a collection of available data that are relevant for a particular prediction task? Many practical data mining problems have a rare class and a majority class. How do we develop computationally efficiently nonlinear methods for these unbalanced problems? The main goal of the research proposed here is to solve these challenging but relevant optimization problems in data mining and apply them to health care, finance, business, and other industries.