Opportunities and Challenges: Lessons from Analyzing Terabytes of Scanner Data
Opportunities and Challenges: Lessons from Analyzing Terabytes of Scanner Data
复制标题
机遇与挑战：分析 TB 级扫描仪数据的经验教训
DOI：
--
复制
发表时间：
2016
期刊：
影响因子：
--
通讯作者：
Serena Ng
中科院分区：
文献类型：
--
作者：
Serena Ng
本文旨在更好地了解大数据分析的不同之处，我们可以和不能用现有的计量经济学工具做什么，以及为了有效地使用数据需要处理什么问题。作为一个案例研究，我着手提取可能存在于每周4TB扫描仪数据中的任何商业周期信息。主要的挑战是在我们的计算环境的约束下处理数据的数量、种类和特征。可扩展和有效的算法可用于减轻计算负担，但它们通常具有未知的统计特性，并且不是为了有效估计或最佳推理而设计的。同样，经济数据具有通用算法可能无法适应的独特特征。由于大数据可能会继续存在，因此需要计算效率高的计量经济学方法。
This paper seeks to better understand what makes big data analysis different, what we can and cannot do with existing econometric tools, and what issues need to be dealt with in order to work with the data efficiently. As a case study, I set out to extract any business cycle information that might exist in four terabytes of weekly scanner data. The main challenge is to handle the volume, variety, and characteristics of the data within the constraints of our computing environment. Scalable and efficient algorithms are available to ease the computation burden, but they often have unknown statistical properties and are not designed for the purpose of efficient estimation or optimal inference. As well, economic data have unique characteristics that generic algorithms may not accommodate. There is a need for computationally efficient econometric methods as big data is likely here to stay.
影响因子：
20.6
作者：
Fan J;Han F;Liu H
通讯作者：
Liu H