CAREER: A Framework for Ad Hoc Model Construction in Data Streaming Environments

在过去十年中，数据量呈指数级增长，部分原因是由计算机程序和观测设备（如卫星、雷达和生态传感器）产生的数据流。考虑到数据量，很难利用这些数据来理解现象和/或做出预测。将模型拟合到观测数据中是实现这一目标的一种方法。建立这种模型的前提是从数据中提取特征。然后，使用这些特征构建的模型可以用来预测结果将是什么以及何时可能发生。这项研究将为科学家和研究人员提供必要的工具来理解流环境中产生的数据流。本研究广泛适用的领域包括智慧城市、交通规划、国土安全和生态监测。该项目包括一个教育部分，重点是提高女性学生在大学STEM专业的参与度。因此，本项目符合美国国家科学基金会促进科学进步，促进国家健康、繁荣和福利，保障国防安全的使命。该项目旨在开展研究，以创建一个使能的基础设施，以支持从大量、多维、大规模的时间序列观测数据中生成、评估和改进特设模型。特别模型创建中的挑战来自于模型实现方式的组合爆炸式增长。Synapse框架旨在支持和简化基于大量流数据的自然迭代和交互式模型构建过程。建模者只需要指定一组基本的bootstrap参数；该框架将管理以下方面的复杂性：(1)如何分散流，(2)如何管理数据访问，(3)处理I/O和内存争用，以及(4)模型生成工作负载的分散。该研究涉及可扩展的数据分散技术，采用分布式哈希表数据结构、基于映射减少的工作流和模型创建工作负载的编排、训练数据管理和模型性能的交互式可视化评估。可视化组件将允许建模者快速有效地评估多个模型的质量，每个模型可能覆盖输入特征空间的不同部分，并使用这些评估来指导关于选择、更新或替换模型的决策。如果成功，该框架将随着数据量、可用数据流数量、模型生成工作负载和实时模型评估的增加而扩展。

Over the past decade there has been an exponential growth in data volumes driven in part by data streams generated by computer programs and observational equipment such as satellites, radars, and ecological sensors. Given the data volumes, it can be difficult to harness the data to understand phenomena and/or to make forecasts. Fitting models to the observational data is one way to accomplish this. A precursor to building such models is extracting features from the data. Models constructed using such features can then be used to predict what the outcome will be and when it is likely to happen. This research will provide scientists and researchers the tools needed to make sense of data streams generated in streaming environments. Domains where this research is broadly applicable include smart cities, traffic planning, homeland security, and ecological monitoring. The project includes an educational component focused on increasing female student participation in college STEM majors. Therefore, this project aligns with the NSF mission to promote the progress of science, to advance the national health, prosperity and welfare, and to secure the national defense.The project aims to carry out research to create an enabling infrastructure to support the generation, assessment, and refinement of ad hoc models from voluminous, multidimensional, time-series observational data at scale. Challenges in ad hoc model creation stem from the combinatorially explosive number of ways in which models can be realized. The framework, Synapse, aims to support and simplify the naturally iterative and interactive model building process over voluminous streaming data. Modelers will only need to specify a basic set of bootstrap parameters; the framework will manage complexities relating to: (1) how streams are dispersed, (2) how data accesses are managed, (3) coping with I/O and memory contentions, and (4) dispersion of model generation workloads. The research involves scalable techniques for data dispersion employing distributed hash table data structures, map-reduce-based workflows and orchestration of model creation workloads, training data management, and interactive visual assessment of model performance. A visualization component will allow modelers to quickly and effectively assess the quality of a multiplicity of models each possibly covering a different portion of the input feature space and to use these assessments to guide decisions about selection, updates or replacements of models. If successful, the framework will scale with increases in data volumes, the number of available data streams, model generation workloads, and live model evaluations.