Distributed Systems Support for Processing Big Data from Sensor Networks

分布式系统支持处理来自传感器网络的大数据

• 批准号: RGPIN-2019-06776
• 负责人: Makaroff, Dwight
• 金额: $ 2.04万
• 依托单位: University of Saskatchewan
• 依托单位国家: 加拿大
• 项目类别: Discovery Grants Program - Individual
• 财政年份: 2021
• 资助国家: 加拿大
• 起止时间: 2021-01-01 至 2022-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=739101
• 关键词: Distributed; Systems; Support; Processing; Big

Modern computer systems consist of multiple components that coordinate their input, output and processing to provide timely information to their human or machine users. These components form distributed systems with processing and data transmission possible at every node. One source of system input has been the Internet of Things, with the massive deployment of sensors (including images and video). A typical deployment scenario is the growing market of wearable devices in fitness/health monitoring. Individual devices can generate large amounts of data over short time periods, processed both locally and by a centralized organization. These body-worn devices may or may not have continuous connection to the Internet for appropriate cost-effective data transmission. Delivery protocols are required that prioritize data requirements and are resource-efficient. Some monitoring data must, however, be transferred in real-time. Similar data streams will be generated by home automation and industrial sensors, such as mining, manufacturing, and agriculture. They, too require delivery of large amounts of data at different volumes, rates and latency requirements, constituting streams of big data. When data arrives at a centralized processing location, another problem is created. The total aggregate of data from many sources (individuals, industrial sensors, security cameras and smartphones) overwhelms the capacities of the largest computers available. Parallel processing techniques and frameworks developed by organizations such as Google, Yahoo! and Facebook for Big Data processing have been the preferred solutions for the last dozen years or so. As new frameworks are introduced, their applicability in emerging domains, such as environmental sensor networks and wearable networks requires configuration, tuning and parameter settings. This will require a mixture of manual resource allocation and scheduling policies. The main research goal investigated through this program is to identify and quantify the performance/resource utilization tradeoff in distributed systems consisting of data collection from multiple diverse sources that require aggregate data processing to find common patterns and characteristics. I aim to build techniques to identify configurations that achieve preferred, cost-effective deployments that meet user goals with regards to this tradeoff. Various strategies for reducing resource requirements (such as energy/communication bandwidth/disk space) result in a loss of quality of information or the rate at which the information can be provided. Appropriate adjustments in scheduling, routing or resource allocation between the various components will enable this degradation in service to be mitigated. The outcomes will be methods of evaluating the tradeoffs, case studies using these evaluation methods and development of prototype systems that implement adaptive strategies fo collecting and processing sensor network big data.

现代计算机系统由多个组件组成，它们协调其输入，输出和处理，以及时向其人或机器使用者提供信息。这些组件形成分布式系统，并在每个节点上进行处理和数据传输。系统输入的一种来源是物联网，传感器的大量部署（包括图像和视频）。典型的部署情况是健身/健康监测中可穿戴设备的不断增长。单个设备可以在本地和集中组织处理的短时间内生成大量数据。这些磨损的设备可能与Internet连续连接，以进行适当的具有成本效益的数据传输。需要交付协议确定数据要求优先级，并且资源效率高。但是，必须实时传输一些监视数据。类似的数据流将由家庭自动化和工业传感器（例如采矿，制造和农业）产生。他们也需要以不同的量，速率和延迟要求提供大量数据，构成大数据流。当数据到达集中处理位置时，将创建另一个问题。来自许多来源（个人，工业传感器，安全摄像机和智能手机）的数据总计淹没了可用的最大计算机的容量。由Google，Yahoo!等组织开发的并行处理技术和框架在过去的十二年中，用于大数据处理的Facebook一直是首选解决方案。当引入新框架时，它们在新兴域中的适用性，例如环境传感器网络和可穿戴网络需要配置，调整和参数设置。这将需要手动资源分配和调度策略的混合。通过该计划调查的主要研究目标是识别和量化分布式系统中的性能/资源利用权权衡，这些分布式系统由来自多种不同来源的数据收集组成，这些来源需要汇总数据处理以找到常见的模式和特征。我的目标是建立技术以确定实现符合用户目标的优先，具有成本效益的部署的配置。减少资源需求的各种策略（例如能源/通信带宽/磁盘空间）会导致信息质量损失或提供信息的速度。各个组件之间的调整，路由或资源分配的适当调整将使服务中的这种降解得到缓解。结果将是评估权衡的方法，使用这些评估方法的案例研究以及开发原型系统，这些系统实施自适应策略，以收集和处理传感器网络大数据。