Distributed Systems Support for Processing Big Data from Sensor Networks

点击翻译按钮获取中文摘要

Modern computer systems consist of multiple components that coordinate their input, output and processing to provide timely information to their human or machine users. These components form distributed systems with processing and data transmission possible at every node. One source of system input has been the Internet of Things, with the massive deployment of sensors (including images and video). A typical deployment scenario is the growing market of wearable devices in fitness/health monitoring. Individual devices can generate large amounts of data over short time periods, processed both locally and by a centralized organization. These body-worn devices may or may not have continuous connection to the Internet for appropriate cost-effective data transmission. Delivery protocols are required that prioritize data requirements and are resource-efficient. Some monitoring data must, however, be transferred in real-time. Similar data streams will be generated by home automation and industrial sensors, such as mining, manufacturing, and agriculture. They, too require delivery of large amounts of data at different volumes, rates and latency requirements, constituting streams of big data.******When data arrives at a centralized processing location, another problem is created. The total aggregate of data from many sources (individuals, industrial sensors, security cameras and smartphones) overwhelms the capacities of the largest computers available. Parallel processing techniques and frameworks developed by organizations such as Google, Yahoo! and Facebook for Big Data processing have been the preferred solutions for the last dozen years or so. As new frameworks are introduced, their applicability in emerging domains, such as environmental sensor networks and wearable networks requires configuration, tuning and parameter settings. This will require a mixture of manual resource allocation and scheduling policies.******The main research goal investigated through this program is to identify and quantify the performance/resource utilization tradeoff in distributed systems consisting of data collection from multiple diverse sources that require aggregate data processing to find common patterns and characteristics. I aim to build techniques to identify configurations that achieve preferred, cost-effective deployments that meet user goals with regards to this tradeoff. Various strategies for reducing resource requirements (such as energy/communication bandwidth/disk space) result in a loss of quality of information or the rate at which the information can be provided. Appropriate adjustments in scheduling, routing or resource allocation between the various components will enable this degradation in service to be mitigated.******The outcomes will be methods of evaluating the tradeoffs, case studies using these evaluation methods and development of prototype systems that implement adaptive strategies fo collecting and processing sensor network big data.*****