Improving Big Data Infrastructure to Effectively Utilize Resources for Data Processing and Services

完善大数据基础设施，有效利用数据处理和服务资源

• 批准号: RGPIN-2015-05390
• 负责人: Shi, Wei
• 金额: $ 1.31万
• 依托单位: Carleton University
• 依托单位国家: 加拿大
• 项目类别: Discovery Grants Program - Individual
• 财政年份: 2016
• 资助国家: 加拿大
• 起止时间: 2016-01-01 至 2017-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=613525
• 关键词: Improving; Big; Data; Infrastructure; Effectively

Big data analytics is hailed as an emerging driving force of the next innovation wave. It has formed the basis for many novel applications across a wide range of fields from business workflows to engineering activities and scientific computing. One such typical application is cloud-based network services for business and individual mobile or vehicular users, where data is collected through daily log files, from stationary or mobile entities such as smartphones equipped with a variety of built-in sensors (e.g., GPS, accelerometers and cameras) and then processed by a big-data infrastructure in order to provision data as a service. For example, vehicle trajectories can be generated in real time from video data to facilitate the analysis of traffic flow parameters for transportation management. Clearly, in such applications, the response time of the underlying infrastructure for data processing has great impact on Quality of Service (QoS). My ongoing overall research goal is the amelioration of QoS through better resource utilization in big data infrastructure. I approach this research goal from two directions: First, I aim at better balancing network traffic and more efficiently sharing cloud resources for applications by exploiting network topology, job characteristics and execution processes. These proposed improvements are particularly important for service providers deploying big data infrastructure in clouds, as the provisioned resources are typically based on pay-as-you-go billing strategies, and effectively utilizing the resources means maximizing the monetary profits for cloud service providers and minimizing cost for users of big data analytics. Second, I search for an effective way to disseminate data to the outside world as a service. Ultimately, I want to increase QoS while also improving the tolerance of big data infrastructure for a variety of service-disrupting faults. According to a recent whitepaper on the Big Data market written by International Data Corporation and sponsored by SAS, a leading developer of analytics software, “Canadian businesses are at risk of falling behind international competitors in the era of Big Data”. In particular, this survey and analysis found that Canadian companies are late and slow in adoption of technology capable of processing Big Data in comparison to international companies that are more willing to invest in and have more defined plans for adoption of these technologies in the near future. My proposed research will provide a more cost effective and more reliable Big Data infrastructure that offers faster processing of Big Data. This will attract more Canadian companies to adopt this new technology in order to become more innovative and productive. My results will bring significant economic benefits to Canada’s telecommunication and automobile industries, in particular, and significantly enhance Canada’s profile in this field.

大数据分析被誉为下一波创新浪潮的新兴驱动力。它为从业务工作流程到工程活动和科学计算等广泛领域的许多新颖应用程序奠定了基础。其中一个典型应用是为企业和个人移动或车辆用户提供基于云的网络服务，其中通过日常日志文件从固定或移动实体（例如配备各种内置传感器（例如 GPS、加速计和摄像头）的智能手机）收集数据，然后由大数据基础设施进行处理，以将数据作为服务提供。例如，可以根据视频数据实时生成车辆轨迹，以便于分析交通流参数以进行交通管理。显然，在此类应用中，数据处理底层基础设施的响应时间对服务质量（QoS）有很大影响。 我正在进行的总体研究目标是通过更好地利用大数据基础设施中的资源来改善服务质量。我从两个方向实现这一研究目标：首先，我的目标是通过利用网络拓扑、作业特征和执行流程，更好地平衡网络流量并更有效地共享应用程序的云资源。这些拟议的改进对于在云中部署大数据基础设施的服务提供商尤其重要，因为所配置的资源通常基于即用即付的计费策略，有效利用资源意味着最大化云服务提供商的货币利润并最小化大数据分析用户的成本。其次，我寻找一种有效的方式将数据作为服务传播给外界。最终，我希望提高 QoS，同时提高大数据基础设施对各种服务中断故障的容忍度。 根据国际数据公司最近撰写、领先分析软件开发商 SAS 赞助的大数据市场白皮书，“在大数据时代，加拿大企业面临着落后于国际竞争对手的风险”。特别是，这项调查和分析发现，与更愿意投资并在不久的将来采用这些技术有更明确计划的国际公司相比，加拿大公司在采用能够处理大数据的技术方面进展缓慢。 我提出的研究将提供更具成本效益和更可靠的大数据基础设施，以提供更快的大数据处理。这将吸引更多加拿大公司采用这项新技术，以提高创新能力和生产力。我的研究成果将为加拿大的电信和汽车行业带来显着的经济效益，并显着提升加拿大在该领域的形象。