Improving Big Data Infrastructure to Effectively Utilize Resources for Data Processing and Services

大数据分析被誉为下一波创新浪潮的新兴驱动力。它已经形成了从业务工作流到工程活动和科学计算等广泛领域的许多新颖应用的基础。其中一个典型的应用是面向企业和个人移动用户或车辆用户的基于云的网络服务，通过日常日志文件从固定或移动实体（如配备各种内置传感器的智能手机）收集数据，然后由大数据基础设施进行处理，以便将数据作为服务提供。例如，可以从视频数据实时生成车辆轨迹，方便交通流参数分析，便于交通管理。显然，在此类应用程序中，用于数据处理的底层基础设施的响应时间对服务质量（QoS）有很大影响。*******我正在进行的总体研究目标是通过在大数据基础设施中更好地利用资源来改善QoS。我从两个方向实现这一研究目标：首先，我的目标是通过利用网络拓扑、作业特征和执行过程，更好地平衡网络流量，更有效地为应用程序共享云资源。这些建议的改进对于在云中部署大数据基础设施的服务提供商来说尤其重要，因为所提供的资源通常基于即用即付计费策略，有效利用资源意味着云服务提供商的货币利润最大化，而大数据分析用户的成本最小化。其次，我寻找一种有效的方式，将数据作为一种服务传播给外界。最终，我希望提高QoS，同时提高大数据基础设施对各种服务中断故障的容忍度。******根据最近由国际数据公司（International Data Corporation）撰写的一份关于大数据市场的白皮书，由领先的分析软件开发商SAS赞助，“在大数据时代，加拿大企业面临落后于国际竞争对手的风险”。特别是，这项调查和分析发现，与国际公司相比，加拿大公司在采用能够处理大数据的技术方面起步较晚，速度较慢，而国际公司更愿意投资，并在不久的将来有更明确的采用这些技术的计划。******我提出的研究将提供一个更具成本效益和更可靠的大数据基础设施，提供更快的大数据处理。这将吸引更多的加拿大公司采用这项新技术，以提高其创新性和生产力。我的研究成果将为加拿大的电信和汽车行业带来显著的经济效益，并显著提升加拿大在这一领域的形象。

Big data analytics is hailed as an emerging driving force of the next innovation wave. It has formed the basis for many novel applications across a wide range of fields from business workflows to engineering activities and scientific computing. One such typical application is cloud-based network services for business and individual mobile or vehicular users, where data is collected through daily log files, from stationary or mobile entities such as smartphones equipped with a variety of built-in sensors (e.g., GPS, accelerometers and cameras) and then processed by a big-data infrastructure in order to provision data as a service. For example, vehicle trajectories can be generated in real time from video data to facilitate the analysis of traffic flow parameters for transportation management. Clearly, in such applications, the response time of the underlying infrastructure for data processing has great impact on Quality of Service (QoS). *******My ongoing overall research goal is the amelioration of QoS through better resource utilization in big data infrastructure. I approach this research goal from two directions: First, I aim at better balancing network traffic and more efficiently sharing cloud resources for applications by exploiting network topology, job characteristics and execution processes. These proposed improvements are particularly important for service providers deploying big data infrastructure in clouds, as the provisioned resources are typically based on pay-as-you-go billing strategies, and effectively utilizing the resources means maximizing the monetary profits for cloud service providers and minimizing cost for users of big data analytics. Second, I search for an effective way to disseminate data to the outside world as a service. Ultimately, I want to increase QoS while also improving the tolerance of big data infrastructure for a variety of service-disrupting faults. ******According to a recent whitepaper on the Big Data market written by International Data Corporation and sponsored by SAS, a leading developer of analytics software, "Canadian businesses are at risk of falling behind international competitors in the era of Big Data". In particular, this survey and analysis found that Canadian companies are late and slow in adoption of technology capable of processing Big Data in comparison to international companies that are more willing to invest in and have more defined plans for adoption of these technologies in the near future.******My proposed research will provide a more cost effective and more reliable Big Data infrastructure that offers faster processing of Big Data. This will attract more Canadian companies to adopt this new technology in order to become more innovative and productive. My results will bring significant economic benefits to Canada's telecommunication and automobile industries, in particular, and significantly enhance Canada's profile in this field.