Collaborative Research: EarthCube Capabilities: ICESpark: An Open-Source Big Data Platform for Science Discoveries in the New Arctic and Beyond

北极气候系统正在经历快速变化，大气和海洋表面温度上升，同时北极冰川、海冰和陆地积雪减少。全球气温的升高和冰盖质量的减少正在推动全球海平面上升。与此同时，随着北极融化，该地区的海上和商业活动正在扩大，带来了新的机遇，也带来了社会和文化挑战。由于北极地区在很大程度上无法使用传统观测技术，卫星遥感系统在监测北极地区的基本气候变量方面发挥着关键作用。然而，新卫星收集的前所未有的数量和种类的地理空间大数据已经远远超出了大多数地球科学家可以访问的计算平台的能力。数据增长和数据发现能力之间的这种差距大大削弱了新兴大数据集的价值。此外，大多数现有的地理空间大数据软件都没有提供先进的分析能力来促进地球科学的发现。该项目旨在通过开发一个名为ICESpark的低成本大规模系统来消除这些障碍，以无缝支持新北极及其他地区支持大数据的地球科学研究的生命周期。结果可能会通过解决关键的气候变化问题来改善公民的福祉，这些问题包括极端事件、自然灾害、海平面上升、干旱和野火。它还将通过开发新的课程材料、对计算和地球科学领域的学生进行交叉培训以及ICESpark网络研讨会系列来改进科学和工程教育。ICESpark是一个分布式平台，可以将当地的商用计算机结合到一个强大的环境中，为地球空间大数据(GeoBD)做好准备。ICESpark建立在阿帕奇Sedona的基础上，首先开发数据集成和清理工具，以利用包括海洋学、冰冻圈科学和生态学在内的各种地球科学领域的GeoBD。此外，ICESpark提供了一个可扩展的数据发现层，以在各种条件下高效地识别来自不同传感平台(例如，ICESat-2、Jason-3、Sentinel-3、GEDI)的所有流中的所有重合数据。第三，ICESpark提供先进的数据分析能力，包括地球特征识别系统和地球模式挖掘程序包，为地球科学家配备地球物理或统计工具，以审查嵌入GeoBD的复杂关系和模式。为了加强研究基础设施，ICESpark将提供各种预装的前端设备，包括Jupyter笔记本电脑以及与EarthCube的QGreenland的互操作，从而改善系统在广泛的学科社区中的可及性。该系统还将是开源的，并遵循EarthCube GeoCODES数据集方案，以实现长期可持续性。多学科团队将共同致力于ICESpark的设计和开发，并对其进行优化，以利用GeoBD并解决具有挑战性的地球科学问题。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

The Arctic climate system is undergoing rapid change with rising air and sea surface temperatures, accompanied by declines in Arctic glaciers, sea ice and snow cover on land. Increases in global air temperatures and ice-sheet mass loss are driving sea level rise around the globe. Meanwhile, as the Arctic melts, maritime and commercial activities in the region are expanding, presenting new opportunities, as well as societal and cultural challenges. As Arctic regions are largely inaccessible to traditional observation techniques, satellite remote sensing systems play a key role in monitoring their essential climate variables. However, the unprecedented volume and variety of geospatial big data collected by new satellites have reached far beyond the capacity of computing platforms accessible to most geoscientists. This gap between data growth and data discovery capacity significantly undermines the value of emerging big datasets. Moreover, most existing software for geospatial big data do not offer advanced analytical capabilities to facilitate geoscience discoveries. The project aims to remove these barriers by developing a low-cost and large-scale system, namely ICESpark, to seamlessly support the lifecycle of big data enabled geoscience research in the New Arctic and beyond. The results may improve the well-being of citizens by addressing key climate change questions, including extreme events, natural disasters, sea-level rise, drought, and wildfires. It will also improve science and engineering education via development of new course materials, cross-training of students from computing and geosciences fields, as well as an ICESpark webinar series.ICESpark is a distributed platform that can combine local commodity computers into a powerful environment that is ready for geospatial big data (GeoBD). Built on Apache Sedona, ICESpark first develops data integration and cleaning tools to harness a wide variety of GeoBD across geoscience domains including oceanography, cryospheric science and ecology. Moreover, ICESpark provides a scalable data discovery layer to efficiently identify all coincidental data across streams from heterogeneous sensing platforms (e.g., ICESat-2, Jason-3, Sentinel-3, GEDI) under various conditions. Third, ICESpark offers advanced data analytics capabilities, including a geo-feature identification system and a geo-pattern mining package, to equip geoscientists with geophysical or statistical tools to examine complex relationships and patterns embedded in GeoBD. To enhance research infrastructure, ICESpark will provide a variety of pre-packed front-ends including Jupyter notebooks as well as interoperation with EarthCube’s QGreenland, improving the accessibility to the system across broad disciplinary communities. The system will also be open-sourced and follow the EarthCube GeoCODES Dataset schema for long-term sustainability. The multidisciplinary team will work together on the design and development of ICESpark and optimize it to harness GeoBD and tackle challenging geoscience problems.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.