ExCALIBUR HES: Exascale Data Testbed for Simulation, Data Analysis & Visualisation

ExCALIBUR HES：用于仿真、数据分析的百亿亿次数据测试台

• 批准号: EP/Y004051/1
• 负责人: Nathan Abraham
• 金额: $ 84.11万
• 依托单位: University of Cambridge
• 依托单位国家: 英国
• 项目类别: Research Grant
• 财政年份: 2023
• 资助国家: 英国
• 起止时间: 2023 至 无数据
• 项目状态: 未结题
• 来源: https://gtr.ukri.org/projects?ref=EP%2FY004051%2F1
• 关键词: ExCALIBUR; HES; Exascale; Data; Testbed

his proposal is phase 3, i.e., a continuation of 2 previous Cambridge ExCALIBUR H&S funded projects under the same name "Exascale Data Testbed for Simulation, Data Analysis & Visualisation" The proposal builds on these 2 previous ExCALIBUR H & S projects to design, build and make available to the ExCALIBUR application community state of the art solid state I/O prototype platforms that can be used to understand and characterise novel emerging solid state file system technologies and help develop UK ExCALIBUR application capable of high I/O performance needed to scale at exascale.This phase 3 proposal will extend the service of the phase 1 and 2 test beds for another 2 years and add functionality and range of files systems provided. This is enabled by new software tools and additional storage hardware. This additional functionality is seen to be needed in light of extensive engagement with ExCALIBUR projects Excalidata and Excalistore PI'd by Bryan Lawrence and direct engagements with the Met office, UKAEA , DiRAC, IRIS and SKA SRC projects. The proposal is split into 2 separate sections: -A) I/O profiling software tools at application and system levelB) Additional hardware for I/O testbedsA) I/O profiling software tools Two commercial I/O profiling tools (Altaire's Mistral & Breeze) will be procured with capital funding from this call and installed on the operational Cambridge CSD3 HPC system and also on dedicated ExCALIBUR IO testbeds resulting from the phase 1 & 2 Cambridge ExCALIBUR H$S awards. Staff time contributed at 2 FTE is funded by the Cambridge Open Exascale Lab. Cambridge have already trialed these tools via evaluation licences from Altair and found them very useful.These products show us the two views of HPC I/O we need to understand, one view from the microscopic application level the other view from the macroscopic system wide level. When we put these together, we should have a much better picture of what is going on with HPC I/O, providing the tools we need to understand the behavior of the different ExCALIBUR prototype file systems and then how to optimise applications we choose to run there.B) Additional hardware for data testbeds - During community engagement with Excalistore, Excalidata, DiRAC, IRIS and SKS SRC community we see a strong need to develop and test high performance object storage for HPC using both SSD and spinning disk platforms. We also need a small spinning disk testbed to evaluate the same candidate file systems we build on the current solid-state testbed developed in the phaseKey deliverables - Technical reports on programme of works items 1-4, these will be fashioned into white papers and published through industrial partners Dell/Intel via Exascale Lab and also written up as academic papers. We have received a lot of interest in the approach of using a large-scale apples for apples NVMe testbed with different file systems, the work is highly publishable. The work will also produce valuable test beds, analysis systems and skilled people for UK HPC user communities mentioned above to test and improve I/O of key UK candidate exascale codes.

他的建议是第三阶段，即，延续前2个剑桥ExCALIBUR H&S资助的项目，名称相同，“Exascale Data Testbed for Simulation，Data Analysis & Visualisation”该提案建立在这2个前ExCALIBUR H & S项目的基础上，构建并向ExCALIBUR应用程序社区提供最先进的固态I/O原型平台，可用于了解和验证新兴的固态文件系统技术，并帮助开发能够实现高I/O性能的英国ExCALIBUR应用程序。O性能需要扩展到亿亿次。第3阶段的提案将把第1阶段和第2阶段测试台的服务再延长2年，并增加所提供的文件系统的功能和范围。这是通过新的软件工具和额外的存储硬件实现的。鉴于Bryan Lawrence与ExCALIBUR项目Excalidata和Excalistore PI'd的广泛接触以及与英国气象局、UKAEA、DiRAC、IRIS和SKA SRC项目的直接接触，这一额外功能被认为是必要的。该提案分为两个单独的部分：-A）应用程序和系统级的I/O分析软件工具B）I/O测试台的附加硬件A）I/O分析软件工具两个商业I/O分析工具（Altaire的Mistral & Breeze）将通过此次电话会议的资本资金进行采购，并安装在运行的剑桥CSD 3 HPC系统上，以及第1阶段和第2阶段产生的专用ExCALIBUR IO测试平台上。2项剑桥ExCALIBUR H$S奖。工作人员的时间贡献在2 FTE是由剑桥开放艾级实验室资助。剑桥已经通过Altair的评估许可证试用了这些工具，发现它们非常有用。这些产品向我们展示了我们需要了解的HPC I/O的两个视图，一个是微观应用级别的视图，另一个是宏观系统级别的视图。当我们把这些放在一起时，我们应该对HPC I/O的情况有一个更好的了解，提供我们需要的工具来了解不同ExCALIBUR原型文件系统的行为，然后如何优化我们选择在那里运行的应用程序。B）数据测试床的额外硬件-在社区参与Excalistore，Excalidata，DiRAC，IRIS和SKS SRC社区认为，迫切需要使用SSD和旋转磁盘平台为HPC开发和测试高性能对象存储。我们还需要一个小型旋转磁盘测试平台，以评估我们在阶段中开发的当前固态测试平台上构建的相同候选文件系统关键可交付成果-关于工作计划项目1-4的技术报告，这些报告将被制成白色文件，并通过工业合作伙伴戴尔/英特尔通过Exascale实验室发布，还将被写成学术论文。我们对使用具有不同文件系统的大规模Apple for Apple NVMe测试床的方法非常感兴趣，这项工作非常容易实现。这项工作还将为上述英国HPC用户社区提供有价值的测试床，分析系统和熟练人员，以测试和改进英国关键候选exascale代码的I/O。