Deploying Jupyter Notebooks at scale on XSEDE resources for Science Gateways and workshops
Deploying Jupyter Notebooks at scale on XSEDE resources for Science Gateways and workshops
复制标题
在科学网关和研讨会的 XSEDE 资源上大规模部署 Jupyter Notebook
DOI：
10.1145/3219104.3219122
复制
发表时间：
2018
期刊：
影响因子：
--
通讯作者：
R. Sinkovits
中科院分区：
文献类型：
--
作者：
A. Zonca;R. Sinkovits
Jupyter Notebooks 已成为各个科学领域交互式计算的主流工具。 Jupyter Notebooks 适合作为 Science Gateways 的配套应用程序，为用户提供更大的灵活性和后处理能力。此外，它们经常用于培训活动和研讨会，以提供对预先配置的交互式计算环境的立即访问。 Jupyter 团队发布了 JupyterHub Web 应用程序，以提供一个可供多个用户登录和访问 Jupyter Notebook 环境的平台。当用户数量和内存要求较低时，很容易在单个服务器上设置 JupyterHub。然而，当我们需要为数十或数百个用户大规模提供 Jupyter Notebook 时，设置会变得更加复杂。在本文中，我们将提出在 XSEDE 资源上大规模部署 JupyterHub 的三种策略。所有选项都共享 XSEDE Jetstream 虚拟机上 JupyterHub 的部署。在第一个场景中，JupyterHub 连接到超级计算机并代表每个用户启动单节点作业，并将 Notebook 从计算节点代理回用户的浏览器。在第二个场景中，在 IRIS 地震学联盟的 XSEDE 咨询中实施，我们以 Swarm 模式部署 Docker，以协调许多 XSEDE Jetstream 虚拟机，为笔记本提供持久存储和配额。在最后一个场景中，我们在 Jetstream 上安装 Kubernetes 容器编排框架，以提供具有分布式文件系统的容错 JupyterHub 部署，并能够扩展到数千个用户。在结论部分，我们提供了分步教程的链接，其中包含复制这些部署所需的所有命令和配置文件。
Jupyter Notebooks have become a mainstream tool for interactive computing in every field of science. Jupyter Notebooks are suitable as companion applications for Science Gateways, providing more flexibility and post-processing capability to the users. Moreover they are often used in training events and workshops to provide immediate access to a pre-configured interactive computing environment. The Jupyter team released the JupyterHub web application to provide a platform where multiple users can login and access a Jupyter Notebook environment. When the number of users and memory requirements are low, it is easy to setup JupyterHub on a single server. However, setup becomes more complicated when we need to serve Jupyter Notebooks at scale to tens or hundreds of users. In this paper we will present three strategies for deploying JupyterHub at scale on XSEDE resources. All options share the deployment of JupyterHub on a Virtual Machine on XSEDE Jetstream. In the first scenario, JupyterHub connects to a supercomputer and launches a single node job on behalf of each user and proxies back the Notebook from the computing node back to the user's browser. In the second scenario, implemented in the context of a XSEDE consultation for the IRIS consortium for Seismology, we deploy Docker in Swarm mode to coordinate many XSEDE Jetstream virtual machines to provide Notebooks with persistent storage and quota. In the last scenario we install the Kubernetes containers orchestration framework on Jetstream to provide a fault-tolerant JupyterHub deployment with a distributed filesystem and capability to scale to thousands of users. In the conclusion section we provide a link to step-by-step tutorials complete with all the necessary commands and configuration files to replicate these deployments.