Cloud computing has grown at a significant rate over the past few years. While cloud computing alleviates much of the hardware management challenge, many researchers and educators have difficulty embracing the aspects of cloud computing that make it unique and preferred for many activities. Among the many features cloud computing brings, elastic computing - resources on demand, may be one of the most appealing. A purpose-built virtual machine can be built quickly and easily as a first step. Taking the example further into scripted launching of more resources as necessary is the logical next step. The next step can use elastic computing techniques to create virtual clusters on demand, bringing compute resources into existence when needed and removing them when they are no longer necessary. This elasticity enables gateway providers and researchers to make efficient use of limited resources, while providing a resource for HPC-style jobs that isn't dependent on external HPC resources. While modest cloud-based virtual clusters won't replace traditional HPC resources for jobs that require high-speed interconnects or large core counts and high memory profiles, many smaller gateway, research, and education projects might benefit from the highly customizable and configurable, programmable cyberinfrastructure afforded by cloud computing environments. This tutorial will explore the basic methods required for interacting with elastic computing environments. It will then show a hands-on approach to creating virtual clusters in an OpenStack environment, including the necessary steps to make the cluster elastic, to take full advantage of the cloud environment. - Skill level: Intermediate and Advanced - Prerequisites: Basic Linux command line knowledge, text editor skills, basic cluster concepts knowledge a plus - Required: Laptop, Terminal application with ssh and ability to copy-paste. - Audience: Gateway developers, Researchers (Grad students, faculty, etc.), Campus Champions/ACI-REF Facilitators, Campus research computing staff
Jupyter Notebooks have become a mainstream tool for interactive computing in every field of science. Being a web-based platform, they are naturally suited to serve as companion application to Science Gateways: a scientist can use a Jupyter Notebook in their browser to pre-process inputs, launch a job on the Science Gateway via web API and then access, analyze, plot and postprocess the job outputs, without ever worrying about setting up and keeping updated their software environment. The JupyterHub project provides a multiple-user platform for Jupyter Notebooks and it is very easy to install and configure on a single server. However, when we need to provide computational resources to a large pool of users, we need to distribute the users on a cluster of machines, the best way to achieve scalability is thanks to the container orchestration platform Kubernetes. In this tutorial we will work through the installation of Kubernetes on a set of Jetstream Virtual Machines, setup persistent storage and install a bare-bone JupyterHub deployment using the zero-to-jupyterhub recipe provided by the Jupyter team. Then we will customize the setup configuring authentication (XSEDE, Globus or Github), choosing our preferred software environment for the users via Docker. Finally we will show how to execute computational jobs, either interfacing with the web APIs of a “test gateway” to submit jobs or launching a pool of workers on Kubernetes and execute a distributed computation (using dask). - Prerequisites: no specific technical skills, but familiarity with the shell, editing text files, and a general knowledge of Jetstream will be helpful. - Required: Participants should bring a laptop to participate in the hand-on parts of the workshop as well as have an XSEDE account to participate in the Kubernetes portion of the tutorial. - Audience: This workshop will be useful for Gateway developers interested in deploying JupyterHub. It could also be of interest of people wanting to deploy JupyterHub for workshops and tutorials about using a Gateway programmatically.
