Navigating the Transition from Flux to Great Lakes: New tools and policy town hall

Abstract

Effective computing is a key component of many statistics workflows. The statistics department offers its graduate students access to a UMich HPC (high performance computing) resource. However, that resource will be changing as Flux retires at the end of November and we transition to Great Lakes. One important change is that rather than having access to a fixed number of resources at any given time, billing will be based on actual usage, so without new policies it may be possible for one user to (accidentally) burn through an entire month’s budget in a single day. Efficient use of this resource will be even more important and will require navigating the job submission system (SLURM), writing code in a way which can be parallelized effectively across multiple machines, and understanding how jobs are billed to the Statistics department account. This special student seminar will have two parts, focused on the new opportunities and new challenges, respectively. The first part will introduce some tips and tools for statisticians to use Great Lakes more effectively and will include a demonstration of the batchtools R package which can greatly simplify and organize the process of running numerical simulations in an HPC setting. The second part will be a town hall regarding proposed policies for our account on Great Lakes. Because the department’s allocation is primarily used by students, department leadership has invited the PhD student council’s computing committee to propose specific policies for this new resource, and we want to get feedback on our policy proposal from current graduate students. If you use (or might use) Flux and/or Great Lakes, we want to hear from you! Our goal is to craft a policy that lets heavy users get their work done while making the experience of lighter users as smooth as possible. If you’re interested in following along the demonstrations in the first part, you need to ensure you’re able to log-on to Great Lakes (ssh uniqname@greatlakes.arc-ts.umich.edu). You may also wish to review this wiki page on using batchtools on Great Lakes and you can install batchtools locally so that you can see a demonstration of how it works. Contact stat-phd-council-computing@umich.edu with any questions.