HPC log in and install softwares

Log in

Once you have been approved, you can access HPC from:

1. Within the NYU network:

ssh NYUNetID@prince.hpc.nyu.edu

Once logged in, the root should be: /home/NYUNetID, so running pwd should print:

[NYUNetID@log-0 ~]$ pwd
/home/NYUNetID

2. From an off-campus location:

First, login to the bastion host:

ssh NYUNetID@gw.hpc.nyu.edu

Then login to the cluster:

ssh prince.hpc.nyu.edu

File Systems

You can get acces to three filesystems: /home, /scratch, and /archive.

Scratch is a file system mounted on Prince that is connected to the compute nodes where we can upload files faster. Notice that the content gets periodically flushed.

[NYUNetID@log-0 ~]$ cd /scratch/NYUNetID
[NYUNetID@log-0 ~]$ pwd
/scratch/NYUNetID

/home and /scratch are separate filesystems in separate places, but you should use /scratch to store your files.

Loading Modules

Slurm allows you to load and manage multiple versions and configurations of software packages.

To see available package environments:

module avail

To load a model:

module load [package name]

For example if you want to use Tensorflow-gpu:

module load cudnn/8.0v6.0
module load cuda/8.0.44
module load tensorflow/python3.6/1.3.0

To check what is currently loaded:

module list

To remove all packages:

module purge

To get helpful information about the package:

module show torch/gnu/20170504

Will print something like

--------------------------------------------------------------------------------------------------------------------------------------------------
   /share/apps/modulefiles/torch/gnu/20170504.lua:
--------------------------------------------------------------------------------------------------------------------------------------------------
whatis("Torch: a scientific computing framework with wide support for machine learning algorithms that puts GPUs first")
whatis("Name: torch version: 20170504 compilers: gnu")
load("cmake/intel/3.7.1")
load("cuda/8.0.44")
load("cudnn/8.0v5.1")
load("magma/intel/2.2.0")
...

load(...) are the dependencies that are also loaded when you load a package.

 

https://wikis.nyu.edu/display/NYUHPC/Slurm+Tutorial

https://github.com/cvalenzuela/hpc

https://wikis.nyu.edu/display/NYUHPC/Getting+or+renewing+an+HPC+account