Differences

This shows you the differences between two versions of the page.

--- tech:slurm [2020/02/10 16:25] – [Controller] kohofer
+++ tech:slurm [2020/04/08 17:39] – [Modules] kohofer
@@ Line 14: / Line 14: @@
 ===== Installation =====
-==== Controller ====
 ===== Controller name: slurm-ctrl =====
@@ Line 131: / Line 129: @@
 EnvironmentFile=-/etc/default/slurmctld
 ExecStart=/usr/sbin/slurmctld $SLURMCTLD_OPTIONS
+ExecStartPost=/bin/sleep 2
 ExecReload=/bin/kill -HUP $MAINPID
 PIDFile=/var/run/slurm-llnl/slurmctld.pid
@@ Line 152: / Line 151: @@
 EnvironmentFile=-/etc/default/slurmd
 ExecStart=/usr/sbin/slurmd $SLURMD_OPTIONS
+ExecStartPost=/bin/sleep 2
 ExecReload=/bin/kill -HUP $MAINPID
 PIDFile=/var/run/slurm-llnl/slurmd.pid
@@ Line 241: / Line 241: @@
   debug*       up   infinite      1   idle linux1
-==== Compute Nodes ====
+If computer node is down
+<code>
+sinfo -a
+PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
+debug*       up   infinite      2   down gpu[02-03]
+</code>
+  scontrol update nodename=gpu02 state=idle
+  scontrol update nodename=gpu03 state=idle
+  scontrol update nodename=gpu02 state=resume
+<code>
+sinfo -a
+PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
+debug*       up   infinite      2   idle gpu[02-03]
+</code>
+===== Compute Nodes =====
 A compute node is a machine which will receive jobs to execute, sent from the Controller, it runs the slurmd service.
@@ Line 251: / Line 271: @@
   ssh -l csadmin <compute-nodes> 10.7.20.109 10.7.20.110
   sudo apt install slurm-wlm libmunge-dev libmunge2 munge
+  sudo vi /lib/systemd/system/slurmd.service
+<code>
+[Unit]
+Description=Slurm node daemon
+After=network.target munge.service
+ConditionPathExists=/etc/slurm-llnl/slurm.conf
+Documentation=man:slurmd(8)
+[Service]
+Type=forking
+EnvironmentFile=-/etc/default/slurmd
+ExecStart=/usr/sbin/slurmd $SLURMD_OPTIONS
+ExecStartPost=/bin/sleep 2
+ExecReload=/bin/kill -HUP $MAINPID
+PIDFile=/var/run/slurm-llnl/slurmd.pid
+KillMode=process
+LimitNOFILE=51200
+LimitMEMLOCK=infinity
+LimitSTACK=infinity
+[Install]
+WantedBy=multi-user.target
+</code>
   sudo systemctl enable slurmd
   sudo systemctl enable munge
@@ Line 261: / Line 307: @@
   ssh-keygen
-Copy ssh-keys to slurm-ctrl (using IP, because no DNS in place)
+Copy ssh-keys to slurm-ctrl
-  ssh-copy-id -i ~/.ssh/id_rsa.pub csadmin@10.7.20.97:
+  ssh-copy-id -i ~/.ssh/id_rsa.pub csadmin@slurm-ctrl.inf.unibz.it:
 Become root to do important things:
@@ Line 307: / Line 353: @@
 {{ :tech:9-slurm.pdf |Linux Clusters Institute: Scheduling and Resource Management 2017}}
+====== Modules ======
+===== Links =====
+http://www.walkingrandomly.com/?p=5680
+https://modules.readthedocs.io/en/latest/index.html