1
<!--#include virtual="header.txt"-->
3
<H1>Gang Scheduling</H1>
6
SLURM version 1.2 and earlier supported dedication of resources
8
Beginning in SLURM version 1.3, gang scheduling is supported.
9
Gang scheduling is when two or more jobs are allocated to the same resources
10
and these jobs are alternately suspended to let all of the tasks of each
11
job have full access to the shared resources for a period of time.
14
A resource manager that supports timeslicing can improve it's responsiveness
15
and utilization by allowing more jobs to begin running sooner. Shorter-running
16
jobs no longer have to wait in a queue behind longer-running jobs. Instead they
17
can be run "in parallel" with the longer-running jobs, which will allow them
18
to finish quicker. Throughput is also improved because overcommitting the
19
resources provides opportunities for "local backfilling" to occur (see example
23
The SLURM 1.3.0 the <I>sched/gang</I> plugin provides timeslicing. When enabled,
24
it monitors each of the partitions in SLURM. If a new job has been allocated to
25
resources in a partition that have already been allocated to an existing job,
26
then the plugin will suspend the new job until the configured
27
<I>SchedulerTimeslice</I> interval has elapsed. Then it will suspend the
28
running job and let the new job make use of the resources for a
29
<I>SchedulerTimeslice</I> interval. This will continue until one of the
33
<H2>Configuration</H2>
35
There are several important configuration parameters relating to
40
<B>SelectType</B>: The SLURM <I>sched/gang</I> plugin supports nodes
41
allocated by the <I>select/linear</I> plugin and socket/core/CPU resources
42
allocated by the <I>select/cons_res</I> plugin.
45
<B>SelectTypeParameter</B>: Since resources will be getting overallocated
46
with jobs, the resource selection plugin should be configured to track the
47
amount of memory used by each job to ensure that memory page swapping does
48
not occur. When <I>select/linear</I> is chosen, we recommend setting
49
<I>SelectTypeParameter=CR_Memory</I>. When <I>select/cons_res</I> is
50
chosen, we recommend including Memory as a resource (ex.
51
<I>SelectTypeParameter=CR_Core_Memory</I>).
54
<B>DefMemPerTask</B>: Since job requests may not explicitly specify
55
a memory requirement, we also recommend configuring <I>DefMemPerTask</I>
56
(default memory per task). It may also be desirable to configure
57
<I>MaxMemPerTask</I> (maximum memory per task) in <I>slurm.conf</I>.
60
<B>JobAcctGatherType and JobAcctGatherFrequency</B>:
61
If you wish to enforce memory limits, accounting must be enabled
62
using the <I>JobAcctGatherType</I> and <I>JobAcctGatherFrequency</I>
63
parameters. If accounting is enabled and a job exceeds its configured
64
memory limits, it will be canceled in order to prevent it from
65
adversely effecting other jobs sharing the same resources.
68
<B>SchedulerType</B>: Configure the <I>sched/gang</I> plugin by setting
69
<I>SchedulerType=sched/gang</I> in <I>slurm.conf</I>.
72
<B>SchedulerTimeSlice</B>: The default timeslice interval is 30 seconds.
73
To change this duration, set <I>SchedulerTimeSlice</I> to the desired interval
74
(in seconds) in <I>slurm.conf</I>. For example, to set the timeslice interval
75
to one minute, set <I>SchedulerTimeSlice=60</I>. Short values can increase
76
the overhead of gang scheduling.
79
<B>Shared</B>: Configure the partitions <I>Shared</I> setting to
80
<I>FORCE</I> for all partitions in which timeslicing is to take place.
81
The <I>FORCE</I> option now supports an additional parameter that controls
82
how many jobs can share a resource (FORCE[:max_share]). By default the
83
max_share value is 4. To allow up to 6 jobs from this partition to be
84
allocated to a common resource, set <I>Shared=FORCE:6</I>.
88
In order to enable gang scheduling after making the configuration changes
89
described above, restart SLURM if it is already running. Any change to the
90
plugin settings in SLURM requires a full restart of the daemons. If you
91
just change the partition <I>Shared</I> setting, this can be updated with
92
<I>scontrol reconfig</I>.
95
For an advanced topic discussion on the potential use of swap space,
96
see "Making use of swap space" in the "Future Work" section below.
99
<H2>Timeslicer Design and Operation</H2>
102
When enabled, the <I>sched/gang</I> plugin keeps track of the resources
103
allocated to all jobs. For each partition an "active bitmap" is maintained that
104
tracks all concurrently running jobs in the SLURM cluster. Each time a new
105
job is allocated to resources in a partition, the <I>sched/gang</I> plugin
106
compares these newly allocated resources with the resources already maintained
107
in the "active bitmap". If these two sets of resources are disjoint then the new
108
job is added to the "active bitmap". If these two sets of resources overlap then
109
the new job is suspended. All jobs are tracked in a per-partition job queue
110
within the <I>sched/gang</I> plugin.
113
A separate <I>timeslicer thread</I> is spawned by the <I>sched/gang</I> plugin
114
on startup. This thread sleeps for the configured <I>SchedulerTimeSlice</I>
115
interval. When it wakes up, it checks each partition for suspended jobs. If
116
suspended jobs are found then the <I>timeslicer thread</I> moves all running
117
jobs to the end of the job queue. It then reconstructs the "active bitmap" for
118
this partition beginning with the suspended job that has waited the longest to
119
run (this will be the first suspended job in the run queue). Each following job
120
is then compared with the new "active bitmap", and if the job can be run
121
concurrently with the other "active" jobs then the job is added. Once this is
122
complete then the <I>timeslicer thread</I> suspends any currently running jobs
123
that are no longer part of the "active bitmap", and resumes jobs that are new to
127
This <I>timeslicer thread</I> algorithm for rotating jobs is designed to prevent
128
jobs from starving (remaining in the suspended state indefinitly) and to be as
129
fair as possible in the distribution of runtime while still keeping all of the
130
resources as busy as possible.
133
The <I>sched/gang</I> plugin suspends jobs via the same internal functions that
134
support <I>scontrol suspend</I> and <I>scontrol resume</I>. A good way to
135
observe the operation of the timeslicer is by running <I>watch squeue</I> in a
139
<H2>A Simple Example</H2>
142
The following example is configured with <I>select/linear</I>,
143
<I>sched/gang</I>, and <I>Shared=FORCE</I>. This example takes place on a small
147
[user@n16 load]$ <B>sinfo</B>
148
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
149
active* up infinite 5 idle n[12-16]
152
Here are the Scheduler settings (the last two settings are the relevant ones):
155
[user@n16 load]$ <B>scontrol show config | grep Sched</B>
158
SchedulerRootFilter = 1
159
SchedulerTimeSlice = 30
160
SchedulerType = sched/gang
164
The <I>myload</I> script launches a simple load-generating app that runs
165
for the given number of seconds. Submit <I>myload</I> to run on all nodes:
168
[user@n16 load]$ <B>sbatch -N5 ./myload 300</B>
169
sbatch: Submitted batch job 3
170
[user@n16 load]$ <B>squeue</B>
171
JOBID PARTITION NAME USER ST TIME NODES NODELIST
172
3 active myload user 0:05 5 n[12-16]
175
Submit it again and watch the <I>sched/gang</I> plugin suspend it:
178
[user@n16 load]$ <B>sbatch -N5 ./myload 300</B>
179
sbatch: Submitted batch job 4
180
[user@n16 load]$ <B>squeue</B>
181
JOBID PARTITION NAME USER ST TIME NODES NODELIST
182
3 active myload user R 0:13 5 n[12-16]
183
4 active myload user S 0:00 5 n[12-16]
186
After 30 seconds the <I>sched/gang</I> plugin swaps jobs, and now job 4 is the
190
[user@n16 load]$ <B>squeue</B>
191
JOBID PARTITION NAME USER ST TIME NODES NODELIST
192
4 active myload user R 0:08 5 n[12-16]
193
3 active myload user S 0:41 5 n[12-16]
194
[user@n16 load]$ <B>squeue</B>
195
JOBID PARTITION NAME USER ST TIME NODES NODELIST
196
4 active myload user R 0:21 5 n[12-16]
197
3 active myload user S 0:41 5 n[12-16]
b'After another 30 seconds the <I>sched/gang</I> plugin sets job 3 running again:'
202
[user@n16 load]$ <B>squeue</B>
203
JOBID PARTITION NAME USER ST TIME NODES NODELIST
204
3 active myload user R 0:50 5 n[12-16]
205
4 active myload user S 0:30 5 n[12-16]
208
<B>A possible side effect of timeslicing</B>: Note that jobs that are
209
immediately suspended may cause their srun commands to produce the following
213
[user@n16 load]$ <B>cat slurm-4.out</B>
214
srun: Job step creation temporarily disabled, retrying
215
srun: Job step creation still disabled, retrying
216
srun: Job step creation still disabled, retrying
217
srun: Job step creation still disabled, retrying
218
srun: Job step created
221
This occurs because <I>srun</I> is attempting to launch a jobstep in an
222
allocation that has been suspended. The <I>srun</I> process will continue in a
223
retry loop to launch the jobstep until the allocation has been resumed and the
224
jobstep can be launched.
227
When the <I>sched/gang</I> plugin is enabled, this type of output in the user
228
jobs should be considered benign.
231
<H2>More examples</H2>
233
The following example shows how the timeslicer algorithm keeps the resources
234
busy. Job 10 runs continually, while jobs 9 and 11 are timesliced:
237
[user@n16 load]$ <B>sbatch -N3 ./myload 300</B>
238
sbatch: Submitted batch job 9
239
[user@n16 load]$ <B>sbatch -N2 ./myload 300</B>
240
sbatch: Submitted batch job 10
241
[user@n16 load]$ <B>sbatch -N3 ./myload 300</B>
242
sbatch: Submitted batch job 11
243
[user@n16 load]$ <B>squeue</B>
244
JOBID PARTITION NAME USER ST TIME NODES NODELIST
245
9 active myload user R 0:11 3 n[12-14]
246
10 active myload user R 0:08 2 n[15-16]
247
11 active myload user S 0:00 3 n[12-14]
248
[user@n16 load]$ <B>squeue</B>
249
JOBID PARTITION NAME USER ST TIME NODES NODELIST
250
10 active myload user R 0:50 2 n[15-16]
251
11 active myload user R 0:12 3 n[12-14]
252
9 active myload user S 0:41 3 n[12-14]
253
[user@n16 load]$ <B>squeue</B>
254
JOBID PARTITION NAME USER ST TIME NODES NODELIST
255
10 active myload user R 1:04 2 n[15-16]
256
11 active myload user R 0:26 3 n[12-14]
257
9 active myload user S 0:41 3 n[12-14]
258
[user@n16 load]$ <B>squeue</B>
259
JOBID PARTITION NAME USER ST TIME NODES NODELIST
260
9 active myload user R 0:46 3 n[12-14]
261
10 active myload user R 1:13 2 n[15-16]
262
11 active myload user S 0:30 3 n[12-14]
267
The next example displays "local backfilling":
270
[user@n16 load]$ <B>sbatch -N3 ./myload 300</B>
271
sbatch: Submitted batch job 12
272
[user@n16 load]$ <B>sbatch -N5 ./myload 300</B>
273
sbatch: Submitted batch job 13
274
[user@n16 load]$ <B>sbatch -N2 ./myload 300</B>
275
sbatch: Submitted batch job 14
276
[user@n16 load]$ <B>squeue</B>
277
JOBID PARTITION NAME USER ST TIME NODES NODELIST
278
12 active myload user R 0:14 3 n[12-14]
279
14 active myload user R 0:06 2 n[15-16]
280
13 active myload user S 0:00 5 n[12-16]
284
Without timeslicing and without the backfill scheduler enabled, job 14 has to
285
wait for job 13 to finish.
287
This is called "local" backfilling because the backfilling only occurs with jobs
288
close enough in the queue to get allocated by the scheduler as part of
289
oversubscribing the resources. Recall that the number of jobs that can
290
overcommit a resource is controlled by the <I>Shared=FORCE:max_share</I> value,
291
so this value effectively controls the scope of "local backfilling".
293
Normal backfill algorithms check <U>all</U> jobs in the wait queue.
296
<H2>Consumable Resource Examples</H2>
298
The following two examples illustrate the primary difference between
299
<I>CR_CPU</I> and <I>CR_Core</I> when consumable resource selection is enabled
300
(<I>select/cons_res</I>).
303
When <I>CR_CPU</I> (or <I>CR_CPU_Memory</I>) is configured then the selector
304
treats the CPUs as simple, <I>interchangeable</I> computing resources. However
305
when <I>CR_Core</I> (or <I>CR_Core_Memory</I>) is enabled the selector treats
306
the CPUs as individual resources that are <U>specifically</U> allocated to jobs.
307
This subtle difference is highlighted when timeslicing is enabled.
310
In both examples 6 jobs are submitted. Each job requests 2 CPUs per node, and
311
all of the nodes contain two quad-core processors. The timeslicer will initially
312
let the first 4 jobs run and suspend the last 2 jobs. The manner in which these
313
jobs are timesliced depends upon the configured <I>SelectTypeParameter</I>.
316
In the first example <I>CR_Core_Memory</I> is configured. Note that jobs 46 and
317
47 don't <U>ever</U> get suspended. This is because they are not sharing their
318
cores with any other job. Jobs 48 and 49 were allocated to the same cores as
319
jobs 45 and 46. The timeslicer recognizes this and timeslices only those jobs:
322
[user@n16 load]$ <B>sinfo</B>
323
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
324
active* up infinite 5 idle n[12-16]
325
[user@n16 load]$ <B>scontrol show config | grep Select</B>
326
SelectType = select/cons_res
327
SelectTypeParameters = CR_CORE_MEMORY
328
[user@n16 load]$ <B>sinfo -o "%20N %5D %5c %5z"</B>
329
NODELIST NODES CPUS S:C:T
333
[user@n16 load]$ <B>sbatch -n10 -N5 ./myload 300</B>
334
sbatch: Submitted batch job 44
335
[user@n16 load]$ <B>sbatch -n10 -N5 ./myload 300</B>
336
sbatch: Submitted batch job 45
337
[user@n16 load]$ <B>sbatch -n10 -N5 ./myload 300</B>
338
sbatch: Submitted batch job 46
339
[user@n16 load]$ <B>sbatch -n10 -N5 ./myload 300</B>
340
sbatch: Submitted batch job 47
341
[user@n16 load]$ <B>sbatch -n10 -N5 ./myload 300</B>
342
sbatch: Submitted batch job 48
343
[user@n16 load]$ <B>sbatch -n10 -N5 ./myload 300</B>
344
sbatch: Submitted batch job 49
345
[user@n16 load]$ <B>squeue</B>
346
JOBID PARTITION NAME USER ST TIME NODES NODELIST
347
44 active myload user R 0:09 5 n[12-16]
348
45 active myload user R 0:08 5 n[12-16]
349
46 active myload user R 0:08 5 n[12-16]
350
47 active myload user R 0:07 5 n[12-16]
351
48 active myload user S 0:00 5 n[12-16]
352
49 active myload user S 0:00 5 n[12-16]
353
[user@n16 load]$ <B>squeue</B>
354
JOBID PARTITION NAME USER ST TIME NODES NODELIST
355
46 active myload user R 0:49 5 n[12-16]
356
47 active myload user R 0:48 5 n[12-16]
357
48 active myload user R 0:06 5 n[12-16]
358
49 active myload user R 0:06 5 n[12-16]
359
44 active myload user S 0:44 5 n[12-16]
360
45 active myload user S 0:43 5 n[12-16]
361
[user@n16 load]$ <B>squeue</B>
362
JOBID PARTITION NAME USER ST TIME NODES NODELIST
363
44 active myload user R 1:23 5 n[12-16]
364
45 active myload user R 1:22 5 n[12-16]
365
46 active myload user R 2:22 5 n[12-16]
366
47 active myload user R 2:21 5 n[12-16]
367
48 active myload user S 1:00 5 n[12-16]
368
49 active myload user S 1:00 5 n[12-16]
372
Note the runtime of all 6 jobs in the output of the last <I>squeue</I> command.
373
Jobs 46 and 47 have been running continuously, while jobs 45 and 46 are
374
splitting their runtime with jobs 48 and 49.
376
The next example has <I>CR_CPU_Memory</I> configured and the same 6 jobs are
377
submitted. Here the selector and the timeslicer treat the CPUs as countable
378
resources which results in all 6 jobs sharing time on the CPUs:
381
[user@n16 load]$ <B>sinfo</B>
382
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
383
active* up infinite 5 idle n[12-16]
384
[user@n16 load]$ <B>scontrol show config | grep Select</B>
385
SelectType = select/cons_res
386
SelectTypeParameters = CR_CPU_MEMORY
387
[user@n16 load]$ <B>sinfo -o "%20N %5D %5c %5z"</B>
388
NODELIST NODES CPUS S:C:T
392
[user@n16 load]$ <B>sbatch -n10 -N5 ./myload 300</B>
393
sbatch: Submitted batch job 51
394
[user@n16 load]$ <B>sbatch -n10 -N5 ./myload 300</B>
395
sbatch: Submitted batch job 52
396
[user@n16 load]$ <B>sbatch -n10 -N5 ./myload 300</B>
397
sbatch: Submitted batch job 53
398
[user@n16 load]$ <B>sbatch -n10 -N5 ./myload 300</B>
399
sbatch: Submitted batch job 54
400
[user@n16 load]$ <B>sbatch -n10 -N5 ./myload 300</B>
401
sbatch: Submitted batch job 55
402
[user@n16 load]$ <B>sbatch -n10 -N5 ./myload 300</B>
403
sbatch: Submitted batch job 56
404
[user@n16 load]$ <B>squeue</B>
405
JOBID PARTITION NAME USER ST TIME NODES NODELIST
406
51 active myload user R 0:11 5 n[12-16]
407
52 active myload user R 0:11 5 n[12-16]
408
53 active myload user R 0:10 5 n[12-16]
409
54 active myload user R 0:09 5 n[12-16]
410
55 active myload user S 0:00 5 n[12-16]
411
56 active myload user S 0:00 5 n[12-16]
412
[user@n16 load]$ <B>squeue</B>
413
JOBID PARTITION NAME USER ST TIME NODES NODELIST
414
51 active myload user R 1:09 5 n[12-16]
415
52 active myload user R 1:09 5 n[12-16]
416
55 active myload user R 0:23 5 n[12-16]
417
56 active myload user R 0:23 5 n[12-16]
418
53 active myload user S 0:45 5 n[12-16]
419
54 active myload user S 0:44 5 n[12-16]
420
[user@n16 load]$ <B>squeue</B>
421
JOBID PARTITION NAME USER ST TIME NODES NODELIST
422
53 active myload user R 0:55 5 n[12-16]
423
54 active myload user R 0:54 5 n[12-16]
424
55 active myload user R 0:40 5 n[12-16]
425
56 active myload user R 0:40 5 n[12-16]
426
51 active myload user S 1:16 5 n[12-16]
427
52 active myload user S 1:16 5 n[12-16]
428
[user@n16 load]$ <B>squeue</B>
429
JOBID PARTITION NAME USER ST TIME NODES NODELIST
430
51 active myload user R 3:18 5 n[12-16]
431
52 active myload user R 3:18 5 n[12-16]
432
53 active myload user R 3:17 5 n[12-16]
433
54 active myload user R 3:16 5 n[12-16]
434
55 active myload user S 3:00 5 n[12-16]
435
56 active myload user S 3:00 5 n[12-16]
439
Note that the runtime of all 6 jobs is roughly equal. Jobs 51-54 ran first so
440
they're slightly ahead, but so far all jobs have run for at least 3 minutes.
442
At the core level this means that SLURM relies on the linux kernel to move jobs
443
around on the cores to maximize performance. This is different than when
444
<I>CR_Core_Memory</I> was configured and the jobs would effectively remain
445
"pinned" to their specific cores for the duration of the job. Note that
446
<I>CR_Core_Memory</I> supports CPU binding, while <I>CR_CPU_Memory</I> does not.
452
Priority scheduling and preemptive scheduling are other forms of gang
453
scheduling that are currently under development for SLURM.
456
<B>Making use of swap space</B>: (note that this topic is not currently
457
scheduled for development, unless someone would like to pursue this) It should
458
be noted that timeslicing does provide an interesting mechanism for high
459
performance jobs to make use of swap space. The optimal scenario is one in which
460
suspended jobs are "swapped out" and active jobs are "swapped in". The swapping
461
activity would only occur once every <I>SchedulerTimeslice</I> interval.
464
However, SLURM should first be modified to include support for scheduling jobs
465
into swap space and to provide controls to prevent overcommitting swap space.
466
For now this idea could be experimented with by disabling memory support in the
467
selector and submitting appropriately sized jobs.
470
<p style="text-align:center;">Last modified 17 March 2008</p>
472
<!--#include virtual="footer.txt"-->
added
removed
doc/html/gang_scheduling.shtml
