Gautschi Scheduler Changes Coming November 2: Shorter Waits, New Burst QOS

As Gautschi's utilization has ramped up, the RCAC team reached out to researchers to understand their workflows and collect feedback to guide the evolution of this resource. While the AI partition's near 100% utilization is a success by many metrics, many of you have pointed out that jobs using the normal QOS have begun to experience longer-than-expected wait times and that preemptible jobs are unlikely to start due to the near 100% occupancy of normal QOS jobs. To make sure that the Gautschi AI cluster delivers a positive experience while providing the desired AI cycles, we're making a set of scheduler changes to shorten waits, make capacity more predictable, enable new ways to get whole-node access when you need it, and make checkpointing easier to perform when using the preemptible QOS. This blog post will provide some of the data driving the decisions that we are making.

What's Changing

The following changes will take effect on Monday, November 2, 2026.

1. Proportional GPU Limits: The per-account limit of 24 GPUs is being reduced to double the number of GPUs purchased or 2 GPUs, whichever is larger.
2. Reduced Max Walltimes: The current two-week max walltime for normal QOS jobs is being reduced to 4 days.
3. New burst QOS: A new quality of service will be created called burst with the following policies:
   1. Jobs using the burst QOS can use up to 8 GPUs at once for up to 8 hours at a time.
   2. Jobs with this QOS will have the highest priority on the cluster for more immediate start time, however they will only be able to preempt preemptible jobs. Normal QOS jobs will remain unaffected.
   3. Only one job from each account on the cluster can use this QOS at a time.
   4. Jobs using this QOS will be charged at 2x the rate for the GPU hours consumed by their job.
4. New Preemptible Features:
   1. Preemptible jobs will have a GraceTime value of 10 minutes which will allow preemptible jobs a guaranteed 10 minutes to clean-up and checkpoint their work before being interrupted.

Why are New Limits Needed?

As of October 1st, Gautschi has been in production for 19 months, and over that time, we have seen utilization continue to ramp up. Due to Gautschi's allocation method of allowing groups to purchase resources based on GPU hours rather than limiting them to the number of GPUs purchased, groups have been able to use many more GPUs in parallel than purchased. This is by design as we'd like for groups to be able to use the massively parallel capabilities of Gautschi, but while an overly generous limit of 24 H100s may have been suitable for a fledgling cluster with few users, it's become clear that it's impacting fair access to resources and consequently wait times for current users.

Below, we'll provide some of the data that are informing these limit changes as well as provide some discussion on the expected outcomes of each change.

Enforcing Proportional GPU Allocation

Over the past nine months, we've watched the total utilization of the AI partition creep towards 100% while simultaneously watching the percentage of jobs using the "preemptible" QOS fall. This indicates that we are at 100% utilization while there are always "normal" QOS jobs that are waiting to start which has resulted in longer and longer wait times for "normal" QOS jobs and rendered the "preemptible" QOS obsolete due to essentially never having idle cycles.

In large part, the reason for this is because we have allowed all groups on Gautschi to use up to 24 GPUs at once regardless of the size of their purchase into Gautschi. This is distinctly different from how our other community clusters operate. On those clusters, the amount of resources purchased by a group directly affects the priority of their jobs on that cluster through the use of Slurm's "Fairshare" feature by increasing the group's respective "share" of the cluster. For reasons explained in this blog, Gautschi cannot use "Fairshare". This presents a challenge for groups who want to capacity plan for their usage on Gautschi.

In order to address this, we'll be reducing the maximum number of GPUs that can be used by a group under the normal QOS from a flat 24 GPUs per group to a limit proportional to the size of a group's purchase. By scaling proportionally, this will allow us to schedule more fairly while guaranteeing capacity to those who choose to invest in more resources within Gautschi. To enable proportional allocation, we will set each group's limit of concurrent H100s under the "normal" QOS to be twice the number of GPUs purchased or two GPUs, whichever is larger.

When reviewing the average usage of all 97 groups on Gautschi since the start of the semester, 78.3% of groups would not notice this limit, and the new configuration would ensure larger availability of resources for all users, including the current largest consumers of Gautschi.

Why was "2x" the purchase chosen as the scaling factor?

A valid question you may be asking is, "How was the 2x factor decided?". In order to inform this decision, we looked at the average number of GPUs each group has used since the start of the semester and compared how many idle GPUs would be available for pending jobs if the number of GPUs for each group was restricted under each new policy.

Design Options	Number of GPUs freed up for pending jobs	% Of Gautschi free for pending jobs
Groups can allocate 2x their purchase	50.45	31.53%
Groups can allocate 4x their purchase	9.38	5.86%
Reduce per group limit from 24 to 16	0.87	0.54%

Initially, we thought to simply reduce the number of GPUs each group can use from 24 to 16, but upon looking at the data, we saw that this would not practically change anything. We next considered allowing groups to use up to 4x the number of GPUs they purchased, and again we saw that this would only increase the availability of GPUs to serve normal jobs by a little more than 5%. Finally, when looking at a factor of 2x we saw that this would greatly improve the availability of GPUs to "preemptible" jobs and still allow groups who need all eight GPUs on a Gautschi node the ability to be able to purchase more GPUs and expect that there be idle GPUs for them to run on.

How to request larger quantities of GPUs?

We're introducing a new QOS called burst that will, in exchange for using twice the GPU hours as a normal QOS job, allow groups to use up to eight GPUs for a maximum walltime of eight hours. This will allow users to leverage the fast GPU-to-GPU connection provided in 8-GPU nodes. These jobs will have the highest priority of any job on the cluster. While a burst job cannot preempt a normal QOS job, it will always start before a normal QOS job. Because of the higher resource limits and priority, each group will only be allowed to have one burst job running and one accruing priority at a time. If you expect to need large numbers of GPUs frequently, consider purchasing enough to cover up to half of your regular GPU usage.

We hope that in addition to meeting the resource needs of groups who only need whole node jobs occasionally that this will also be a tool for groups who are willing to pay for more "interactive" access as this has been a request from many PIs.

Reducing Max Walltime per Job

Another limit that we are reducing in this tuning pass is the maximum walltime for each job under the normal QOS. There are a couple of reasons for this, but it essentially boils down to the fact that, empirically, users are requesting many, many more hours/days than their jobs require, and this makes it harder for Slurm to give estimates on start time to users.

When looking at the statistics of all 508,234 jobs that have run in the AI partition from March 1, 2025, to October 12, 2026, we see that less than 0.1% of jobs are using more than 3 days of walltime despite around 5% of jobs requesting it. We're choosing to extend an additional day to give a little bit of wiggle room to that last 0.1%.

Percentile	Requested Walltime	Used Walltime
90%	1-02:00:00	01:54:00
99%	14-00:00:00	22:54:00
99.9%	14-00:00:00	3-00:00:00

Reducing the maximum walltime should allow Slurm to be more efficient in backfilling users' jobs. For a more in-depth discussion as to why that is the case, see the discussion on backfill scheduling in this blog. We recommend that users continue to implement and leverage checkpointing as a best practice to make efficient use of shorter walltimes and preemptible jobs.

New Preemptible Feature for Enabling Checkpointing

As discussed above, in recent months the preemptible QOS has been rendered unusable by the fact that there is always an eligible "normal" QOS job ready to preempt any preemptible job that starts. We expect under these new limits on Gautschi, the resources available to run preemptible jobs without interruption will increase considerably, and we encourage groups, particularly those with multi-GPU needs, to take advantage of these resources by submitting jobs under this quality of service.

To help make this transition smoother for groups that haven't used the preemptible QOS before (and for those that have), we will be adding a GraceTime of 10 minutes to preemptible jobs. What this means is that once your job has been selected to be interrupted by a higher priority job, it will be "told" through a Unix signal, that it has 10 minutes to gracefully shut down. We hope that this will provide an easy mechanism for users to checkpoint their work.