job_submit/lua - add "core_spec" to job_desc HPC-12614

CHANGELOG/slurm-25.11.md

@@ -1,3 +1,47 @@
+## Changes in 25.11.3
+
+* Fix regression from af2c0bd which caused usercpu and systemcpu to be missing for job steps.
+* Fixed issue where RestrictedCoresPerGPU with shared gres are limited to using restricted cores on one job per sharing gres.
+* slurmd - Fix regression that could cause thread limits to not be enforced for handling incoming RPCs.
+* Fix "sacctmgr show conf" to properly display CommitDelay in seconds instead of as a boolean.
+* Fix cron/requeued jobs being incorrectly reported as runaway
+* slurmctld - Prevent the double-removal of accounting usage for jobs being requeued that are in the COMPLETED or COMPLETING state.
+* When deleting a QOS from the DB, also remove it from partition QOS, AllowQOS and DenyQOS fields.
+* Fixed bug that could cause the detected CPU count to be lower than actual available CPU count. This bug could have resulted in the default value for conmgr_threads being lower than the number of available CPUs in sackd, scrun, slurmctld, slurmscriptd, slurmd, slurmstepd, slurmdbd, and slurmrestd when the assigned CPUs are not sequential.
+* slurmdbd - Prevent the following slurmdbd.conf options from overriding the default values of any in the list not specified: AllowNoDefAcct, AllResourcesAbsolute, DisableCoordDBD, DisableArchiveCommands.
+* salloc/sbatch - Nesting a non-stepmgr salloc or sbatch inside an existing job allocation that enabled the stepmgr will no longer result in the inner job’s steps failing to launch.
+* Prevent slurmd -G from initializing sack processing thread.
+* Added SLURM_CLUSTER_NAME, SLURM_JOB_ACCOUNT and SLURM_JOB_GROUP environment variables when a step is launched.
+* slurmctld - Prevent marking external nodes as being unresponsive when reconfiguring if SlurmctldParameters=enable_configless is used.
+* Fix potential segfault when attempting to look up the controller address via DNS in configless mode.
+* Fix "undefined symbol: gpu_common_underscorify_tolower" when gpu/nrt plugin in use.
+* slurmrestd - Avoid memory leak on authentication failures with invalid bearer tokens.
+* Fix potential deadlock in _x11_signal_handler() during stepd_cleanup().
+* slurmctld - Fix reservations AllowedPartitions logic leading to incorrect purge of valid reservations in some use-cases.
+* slurmcltd - Avoid persistent connections hangs when enable_async_reply is configured.
+* Prevent potential controller segfault when reconfiguring after gres file updates.
+* Reparent slurmd to a subcgroup to avoid conflicting with systemd.
+* Fix sprio regression not handling comma separated list of jobids.
+* slurmctld,slurmd - Fix memory leak when container ID is populated.
+* slurmd - Fix P-core detection on processors with varying P-core frequencies and in cpuset-restricted environments.
+* namespace/linux - add disable_bpf_token option.
+* slurmctld - Avoid expedited requeue triggering a job to requeue when job exit code was zero.
+* slurmctld - Avoid expedited requeue of jobs while waiting for job epilog script to complete.
+* slurmctld - Prevent removing cloud nodes from the topology when putting them in the POWERED_DOWN state if they are present in topology.conf or topology.yaml and their node configuration did not specify the Topology option.
+* interfaces/topology - When modifying a nodes topology with the Topology option in slurm.conf or the slurmd --conf Topology, change the topology to fully match the new topology.
+* slurmctld - Allow changes to topology.conf or topology.yaml, and slurm.conf node configuration Topology option to take effect on a reconfigure or restart when power saving is enabled.
+* slurmctld - Prevent backfill from combining future timeslots if they have different license reservations.
+* Fix CLOUD nodes infrequently becoming FUTURE on slurmctld restart.
+* slurmdbd - Avoid race condition that could cause a hang during shutdown when incoming connection fails.
+* slurmdbd - Avoid crash during shutdown due to `sacctmgr shutdown` request.
+* Fix slurmctld assertion when using "enable_async_reply" and certmgr is used for a TLS enabled cluster.
+* Fix potential slurmd process leak when handling --get-user-env.
+* slurmcltd - Avoid race condition that could cause the StateSaveLocation updates to be missed during shutdown.
+* slurmcltd - Avoid race condition that could cause slurmctld to hang during shutdown before updating StateSaveLocation.
+* slurmctld - Avoid race condition that could cause shutdown to wait on the wrong thread.
+* Fix handling of 0 node test allocations in topology/block.
+* slurmctld - In backfill, prevent unnecessarily testing jobs at future times using the select plugin if it is guaranteed to fail.
+
 ## Changes in 25.11.2
 
 * slurmstepd - Revert regression that would apply job environment to container runtime invocation.

META

@@ -7,8 +7,8 @@
   Name:		slurm
   Major:	25
   Minor:	11
-  Micro:	2
-  Version:	25.11.2
+  Micro:	3
+  Version:	25.11.3
   Release:	1
 
 ##

debian/changelog

@@ -1,4 +1,4 @@
-slurm-smd (25.11.2-1) UNRELEASED; urgency=medium
+slurm-smd (25.11.3-1) UNRELEASED; urgency=medium
 
   * Initial release.
 

doc/html/documentation.shtml

@@ -45,7 +45,6 @@ may be found in the <a href="https://slurm.schedmd.com/archive/">archive</a>.
 <li><a href="dynamic_nodes.html">Dynamic Nodes</a></li>
 <li><a href="elasticsearch.html">Elasticsearch Guide</a></li>
 <li><a href="jobcomp_kafka.html">Job Completion Kafka plugin Guide</a></li>
-<li><a href="namespace_tmpfs.html">namespace/tmpfs - Job Specific Temporary File Management</a></li>
 <li><a href="jwt.html">JSON Web Tokens Authentication</a></li>
 <li><a href="federation.html">Federated Scheduling Guide</a></li>
 <li><a href="pam_slurm_adopt.html">Job Containment (SSH Session Control) with pam_slurm_adopt</a></li>
@@ -55,6 +54,7 @@ may be found in the <a href="https://slurm.schedmd.com/archive/">archive</a>.
 <li><a href="hres.html">Hierarchical Resources (HRES)</a> (in Beta)</li>
 <li><a href="mcs.html">Multi-Category Security (MCS) Guide</a></li>
 <li><a href="nss_slurm.html">Name Service Caching Through NSS Slurm</a></li>
+<li><a href="namespace.html">Namespace Plugins (replaces job_container plugins)</a></li>
 <li><a href="network.html">Network Configuration Guide</a></li>
 <li><a href="openapi_release_notes.html">OpenAPI Plugin Release Notes</a></li>
 <li><a href="power_save.html">Power Saving Guide (power down idle nodes)</a></li>

doc/html/openapi_release_notes/slurm-25.11.html

@@ -1,3 +1,69 @@
+<h2 id="25113">Slurm 25.11.3<a class="slurm_link" href="#25113"></a></h2>
+<h3>data_parser/v0.0.44</h3>
+<h4>Modified Fields</h4>
+<table class="tchange">
+<tr class="trchange">
+	<td class="tdfield">Replaced</td>
+	<td class="tdchange">.info.version = "Slurm-25.11.3";</td>
+</tr>
+<tr class="trchange">
+	<td class="tdfield">Replaced</td>
+	<td class="tdchange">.info["x-slurm"].release = "25.11.3";</td>
+</tr>
+<tr class="trchange">
+	<td class="tdfield">Replaced</td>
+	<td class="tdchange">.info["x-slurm"].version.micro = "3";</td>
+</tr>
+</table>
+<h3>data_parser/v0.0.43</h3>
+<h4>Modified Fields</h4>
+<table class="tchange">
+<tr class="trchange">
+	<td class="tdfield">Replaced</td>
+	<td class="tdchange">.info.version = "Slurm-25.11.3";</td>
+</tr>
+<tr class="trchange">
+	<td class="tdfield">Replaced</td>
+	<td class="tdchange">.info["x-slurm"].release = "25.11.3";</td>
+</tr>
+<tr class="trchange">
+	<td class="tdfield">Replaced</td>
+	<td class="tdchange">.info["x-slurm"].version.micro = "3";</td>
+</tr>
+</table>
+<h3>data_parser/v0.0.42</h3>
+<h4>Modified Fields</h4>
+<table class="tchange">
+<tr class="trchange">
+	<td class="tdfield">Replaced</td>
+	<td class="tdchange">.info.version = "Slurm-25.11.3";</td>
+</tr>
+<tr class="trchange">
+	<td class="tdfield">Replaced</td>
+	<td class="tdchange">.info["x-slurm"].release = "25.11.3";</td>
+</tr>
+<tr class="trchange">
+	<td class="tdfield">Replaced</td>
+	<td class="tdchange">.info["x-slurm"].version.micro = "3";</td>
+</tr>
+</table>
+<h3>data_parser/v0.0.41</h3>
+<h4>Modified Fields</h4>
+<table class="tchange">
+<tr class="trchange">
+	<td class="tdfield">Replaced</td>
+	<td class="tdchange">.info.version = "Slurm-25.11.3";</td>
+</tr>
+<tr class="trchange">
+	<td class="tdfield">Replaced</td>
+	<td class="tdchange">.info["x-slurm"].release = "25.11.3";</td>
+</tr>
+<tr class="trchange">
+	<td class="tdfield">Replaced</td>
+	<td class="tdchange">.info["x-slurm"].version.micro = "3";</td>
+</tr>
+</table>
+
 <h2 id="25112">Slurm 25.11.2<a class="slurm_link" href="#25112"></a></h2>
 <h3>data_parser/v0.0.44</h3>
 <h4>Modified Fields</h4>

doc/html/rest_api.shtml

@@ -3,7 +3,7 @@
     <div class="app-desc">API to access and control Slurm</div>
     <div class="app-desc">More information: <a href="https://www.schedmd.com/">https://www.schedmd.com/</a></div>
     <div class="app-desc">Contact Info: <a href="sales@schedmd.com">sales@schedmd.com</a></div>
-    <div class="app-desc">Version: Slurm-25.11.2</div>
+    <div class="app-desc">Version: Slurm-25.11.3</div>
     <div class="app-desc">BasePath:</div>
     <div class="license-info">Apache 2.0</div>
     <div class="license-url">https://www.apache.org/licenses/LICENSE-2.0.html</div>

doc/man/man1/sshare.1

@@ -157,8 +157,8 @@ from sibling accounts.
 
 .TP
 \f3FairShare\fP
-The Fair\-Share factor, based on a user or account's assigned shares and
-the effective usage charged to them or their accounts.
+The Fair\-Share factor, based on a user's assigned shares and
+the effective usage charged to them.
 .IP
 
 .TP

doc/man/man5/namespace.yaml.5

@@ -45,6 +45,15 @@ Default namespace configuration. If specified this contains the fields described
 in options.
 .IP
 
+.TP
+\fBdisable_bpf_token\fR
+If specified this disables the use of bpf tokens in the namespace plugin. Use of
+this option when ConstrainDevices is set in the cgroup.conf file will result in
+devices only being constrained at the job level, individual steps won't actually
+be constrained if requesting fewer devices than the job does.  This will also
+permit ConstrainDevices to work on kernels where bpf tokens are not available.
+.IP
+
 .TP
 \fBnode_confs\fR
 List of node namespace configurations. The list element attributes are described

doc/man/man5/slurm.conf.5

@@ -6949,17 +6949,12 @@ Multiple node names may be comma separated (e.g. "alpha,beta,gamma")
 and/or a simple node range expression may optionally be used to
 specify numeric ranges of nodes to avoid building a configuration
 file with large numbers of entries.
-The node range expression can contain one pair of square brackets
+The node range expression can contain one or more pairs of square brackets
 with a sequence of comma\-separated numbers and/or ranges of numbers
-separated by a "\-" (e.g. "linux[0\-64,128]", or "lx[15,18,32\-33]").
-Note that the numeric ranges can include one or more leading
-zeros to indicate the numeric portion has a fixed number of digits
+separated by a "\-" (e.g., "linux[0\-64,128]", "lx[15,18,32\-33]", or
+"rack[0\-63]_blade[0\-41]"). Note that the numeric ranges can include one or
+more leading zeros to indicate the numeric portion has a fixed number of digits
 (e.g. "linux[0000\-1023]").
-Multiple numeric ranges can be included in the expression
-(e.g. "rack[0\-63]_blade[0\-41]").
-If one or more numeric expressions are included, one of them
-must be at the end of the name (e.g. "unit[0\-31]rack" is invalid),
-but arbitrary names can always be used in a comma\-separated list.
 .LP
 The node configuration specified the following information:
 

etc/slurmrestd.service.in

@@ -17,7 +17,6 @@ ExecStart=@sbindir@/slurmrestd $SLURMRESTD_OPTIONS
 Environment=SLURM_JWT=daemon
 # Listen on TCP socket by default.
 Environment=SLURMRESTD_LISTEN=:@SLURMRESTD_PORT@
-ExecReload=/bin/kill -HUP $MAINPID
 LimitMEMLOCK=infinity
 
 [Install]

slurm.spec

@@ -1,5 +1,5 @@
 Name:		slurm
-Version:	25.11.2
+Version:	25.11.3
 %define rel	1
 Release:        %{rel}.%{gittag}%{?dist}%{?gpu}.ug
 Summary:	Slurm Workload Manager

src/common/env.c

@@ -2327,7 +2327,7 @@ char **env_array_user_default(const char *username)
 		timeleft -= (now.tv_usec - begin.tv_usec) / 1000;
 		if (timeleft <= 0) {
 			verbose("timeout waiting for "SUCMD" to complete");
-			kill(-child, 9);
+			kill(child, 9);
 			break;
 		}
 		if ((rc = poll(&ufds, 1, timeleft)) <= 0) {
@@ -2370,7 +2370,7 @@ char **env_array_user_default(const char *username)
 	env_array_free(child_args.tmp_env);
 
 	for (config_timeout=0; ; config_timeout++) {
-		kill(-child, SIGKILL);	/* Typically a no-op */
+		kill(child, SIGKILL); /* Typically a no-op */
 		if (config_timeout)
 			sleep(1);
 		if (waitpid(child, &rc, WNOHANG) > 0)

src/common/job_record.c

@@ -243,6 +243,7 @@ extern void job_record_delete(void *job_entry)
 	xfree(job_ptr->burst_buffer_state);
 	xfree(job_ptr->comment);
 	xfree(job_ptr->container);
+	xfree(job_ptr->container_id);
 	xfree(job_ptr->clusters);
 	xfree(job_ptr->cpus_per_tres);
 	xfree(job_ptr->extra);

src/common/persist_conn.c

@@ -424,9 +424,11 @@ extern void slurm_persist_conn_recv_server_fini(void)
 			 * thread_count mutex which this has locked.
 			 * After joining persist_service_conn[i] could be NULL
 			 */
-			slurm_mutex_unlock(&thread_count_lock);
-			slurm_thread_join(thread_id);
-			slurm_mutex_lock(&thread_count_lock);
+			if (thread_id != pthread_self()) {
+				slurm_mutex_unlock(&thread_count_lock);
+				slurm_thread_join(thread_id);
+				slurm_mutex_lock(&thread_count_lock);
+			}
 		}
 		if (persist_service_conn[i] && persist_service_conn[i]->pcon) {
 			void *conn = persist_service_conn[i]->pcon->conn;

src/common/slurm_protocol_defs.c

@@ -1878,6 +1878,7 @@ extern void slurm_free_job_info_members(job_info_t * job)
 		xfree(job->het_job_id_set);
 		xfree(job->job_size_str);
 		xfree(job->licenses);
+		xfree(job->licenses_allocated);
 		xfree(job->mail_user);
 		xfree(job->mcs_label);
 		xfree(job->mem_per_tres);
@@ -1894,6 +1895,7 @@ extern void slurm_free_job_info_members(job_info_t * job)
 		xfree(job->req_node_inx);
 		xfree(job->req_nodes);
 		xfree(job->resv_name);
+		xfree(job->resv_ports);
 		free_job_resources(&job->job_resrcs);
 		xfree(job->selinux_context);
 		xfree(job->state_desc);
@@ -2119,6 +2121,7 @@ extern void slurm_free_job_step_create_request_msg(
 {
 	if (msg) {
 		xfree(msg->container);
+		xfree(msg->container_id);
 		xfree(msg->cpus_per_tres);
 		xfree(msg->exc_nodes);
 		xfree(msg->features);

src/common/slurm_protocol_defs.h

@@ -342,6 +342,8 @@ typedef struct slurm_msg {
 			 * our trust store) */
 	void *conn; /* interfaces/conn data used for sending this message and
 		     * receiving a response */
+	bool conn_is_mtls; /* True if the connection used to receive this message
+			    * used mTLS */
 
 	uint16_t msg_type; /* really a slurm_msg_type_t but needs to be
 			    * this way for packing purposes.  message type */

src/conmgr/rpc.c

@@ -49,6 +49,7 @@
 
 #include "src/conmgr/conmgr.h"
 #include "src/conmgr/mgr.h"
+#include "src/conmgr/tls.h"
 
 static int _try_parse_rpc(conmgr_fd_t *con, slurm_msg_t **msg_ptr)
 {
@@ -153,6 +154,9 @@ static int _try_parse_rpc(conmgr_fd_t *con, slurm_msg_t **msg_ptr)
 			set_buf_offset(msg->buffer, size_buf(rpc));
 		}
 
+		if (con->tls)
+			msg->conn_is_mtls = tls_is_client_authenticated(con);
+
 		/* notify conmgr we processed some data successfully */
 		set_buf_offset(con->in, need);
 	}

src/conmgr/tls.c

@@ -808,3 +808,10 @@ extern void tls_check_fingerprint(conmgr_callback_args_t conmgr_args, void *arg)
 		close_con(false, con);
 	}
 }
+
+extern bool tls_is_client_authenticated(conmgr_fd_t *con)
+{
+	xassert(con->tls);
+
+	return tls_g_is_client_authenticated(con->tls);
+}

src/conmgr/tls.h

@@ -68,4 +68,9 @@ extern void tls_handle_write(conmgr_callback_args_t conmgr_args, void *arg);
 extern void tls_check_fingerprint(conmgr_callback_args_t conmgr_args,
 				  void *arg);
 
+/*
+ * Return true if client is authenticated via TLS
+ */
+extern bool tls_is_client_authenticated(conmgr_fd_t *con);
+
 #endif /* _CONMGR_TLS_H */

src/curl/slurm_curl.c

@@ -170,20 +170,20 @@ extern int slurm_curl_request(const char *data, const char *url,
 	CURL_SETOPT(c, CURLOPT_READFUNCTION, _read_function);
 	CURL_SETOPT(c, CURLOPT_WRITEFUNCTION, _write_callback);
 	CURL_SETOPT(c, CURLOPT_WRITEDATA, (void *) &chunk);
-	CURL_SETOPT(c, CURLOPT_TIMEOUT, timeout);
+	CURL_SETOPT(c, CURLOPT_TIMEOUT, (long) timeout);
 	CURL_SETOPT(c, CURLOPT_URL, url);
 
 	if (!verify_cert) {
 		/* These are needed to work with self-signed certificates */
-		CURL_SETOPT(c, CURLOPT_SSL_VERIFYPEER, 0);
-		CURL_SETOPT(c, CURLOPT_SSL_VERIFYHOST, 0);
+		CURL_SETOPT(c, CURLOPT_SSL_VERIFYPEER, 0L);
+		CURL_SETOPT(c, CURLOPT_SSL_VERIFYHOST, 0L);
 	}
 
 	if (use_mtls) {
 		if (mtls_ca_path) {
 			CURL_SETOPT(c, CURLOPT_CAINFO, mtls_ca_path);
-			CURL_SETOPT(c, CURLOPT_SSL_VERIFYPEER, 1);
-			CURL_SETOPT(c, CURLOPT_SSL_VERIFYHOST, 2);
+			CURL_SETOPT(c, CURLOPT_SSL_VERIFYPEER, (long) 1);
+			CURL_SETOPT(c, CURLOPT_SSL_VERIFYHOST, (long) 2);
 		}
 		CURL_SETOPT(c, CURLOPT_SSLCERT, mtls_cert_path);
 		CURL_SETOPT(c, CURLOPT_SSLKEY, mtls_key_path);
@@ -197,27 +197,27 @@ extern int slurm_curl_request(const char *data, const char *url,
 	switch (request_method) {
 	case HTTP_REQUEST_POST:
 		CURL_SETOPT(c, CURLOPT_CUSTOMREQUEST, NULL);
-		CURL_SETOPT(c, CURLOPT_POST, 1);
+		CURL_SETOPT(c, CURLOPT_POST, 1L);
 		CURL_SETOPT(c, CURLOPT_POSTFIELDS, data);
-		CURL_SETOPT(c, CURLOPT_HTTPGET, 0);
+		CURL_SETOPT(c, CURLOPT_HTTPGET, 0L);
 		break;
 	case HTTP_REQUEST_PATCH:
 		CURL_SETOPT(c, CURLOPT_CUSTOMREQUEST, "PATCH");
-		CURL_SETOPT(c, CURLOPT_POST, 1);
+		CURL_SETOPT(c, CURLOPT_POST, 1L);
 		CURL_SETOPT(c, CURLOPT_POSTFIELDS, data);
-		CURL_SETOPT(c, CURLOPT_HTTPGET, 0);
+		CURL_SETOPT(c, CURLOPT_HTTPGET, 0L);
 		break;
 	case HTTP_REQUEST_GET:
 		CURL_SETOPT(c, CURLOPT_CUSTOMREQUEST, NULL);
-		CURL_SETOPT(c, CURLOPT_POST, 0);
+		CURL_SETOPT(c, CURLOPT_POST, 0L);
 		CURL_SETOPT(c, CURLOPT_POSTFIELDS, NULL);
-		CURL_SETOPT(c, CURLOPT_HTTPGET, 1);
+		CURL_SETOPT(c, CURLOPT_HTTPGET, 1L);
 		break;
 	case HTTP_REQUEST_DELETE:
 		CURL_SETOPT(c, CURLOPT_CUSTOMREQUEST, "DELETE");
-		CURL_SETOPT(c, CURLOPT_POST, 0);
+		CURL_SETOPT(c, CURLOPT_POST, 0L);
 		CURL_SETOPT(c, CURLOPT_POSTFIELDS, NULL);
-		CURL_SETOPT(c, CURLOPT_HTTPGET, 0);
+		CURL_SETOPT(c, CURLOPT_HTTPGET, 0L);
 		break;
 	default:
 		error("%s: Unable to process this request: %s", __func__,