background_add_entries hangs (specific to Snowflake?)

[jaclark5]

Describe the bug
I try to load the scaffold.json from QDS #490 and it fails

To Reproduce

1. Install QDS from #490 and navigate to submissions/2025-08-14-tmQM-xtb-Dataset-T=100K-Pd-Zn-Fe-Cu-low-mw-v0.0

2a.

from qcfractal.snowflake import FractalSnowflake
from qcportal.external import scaffold

snowflake = FractalSnowflake()
client = snowflake.client()

2b.

dataset = scaffold.from_json("scaffold.json.bz2", client)

Additional context
Output for (2b):

Creating new dataset.
Connection error for http://localhost:58222/api/v1/datasets/singlepoint/1/background_add_entries: HTTPConnectionPool(host='localhost', port=58222): Read timed out. (read timeout=60) - retrying in 0.48 seconds [1/5]
Connection error for http://localhost:58222/api/v1/datasets/singlepoint/1/background_add_entries: HTTPConnectionPool(host='localhost', port=58222): Read timed out. (read timeout=60) - retrying in 1.01 seconds [2/5]
Connection error for http://localhost:58222/api/v1/datasets/singlepoint/1/background_add_entries: HTTPConnectionPool(host='localhost', port=58222): Read timed out. (read timeout=60) - retrying in 2.09 seconds [3/5]
Connection error for http://localhost:58222/api/v1/datasets/singlepoint/1/background_add_entries: HTTPConnectionPool(host='localhost', port=58222): Read timed out. (read timeout=60) - retrying in 3.87 seconds [4/5]
Connection error for http://localhost:58222/api/v1/datasets/singlepoint/1/background_add_entries: HTTPConnectionPool(host='localhost', port=58222): Read timed out. (read timeout=60) - retrying in 7.62 seconds [5/5]
---------------------------------------------------------------------------
ConnectionRefusedError                    Traceback (most recent call last)
Cell In[2], line 1
----> 1 dataset = scaffold.from_json("scaffold.json.bz2", client)

File ~/bin/QCFractal/qcportal/qcportal/external/scaffold.py:111, in from_json(filename, client, append)
    108     else:
    109         entries.append(entry_type(**entry))
--> 111 ds.background_add_entries(entries)
    113 return ds

File ~/bin/QCFractal/qcportal/qcportal/singlepoint/dataset_models.py:98, in SinglepointDataset.background_add_entries(self, entries)
     95 def background_add_entries(
     96     self, entries: Union[SinglepointDatasetNewEntry, Iterable[SinglepointDatasetNewEntry]]
     97 ) -> InternalJob:
---> 98     return self._background_add_entries(entries)

File ~/bin/QCFractal/qcportal/qcportal/dataset_models.py:291, in BaseDataset._background_add_entries(self, entries)
    288 entries = make_list(entries)
    289 assert all(isinstance(x, self._new_entry_type) for x in entries), "Incorrect entry type"
--> 291 job_id = self._client.make_request("post", f"{self._base_url}/background_add_entries", int, body=entries)
    293 return self.get_internal_job(job_id)

File ~/bin/QCFractal/qcportal/qcportal/client_base.py:530, in PortalClientBase.make_request(self, method, endpoint, response_model, body_model, url_params_model, body, url_params, upload_files, allow_retries, additional_headers)
    526     file_data = None
    528 assert (serialized_body is None) or (file_data is None)  # Just to check my logic
--> 530 r = self._request(
    531     method,
    532     endpoint,
    533     body=serialized_body,
    534     url_params=parsed_url_params,
    535     file_data=file_data,
    536     allow_retries=allow_retries,
    537     additional_headers=additional_headers,
    538 )
    539 d = deserialize(r.content, r.headers["Content-Type"])
    541 if response_model is None:

File ~/bin/QCFractal/qcportal/qcportal/client_base.py:460, in PortalClientBase._request(self, method, endpoint, body, url_params, file_data, internal_retry, allow_retries, additional_headers)
    455     headers.update(additional_headers)
    457 req = requests.Request(
    458     method=method.upper(), url=full_uri, data=body, params=url_params, files=file_data, headers=headers
    459 )
--> 460 r = self._send_request(req, allow_retries=allow_retries)
    462 # If JWT token expired, automatically renew it and retry once. This should have been caught above,
    463 # but can happen in rare instances where the token expires between the time we check it and the time
    464 # we use it.
    465 if internal_retry and (r.status_code == 401) and "Token has expired" in r.json()["msg"]:

File ~/bin/QCFractal/qcportal/qcportal/client_base.py:353, in PortalClientBase._send_request(self, req, allow_retries)
    351     raise ConnectionRefusedError(_ssl_error_msg) from None
    352 except AllowedConnectionExceptions:
--> 353     raise ConnectionRefusedError(_connection_error_msg.format(self.address)) from None
    355 if self.debug_requests:
    356     pretty_print_response(ret)

ConnectionRefusedError: 

Could not connect to server http://localhost:58222/, please check the address and try again.

Output for (2a):

Heartbeat failed: HTTPConnectionPool(host='localhost', port=58222): Read timed out. (read timeout=60). QCFractal server down?
Missed 1 heartbeats so far
Exception on [/api/v1/datasets/singlepoint/1/background_add_entries](https://file+.vscode-resource.vscode-cdn.net/api/v1/datasets/singlepoint/1/background_add_entries) [POST]
Traceback (most recent call last):
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1963, in _exec_single_context
    self.dialect.do_execute(
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 943, in do_execute
    cursor.execute(statement, parameters)
psycopg2.OperationalError: server closed the connection unexpectedly
	This probably means the server terminated abnormally
	before or while processing the request.


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/flask/app.py", line 1511, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/flask/app.py", line 919, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/flask/app.py", line 917, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/flask/app.py", line 902, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/bin/QCFractal/qcfractal/qcfractal/flask_app/wrap_route.py", line 149, in wrapper
    ret = fn(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/bin/QCFractal/qcfractal/qcfractal/components/singlepoint/routes.py", line 82, in background_add_singlepoint_dataset_entries_v1
    return storage_socket.datasets.singlepoint.background_add_entries(dataset_id, new_entries=body_data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/bin/QCFractal/qcfractal/qcfractal/components/base_dataset_socket.py", line 806, in background_add_entries
    job_id = self.root_socket.internal_jobs.add(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/bin/QCFractal/qcfractal/qcfractal/components/internal_jobs/socket.py", line 169, in add
    session.flush()
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 4345, in flush
    self._flush(objects)
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 4480, in _flush
    with util.safe_reraise():
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/util/langhelpers.py", line 224, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 4441, in _flush
    flush_context.execute()
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/orm/unitofwork.py", line 466, in execute
    rec.execute(self)
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/orm/unitofwork.py", line 642, in execute
    util.preloaded.orm_persistence.save_obj(
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/orm/persistence.py", line 93, in save_obj
    _emit_insert_statements(
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/orm/persistence.py", line 1233, in _emit_insert_statements
    result = connection.execute(
             ^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1415, in execute
    return meth(
           ^^^^^
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/sql/elements.py", line 523, in _execute_on_connection
    return connection._execute_clauseelement(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1637, in _execute_clauseelement
    ret = self._execute_context(
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1842, in _execute_context
    return self._exec_single_context(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1982, in _exec_single_context
    self._handle_dbapi_exception(
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 2351, in _handle_dbapi_exception
    raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1963, in _exec_single_context
    self.dialect.do_execute(
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 943, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) server closed the connection unexpectedly
	This probably means the server terminated abnormally
	before or while processing the request.

[SQL: INSERT INTO internal_jobs (name, status, added_date, scheduled_date, started_date, last_updated, ended_date, runner_hostname, runner_uuid, progress, progress_description, function, kwargs, after_function, after_function_kwargs, repeat_delay, user_id, unique_name, serial_group) VALUES (%(name)s, %(status)s, %(added_date)s, %(scheduled_date)s, %(started_date)s, %(last_updated)s, %(ended_date)s, %(runner_hostname)s, %(runner_uuid)s, %(progress)s, %(progress_description)s, %(function)s, %(kwargs)s::JSON, %(after_function)s, %(after_function_kwargs)s::JSON, %(repeat_delay)s, %(user_id)s, %(unique_name)s, %(serial_group)s) RETURNING internal_jobs.id]
[parameters: {'name': 'dataset_add_entries_1', 'status': 'waiting', 'added_date': datetime.datetime(2025, 8, 22, 12, 39, 43, 774009, tzinfo=datetime.timezone.utc), 'scheduled_date': datetime.datetime(2025, 8, 22, 12, 39, 22, 240492, tzinfo=datetime.timezone.utc), 'started_date': None, 'last_updated': None, 'ended_date': None, 'runner_hostname': None, 'runner_uuid': None, 'progress': 0, 'progress_description': None, 'function': 'datasets.add_entry_dicts', 'kwargs': '{"dataset_id": 1, "entry_dicts": [{"name": "ABACAL_sm1_0", "molecule": {"schema_name": "qcschema_molecule", "schema_version": 2, "validated": true, " ... (1436563853 characters truncated) ... arse.from_schema"}, "id": 480335, "extras": {"molecular_weight": 421.3601200000001}}, "additional_keywords": {}, "attributes": {}, "comment": null}]}', 'after_function': None, 'after_function_kwargs': None, 'repeat_delay': None, 'user_id': None, 'unique_name': None, 'serial_group': 'ds_add_entries_1'}]
(Background on this error at: https://sqlalche.me/e/20/e3q8)
Manager snowflake_compute-Jennifers-MacBook-Air.local-bcaf245d-23f9-43c8-bfee-aed2b241abef exists but is not active! Will not give it tasks
Exception in thread Thread-5 (start):
Traceback (most recent call last):
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
    self.run()
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/ipykernel/ipkernel.py", line 766, in run_closure
    _threading_Thread_run(self)
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/threading.py", line 982, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/jenniferclark/bin/QCFractal/qcfractalcompute/qcfractalcompute/compute_manager.py", line 313, in start
    self.scheduler.run(blocking=True)
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/sched.py", line 151, in run
    action(*argument, **kwargs)
  File "/Users/jenniferclark/bin/QCFractal/qcfractalcompute/qcfractalcompute/compute_manager.py", line 290, in scheduler_update
    self.update(new_tasks=True)
  File "/Users/jenniferclark/bin/QCFractal/qcfractalcompute/qcfractalcompute/compute_manager.py", line 675, in update
    self._update(new_tasks=new_tasks)
  File "/Users/jenniferclark/bin/QCFractal/qcfractalcompute/qcfractalcompute/compute_manager.py", line 654, in _update
    new_task_info = self.client.claim(executor_programs, executor_config.compute_tags, open_slots)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/bin/QCFractal/qcportal/qcportal/manager_client.py", line 132, in claim
    return self.make_request("post", "compute/v1/tasks/claim", List[RecordTask], body=body, allow_retries=False)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/bin/QCFractal/qcportal/qcportal/client_base.py", line 530, in make_request
    r = self._request(
        ^^^^^^^^^^^^^^
  File "/Users/jenniferclark/bin/QCFractal/qcportal/qcportal/client_base.py", line 479, in _request
    raise PortalRequestError(f"Request failed: {details['msg']}", r.status_code, details)
qcportal.client_base.PortalRequestError: Request failed: Manager is not active! (HTTP status 400)
Exception on [/api/v1/datasets/singlepoint/1/background_add_entries](https://file+.vscode-resource.vscode-cdn.net/api/v1/datasets/singlepoint/1/background_add_entries) [POST]
Traceback (most recent call last):
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1963, in _exec_single_context
    self.dialect.do_execute(
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 943, in do_execute
    cursor.execute(statement, parameters)
psycopg2.OperationalError: server closed the connection unexpectedly
	This probably means the server terminated abnormally
	before or while processing the request.


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/flask/app.py", line 1511, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/flask/app.py", line 919, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/flask/app.py", line 917, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/flask/app.py", line 902, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/bin/QCFractal/qcfractal/qcfractal/flask_app/wrap_route.py", line 149, in wrapper
    ret = fn(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/bin/QCFractal/qcfractal/qcfractal/components/singlepoint/routes.py", line 82, in background_add_singlepoint_dataset_entries_v1
    return storage_socket.datasets.singlepoint.background_add_entries(dataset_id, new_entries=body_data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/bin/QCFractal/qcfractal/qcfractal/components/base_dataset_socket.py", line 806, in background_add_entries
    job_id = self.root_socket.internal_jobs.add(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/bin/QCFractal/qcfractal/qcfractal/components/internal_jobs/socket.py", line 169, in add
    session.flush()
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 4345, in flush
    self._flush(objects)
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 4480, in _flush
    with util.safe_reraise():
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/util/langhelpers.py", line 224, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 4441, in _flush
    flush_context.execute()
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/orm/unitofwork.py", line 466, in execute
    rec.execute(self)
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/orm/unitofwork.py", line 642, in execute
    util.preloaded.orm_persistence.save_obj(
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/orm/persistence.py", line 93, in save_obj
    _emit_insert_statements(
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/orm/persistence.py", line 1233, in _emit_insert_statements
    result = connection.execute(
             ^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1415, in execute
    return meth(
           ^^^^^
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/sql/elements.py", line 523, in _execute_on_connection
    return connection._execute_clauseelement(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1637, in _execute_clauseelement
    ret = self._execute_context(
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1842, in _execute_context
    return self._exec_single_context(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1982, in _exec_single_context
    self._handle_dbapi_exception(
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 2351, in _handle_dbapi_exception
    raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1963, in _exec_single_context
    self.dialect.do_execute(
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 943, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) server closed the connection unexpectedly
	This probably means the server terminated abnormally
	before or while processing the request.

[SQL: INSERT INTO internal_jobs (name, status, added_date, scheduled_date, started_date, last_updated, ended_date, runner_hostname, runner_uuid, progress, progress_description, function, kwargs, after_function, after_function_kwargs, repeat_delay, user_id, unique_name, serial_group) VALUES (%(name)s, %(status)s, %(added_date)s, %(scheduled_date)s, %(started_date)s, %(last_updated)s, %(ended_date)s, %(runner_hostname)s, %(runner_uuid)s, %(progress)s, %(progress_description)s, %(function)s, %(kwargs)s::JSON, %(after_function)s, %(after_function_kwargs)s::JSON, %(repeat_delay)s, %(user_id)s, %(unique_name)s, %(serial_group)s) RETURNING internal_jobs.id]
[parameters: {'name': 'dataset_add_entries_1', 'status': 'waiting', 'added_date': datetime.datetime(2025, 8, 22, 12, 41, 30, 747472, tzinfo=datetime.timezone.utc), 'scheduled_date': datetime.datetime(2025, 8, 22, 12, 40, 57, 11346, tzinfo=datetime.timezone.utc), 'started_date': None, 'last_updated': None, 'ended_date': None, 'runner_hostname': None, 'runner_uuid': None, 'progress': 0, 'progress_description': None, 'function': 'datasets.add_entry_dicts', 'kwargs': '{"dataset_id": 1, "entry_dicts": [{"name": "ABACAL_sm1_0", "molecule": {"schema_name": "qcschema_molecule", "schema_version": 2, "validated": true, " ... (1436563853 characters truncated) ... arse.from_schema"}, "id": 480335, "extras": {"molecular_weight": 421.3601200000001}}, "additional_keywords": {}, "attributes": {}, "comment": null}]}', 'after_function': None, 'after_function_kwargs': None, 'repeat_delay': None, 'user_id': None, 'unique_name': None, 'serial_group': 'ds_add_entries_1'}]
(Background on this error at: https://sqlalche.me/e/20/e3q8)
Exception on [/api/v1/datasets/singlepoint/1/background_add_entries](https://file+.vscode-resource.vscode-cdn.net/api/v1/datasets/singlepoint/1/background_add_entries) [POST]
Traceback (most recent call last):
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1963, in _exec_single_context
    self.dialect.do_execute(
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 943, in do_execute
    cursor.execute(statement, parameters)
psycopg2.OperationalError: server closed the connection unexpectedly
	This probably means the server terminated abnormally
	before or while processing the request.


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/flask/app.py", line 1511, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/flask/app.py", line 919, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/flask/app.py", line 917, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/flask/app.py", line 902, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/bin/QCFractal/qcfractal/qcfractal/flask_app/wrap_route.py", line 149, in wrapper
    ret = fn(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/bin/QCFractal/qcfractal/qcfractal/components/singlepoint/routes.py", line 82, in background_add_singlepoint_dataset_entries_v1
    return storage_socket.datasets.singlepoint.background_add_entries(dataset_id, new_entries=body_data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/bin/QCFractal/qcfractal/qcfractal/components/base_dataset_socket.py", line 806, in background_add_entries
    job_id = self.root_socket.internal_jobs.add(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/bin/QCFractal/qcfractal/qcfractal/components/internal_jobs/socket.py", line 169, in add
    session.flush()
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 4345, in flush
    self._flush(objects)
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 4480, in _flush
    with util.safe_reraise():
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/util/langhelpers.py", line 224, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 4441, in _flush
    flush_context.execute()
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/orm/unitofwork.py", line 466, in execute
    rec.execute(self)
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/orm/unitofwork.py", line 642, in execute
    util.preloaded.orm_persistence.save_obj(
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/orm/persistence.py", line 93, in save_obj
    _emit_insert_statements(
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/orm/persistence.py", line 1233, in _emit_insert_statements
    result = connection.execute(
             ^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1415, in execute
    return meth(
           ^^^^^
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/sql/elements.py", line 523, in _execute_on_connection
    return connection._execute_clauseelement(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1637, in _execute_clauseelement
    ret = self._execute_context(
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1842, in _execute_context
    return self._exec_single_context(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1982, in _exec_single_context
    self._handle_dbapi_exception(
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 2351, in _handle_dbapi_exception
    raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1963, in _exec_single_context
    self.dialect.do_execute(
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 943, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) server closed the connection unexpectedly
	This probably means the server terminated abnormally
	before or while processing the request.

[SQL: INSERT INTO internal_jobs (name, status, added_date, scheduled_date, started_date, last_updated, ended_date, runner_hostname, runner_uuid, progress, progress_description, function, kwargs, after_function, after_function_kwargs, repeat_delay, user_id, unique_name, serial_group) VALUES (%(name)s, %(status)s, %(added_date)s, %(scheduled_date)s, %(started_date)s, %(last_updated)s, %(ended_date)s, %(runner_hostname)s, %(runner_uuid)s, %(progress)s, %(progress_description)s, %(function)s, %(kwargs)s::JSON, %(after_function)s, %(after_function_kwargs)s::JSON, %(repeat_delay)s, %(user_id)s, %(unique_name)s, %(serial_group)s) RETURNING internal_jobs.id]
[parameters: {'name': 'dataset_add_entries_1', 'status': 'waiting', 'added_date': datetime.datetime(2025, 8, 22, 12, 43, 16, 60591, tzinfo=datetime.timezone.utc), 'scheduled_date': datetime.datetime(2025, 8, 22, 12, 42, 54, 458191, tzinfo=datetime.timezone.utc), 'started_date': None, 'last_updated': None, 'ended_date': None, 'runner_hostname': None, 'runner_uuid': None, 'progress': 0, 'progress_description': None, 'function': 'datasets.add_entry_dicts', 'kwargs': '{"dataset_id": 1, "entry_dicts": [{"name": "ABACAL_sm1_0", "molecule": {"schema_name": "qcschema_molecule", "schema_version": 2, "validated": true, " ... (1436563853 characters truncated) ... arse.from_schema"}, "id": 480335, "extras": {"molecular_weight": 421.3601200000001}}, "additional_keywords": {}, "attributes": {}, "comment": null}]}', 'after_function': None, 'after_function_kwargs': None, 'repeat_delay': None, 'user_id': None, 'unique_name': None, 'serial_group': 'ds_add_entries_1'}]
(Background on this error at: https://sqlalche.me/e/20/e3q8)
Exception on [/api/v1/datasets/singlepoint/1/background_add_entries](https://file+.vscode-resource.vscode-cdn.net/api/v1/datasets/singlepoint/1/background_add_entries) [POST]
Traceback (most recent call last):
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1963, in _exec_single_context
    self.dialect.do_execute(
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 943, in do_execute
    cursor.execute(statement, parameters)
psycopg2.OperationalError: server closed the connection unexpectedly
	This probably means the server terminated abnormally
	before or while processing the request.


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/flask/app.py", line 1511, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/flask/app.py", line 919, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/flask/app.py", line 917, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/flask/app.py", line 902, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/bin/QCFractal/qcfractal/qcfractal/flask_app/wrap_route.py", line 149, in wrapper
    ret = fn(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/bin/QCFractal/qcfractal/qcfractal/components/singlepoint/routes.py", line 82, in background_add_singlepoint_dataset_entries_v1
    return storage_socket.datasets.singlepoint.background_add_entries(dataset_id, new_entries=body_data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/bin/QCFractal/qcfractal/qcfractal/components/base_dataset_socket.py", line 806, in background_add_entries
    job_id = self.root_socket.internal_jobs.add(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/bin/QCFractal/qcfractal/qcfractal/components/internal_jobs/socket.py", line 169, in add
    session.flush()
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 4345, in flush
    self._flush(objects)
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 4480, in _flush
    with util.safe_reraise():
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/util/langhelpers.py", line 224, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 4441, in _flush
    flush_context.execute()
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/orm/unitofwork.py", line 466, in execute
    rec.execute(self)
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/orm/unitofwork.py", line 642, in execute
    util.preloaded.orm_persistence.save_obj(
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/orm/persistence.py", line 93, in save_obj
    _emit_insert_statements(
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/orm/persistence.py", line 1233, in _emit_insert_statements
    result = connection.execute(
             ^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1415, in execute
    return meth(
           ^^^^^
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/sql/elements.py", line 523, in _execute_on_connection
    return connection._execute_clauseelement(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1637, in _execute_clauseelement
    ret = self._execute_context(
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1842, in _execute_context
    return self._exec_single_context(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1982, in _exec_single_context
    self._handle_dbapi_exception(
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 2351, in _handle_dbapi_exception
    raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1963, in _exec_single_context
    self.dialect.do_execute(
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 943, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) server closed the connection unexpectedly
	This probably means the server terminated abnormally
	before or while processing the request.

[SQL: INSERT INTO internal_jobs (name, status, added_date, scheduled_date, started_date, last_updated, ended_date, runner_hostname, runner_uuid, progress, progress_description, function, kwargs, after_function, after_function_kwargs, repeat_delay, user_id, unique_name, serial_group) VALUES (%(name)s, %(status)s, %(added_date)s, %(scheduled_date)s, %(started_date)s, %(last_updated)s, %(ended_date)s, %(runner_hostname)s, %(runner_uuid)s, %(progress)s, %(progress_description)s, %(function)s, %(kwargs)s::JSON, %(after_function)s, %(after_function_kwargs)s::JSON, %(repeat_delay)s, %(user_id)s, %(unique_name)s, %(serial_group)s) RETURNING internal_jobs.id]
[parameters: {'name': 'dataset_add_entries_1', 'status': 'waiting', 'added_date': datetime.datetime(2025, 8, 22, 12, 45, 13, 905254, tzinfo=datetime.timezone.utc), 'scheduled_date': datetime.datetime(2025, 8, 22, 12, 44, 41, 520367, tzinfo=datetime.timezone.utc), 'started_date': None, 'last_updated': None, 'ended_date': None, 'runner_hostname': None, 'runner_uuid': None, 'progress': 0, 'progress_description': None, 'function': 'datasets.add_entry_dicts', 'kwargs': '{"dataset_id": 1, "entry_dicts": [{"name": "ABACAL_sm1_0", "molecule": {"schema_name": "qcschema_molecule", "schema_version": 2, "validated": true, " ... (1436563853 characters truncated) ... arse.from_schema"}, "id": 480335, "extras": {"molecular_weight": 421.3601200000001}}, "additional_keywords": {}, "attributes": {}, "comment": null}]}', 'after_function': None, 'after_function_kwargs': None, 'repeat_delay': None, 'user_id': None, 'unique_name': None, 'serial_group': 'ds_add_entries_1'}]
(Background on this error at: https://sqlalche.me/e/20/e3q8)
Exception on [/api/v1/datasets/singlepoint/1/background_add_entries](https://file+.vscode-resource.vscode-cdn.net/api/v1/datasets/singlepoint/1/background_add_entries) [POST]
Traceback (most recent call last):
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1963, in _exec_single_context
    self.dialect.do_execute(
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 943, in do_execute
    cursor.execute(statement, parameters)
psycopg2.OperationalError: server closed the connection unexpectedly
	This probably means the server terminated abnormally
	before or while processing the request.


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/flask/app.py", line 1511, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/flask/app.py", line 919, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/flask/app.py", line 917, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/flask/app.py", line 902, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/bin/QCFractal/qcfractal/qcfractal/flask_app/wrap_route.py", line 149, in wrapper
    ret = fn(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/bin/QCFractal/qcfractal/qcfractal/components/singlepoint/routes.py", line 82, in background_add_singlepoint_dataset_entries_v1
    return storage_socket.datasets.singlepoint.background_add_entries(dataset_id, new_entries=body_data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/bin/QCFractal/qcfractal/qcfractal/components/base_dataset_socket.py", line 806, in background_add_entries
    job_id = self.root_socket.internal_jobs.add(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/bin/QCFractal/qcfractal/qcfractal/components/internal_jobs/socket.py", line 169, in add
    session.flush()
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 4345, in flush
    self._flush(objects)
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 4480, in _flush
    with util.safe_reraise():
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/util/langhelpers.py", line 224, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 4441, in _flush
    flush_context.execute()
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/orm/unitofwork.py", line 466, in execute
    rec.execute(self)
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/orm/unitofwork.py", line 642, in execute
    util.preloaded.orm_persistence.save_obj(
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/orm/persistence.py", line 93, in save_obj
    _emit_insert_statements(
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/orm/persistence.py", line 1233, in _emit_insert_statements
    result = connection.execute(
             ^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1415, in execute
    return meth(
           ^^^^^
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/sql/elements.py", line 523, in _execute_on_connection
    return connection._execute_clauseelement(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1637, in _execute_clauseelement
    ret = self._execute_context(
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1842, in _execute_context
    return self._exec_single_context(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1982, in _exec_single_context
    self._handle_dbapi_exception(
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 2351, in _handle_dbapi_exception
    raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1963, in _exec_single_context
    self.dialect.do_execute(
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 943, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) server closed the connection unexpectedly
	This probably means the server terminated abnormally
	before or while processing the request.

[SQL: INSERT INTO internal_jobs (name, status, added_date, scheduled_date, started_date, last_updated, ended_date, runner_hostname, runner_uuid, progress, progress_description, function, kwargs, after_function, after_function_kwargs, repeat_delay, user_id, unique_name, serial_group) VALUES (%(name)s, %(status)s, %(added_date)s, %(scheduled_date)s, %(started_date)s, %(last_updated)s, %(ended_date)s, %(runner_hostname)s, %(runner_uuid)s, %(progress)s, %(progress_description)s, %(function)s, %(kwargs)s::JSON, %(after_function)s, %(after_function_kwargs)s::JSON, %(repeat_delay)s, %(user_id)s, %(unique_name)s, %(serial_group)s) RETURNING internal_jobs.id]
[parameters: {'name': 'dataset_add_entries_1', 'status': 'waiting', 'added_date': datetime.datetime(2025, 8, 22, 12, 47, 7, 310565, tzinfo=datetime.timezone.utc), 'scheduled_date': datetime.datetime(2025, 8, 22, 12, 46, 33, 731979, tzinfo=datetime.timezone.utc), 'started_date': None, 'last_updated': None, 'ended_date': None, 'runner_hostname': None, 'runner_uuid': None, 'progress': 0, 'progress_description': None, 'function': 'datasets.add_entry_dicts', 'kwargs': '{"dataset_id": 1, "entry_dicts": [{"name": "ABACAL_sm1_0", "molecule": {"schema_name": "qcschema_molecule", "schema_version": 2, "validated": true, " ... (1436563853 characters truncated) ... arse.from_schema"}, "id": 480335, "extras": {"molecular_weight": 421.3601200000001}}, "additional_keywords": {}, "attributes": {}, "comment": null}]}', 'after_function': None, 'after_function_kwargs': None, 'repeat_delay': None, 'user_id': None, 'unique_name': None, 'serial_group': 'ds_add_entries_1'}]
(Background on this error at: https://sqlalche.me/e/20/e3q8)
Exception on [/api/v1/datasets/singlepoint/1/background_add_entries](https://file+.vscode-resource.vscode-cdn.net/api/v1/datasets/singlepoint/1/background_add_entries) [POST]
Traceback (most recent call last):
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1963, in _exec_single_context
    self.dialect.do_execute(
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 943, in do_execute
    cursor.execute(statement, parameters)
psycopg2.OperationalError: server closed the connection unexpectedly
	This probably means the server terminated abnormally
	before or while processing the request.


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/flask/app.py", line 1511, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/flask/app.py", line 919, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/flask/app.py", line 917, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/flask/app.py", line 902, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/bin/QCFractal/qcfractal/qcfractal/flask_app/wrap_route.py", line 149, in wrapper
    ret = fn(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/bin/QCFractal/qcfractal/qcfractal/components/singlepoint/routes.py", line 82, in background_add_singlepoint_dataset_entries_v1
    return storage_socket.datasets.singlepoint.background_add_entries(dataset_id, new_entries=body_data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/bin/QCFractal/qcfractal/qcfractal/components/base_dataset_socket.py", line 806, in background_add_entries
    job_id = self.root_socket.internal_jobs.add(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/bin/QCFractal/qcfractal/qcfractal/components/internal_jobs/socket.py", line 169, in add
    session.flush()
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 4345, in flush
    self._flush(objects)
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 4480, in _flush
    with util.safe_reraise():
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/util/langhelpers.py", line 224, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 4441, in _flush
    flush_context.execute()
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/orm/unitofwork.py", line 466, in execute
    rec.execute(self)
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/orm/unitofwork.py", line 642, in execute
    util.preloaded.orm_persistence.save_obj(
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/orm/persistence.py", line 93, in save_obj
    _emit_insert_statements(
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/orm/persistence.py", line 1233, in _emit_insert_statements
    result = connection.execute(
             ^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1415, in execute
    return meth(
           ^^^^^
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/sql/elements.py", line 523, in _execute_on_connection
    return connection._execute_clauseelement(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1637, in _execute_clauseelement
    ret = self._execute_context(
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1842, in _execute_context
    return self._exec_single_context(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1982, in _exec_single_context
    self._handle_dbapi_exception(
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 2351, in _handle_dbapi_exception
    raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1963, in _exec_single_context
    self.dialect.do_execute(
  File "/Users/jenniferclark/mamba/envs/qca/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 943, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) server closed the connection unexpectedly
	This probably means the server terminated abnormally
	before or while processing the request.

[SQL: INSERT INTO internal_jobs (name, status, added_date, scheduled_date, started_date, last_updated, ended_date, runner_hostname, runner_uuid, progress, progress_description, function, kwargs, after_function, after_function_kwargs, repeat_delay, user_id, unique_name, serial_group) VALUES (%(name)s, %(status)s, %(added_date)s, %(scheduled_date)s, %(started_date)s, %(last_updated)s, %(ended_date)s, %(runner_hostname)s, %(runner_uuid)s, %(progress)s, %(progress_description)s, %(function)s, %(kwargs)s::JSON, %(after_function)s, %(after_function_kwargs)s::JSON, %(repeat_delay)s, %(user_id)s, %(unique_name)s, %(serial_group)s) RETURNING internal_jobs.id]
[parameters: {'name': 'dataset_add_entries_1', 'status': 'waiting', 'added_date': datetime.datetime(2025, 8, 22, 12, 48, 58, 859750, tzinfo=datetime.timezone.utc), 'scheduled_date': datetime.datetime(2025, 8, 22, 12, 48, 25, 137839, tzinfo=datetime.timezone.utc), 'started_date': None, 'last_updated': None, 'ended_date': None, 'runner_hostname': None, 'runner_uuid': None, 'progress': 0, 'progress_description': None, 'function': 'datasets.add_entry_dicts', 'kwargs': '{"dataset_id": 1, "entry_dicts": [{"name": "ABACAL_sm1_0", "molecule": {"schema_name": "qcschema_molecule", "schema_version": 2, "validated": true, " ... (1436563853 characters truncated) ... arse.from_schema"}, "id": 480335, "extras": {"molecular_weight": 421.3601200000001}}, "additional_keywords": {}, "attributes": {}, "comment": null}]}', 'after_function': None, 'after_function_kwargs': None, 'repeat_delay': None, 'user_id': None, 'unique_name': None, 'serial_group': 'ds_add_entries_1'}]
(Background on this error at: https://sqlalche.me/e/20/e3q8)

[jaclark5]

It just occurred to me that our QDS validation ran fine so maybe a MAC/Jupyter thing?

[jaclark5]

Had this issue again using the regular client (not Snowflake) after 10 min I got the error:

---------------------------------------------------------------------------
PortalRequestError                        Traceback (most recent call last)
Cell In[13], [line 3](vscode-notebook-cell:?execution_count=13&line=3)
      1 ds = client.get_dataset("singlepoint", "tmQM xtb Dataset T=100K low-mw high-coordinate mult=1 v0.0")
      2 client.delete_dataset(ds.id, False)
----> [3](vscode-notebook-cell:?execution_count=13&line=3) ds = scaffold.from_json("scaffold.json.bz2", client)

File ~/bin/QCFractal/qcportal/qcportal/external/scaffold.py:111, in from_json(filename, client, append)
    108     else:
    109         entries.append(entry_type(**entry))
--> [111](https://file+.vscode-resource.vscode-cdn.net/Users/jenniferclark/bin/qca-dataset-submission/submissions/2025-08-14-tmQM-xtb-Dataset-T%3D100K-low-mw-high-coordinate-mult%3D1-v0.0/~/bin/QCFractal/qcportal/qcportal/external/scaffold.py:111) ds.background_add_entries(entries)
    113 return ds

File ~/bin/QCFractal/qcportal/qcportal/singlepoint/dataset_models.py:98, in SinglepointDataset.background_add_entries(self, entries)
     95 def background_add_entries(
     96     self, entries: Union[SinglepointDatasetNewEntry, Iterable[SinglepointDatasetNewEntry]]
     97 ) -> InternalJob:
---> [98](https://file+.vscode-resource.vscode-cdn.net/Users/jenniferclark/bin/qca-dataset-submission/submissions/2025-08-14-tmQM-xtb-Dataset-T%3D100K-low-mw-high-coordinate-mult%3D1-v0.0/~/bin/QCFractal/qcportal/qcportal/singlepoint/dataset_models.py:98)     return self._background_add_entries(entries)

File ~/bin/QCFractal/qcportal/qcportal/dataset_models.py:291, in BaseDataset._background_add_entries(self, entries)
    288 entries = make_list(entries)
    289 assert all(isinstance(x, self._new_entry_type) for x in entries), "Incorrect entry type"
--> [291](https://file+.vscode-resource.vscode-cdn.net/Users/jenniferclark/bin/qca-dataset-submission/submissions/2025-08-14-tmQM-xtb-Dataset-T%3D100K-low-mw-high-coordinate-mult%3D1-v0.0/~/bin/QCFractal/qcportal/qcportal/dataset_models.py:291) job_id = self._client.make_request("post", f"{self._base_url}/background_add_entries", int, body=entries)
    293 return self.get_internal_job(job_id)
...
    477         details = {"msg": r.reason}
--> [479](https://file+.vscode-resource.vscode-cdn.net/Users/jenniferclark/bin/qca-dataset-submission/submissions/2025-08-14-tmQM-xtb-Dataset-T%3D100K-low-mw-high-coordinate-mult%3D1-v0.0/~/bin/QCFractal/qcportal/qcportal/client_base.py:479)     raise PortalRequestError(f"Request failed: {details['msg']}", r.status_code, details)
    481 return r

PortalRequestError: Request failed: Authentication failure - JWT Token has expired: Signature has expired (HTTP status 401)
Output is truncated. View as a [scrollable element](command:cellOutput.enableScrolling?8e54811d-42d0-4c9d-a75e-15f1c13ce14f) or open in a [text editor](command:workbench.action.openLargeOutput?8e54811d-42d0-4c9d-a75e-15f1c13ce14f). Adjust cell output [settings](command:workbench.action.openSettings?%5B%22%40tag%3AnotebookOutputLayout%22%5D)...

[jaclark5]

@bennybp Ran again in the command line and had the same issue after 9 min:

>>> import qcportal, os
>>> from qcportal.external import scaffold
>>> client = qcportal.PortalClient(    "https://api.qcarchive.molssi.org:443", username=os.environ['QCARCHIVE_USER'], password=os.environ['QCARCHIVE_PASSWORD'], cache_dir=".",)
>>> ds = client.get_dataset("singlepoint", "tmQM xtb Dataset T=100K low-mw high-coordinate mult=1 v0.0")
>>> client.delete_dataset(ds.id, False)
>>> ds = scaffold.from_json("scaffold.json.bz2", client)
Creating new dataset.
Connection error for https://api.qcarchive.molssi.org:443/api/v1/datasets/singlepoint/459/background_add_entries: HTTPSConnectionPool(host='api.qcarchive.molssi.org', port=443): Read timed out. (read timeout=60) - retrying in 0.51 seconds [1/5]
Connection error for https://api.qcarchive.molssi.org:443/api/v1/datasets/singlepoint/459/background_add_entries: ('Connection aborted.', TimeoutError('The write operation timed out')) - retrying in 1.03 seconds [2/5]
Connection error for https://api.qcarchive.molssi.org:443/api/v1/datasets/singlepoint/459/background_add_entries: HTTPSConnectionPool(host='api.qcarchive.molssi.org', port=443): Read timed out. (read timeout=60) - retrying in 0.52 seconds [1/5]
Connection error for https://api.qcarchive.molssi.org:443/api/v1/datasets/singlepoint/459/background_add_entries: ('Connection aborted.', TimeoutError('The write operation timed out')) - retrying in 1.04 seconds [2/5]
Connection error for https://api.qcarchive.molssi.org:443/api/v1/datasets/singlepoint/459/background_add_entries: HTTPSConnectionPool(host='api.qcarchive.molssi.org', port=443): Read timed out. (read timeout=60) - retrying in 2.04 seconds [3/5]
Connection error for https://api.qcarchive.molssi.org:443/api/v1/datasets/singlepoint/459/background_add_entries: ('Connection aborted.', TimeoutError('The write operation timed out')) - retrying in 4.05 seconds [4/5]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/jenniferclark/bin/QCFractal/qcportal/qcportal/external/scaffold.py", line 111, in from_json
    ds.background_add_entries(entries)
  File "/Users/jenniferclark/bin/QCFractal/qcportal/qcportal/singlepoint/dataset_models.py", line 98, in background_add_entries
    return self._background_add_entries(entries)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/bin/QCFractal/qcportal/qcportal/dataset_models.py", line 291, in _background_add_entries
    job_id = self._client.make_request("post", f"{self._base_url}/background_add_entries", int, body=entries)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/bin/QCFractal/qcportal/qcportal/client_base.py", line 530, in make_request
    r = self._request(
        ^^^^^^^^^^^^^^
  File "/Users/jenniferclark/bin/QCFractal/qcportal/qcportal/client_base.py", line 467, in _request
    return self._request(method, endpoint, body=body, url_params=url_params, internal_retry=False)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jenniferclark/bin/QCFractal/qcportal/qcportal/client_base.py", line 479, in _request
    raise PortalRequestError(f"Request failed: {details['msg']}", r.status_code, details)
qcportal.client_base.PortalRequestError: Request failed: Authentication failure - JWT Token has expired: Signature has expired (HTTP status 401)

From env (using latest on github):

    - qcarchivetesting==0.61.post39+g0efd61073
    - qcfractal==0.61.post39+g0efd61073
    - qcfractalcompute==0.61.post39+g0efd61073
    - qcportal==0.61.post39+g0efd61073