After upgrading to the 2.0.0 image, we saw the following crash in our superset-celery pods:
[2022-09-27 16:50:14,091: CRITICAL/MainProcess] Unrecoverable error: TypeError("__init__() got an unexpected keyword argument 'username'")
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/kombu/transport/virtual/base.py", line 925, in create_channel
return self._avail_channels.pop()
IndexError: pop from empty list
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/redis/connection.py", line 988, in get_connection
connection = self._available_connections.pop()
IndexError: pop from empty list
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/celery/worker/worker.py", line 203, in start
self.blueprint.start(self)
File "/usr/local/lib/python3.8/site-packages/celery/bootsteps.py", line 116, in start
step.start(parent)
File "/usr/local/lib/python3.8/site-packages/celery/bootsteps.py", line 365, in start
return self.obj.start()
File "/usr/local/lib/python3.8/site-packages/celery/worker/consumer/consumer.py", line 332, in start
blueprint.start(self)
File "/usr/local/lib/python3.8/site-packages/celery/bootsteps.py", line 116, in start
step.start(parent)
File "/usr/local/lib/python3.8/site-packages/celery/worker/consumer/connection.py", line 21, in start
c.connection = c.connect()
File "/usr/local/lib/python3.8/site-packages/celery/worker/consumer/consumer.py", line 428, in connect
conn = self.connection_for_read(heartbeat=self.amqheartbeat)
File "/usr/local/lib/python3.8/site-packages/celery/worker/consumer/consumer.py", line 434, in connection_for_read
return self.ensure_connected(
File "/usr/local/lib/python3.8/site-packages/celery/worker/consumer/consumer.py", line 460, in ensure_connected
conn = conn.ensure_connection(
File "/usr/local/lib/python3.8/site-packages/kombu/connection.py", line 381, in ensure_connection
self._ensure_connection(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/kombu/connection.py", line 433, in _ensure_connection
return retry_over_time(
File "/usr/local/lib/python3.8/site-packages/kombu/utils/functional.py", line 312, in retry_over_time
return fun(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/kombu/connection.py", line 877, in _connection_factory
self._connection = self._establish_connection()
File "/usr/local/lib/python3.8/site-packages/kombu/connection.py", line 812, in _establish_connection
conn = self.transport.establish_connection()
File "/usr/local/lib/python3.8/site-packages/kombu/transport/virtual/base.py", line 949, in establish_connection
self._avail_channels.append(self.create_channel(self))
File "/usr/local/lib/python3.8/site-packages/kombu/transport/virtual/base.py", line 927, in create_channel
channel = self.Channel(connection)
File "/usr/local/lib/python3.8/site-packages/kombu/transport/redis.py", line 737, in __init__
self.client.ping()
File "/usr/local/lib/python3.8/site-packages/redis/client.py", line 1037, in ping
return self.execute_command('PING')
File "/usr/local/lib/python3.8/site-packages/redis/client.py", line 772, in execute_command
connection = pool.get_connection(command_name, **options)
File "/usr/local/lib/python3.8/site-packages/redis/connection.py", line 990, in get_connection
connection = self.make_connection()
File "/usr/local/lib/python3.8/site-packages/redis/connection.py", line 1025, in make_connection
return self.connection_class(**self.connection_kwargs)
TypeError: __init__() got an unexpected keyword argument 'username'
After some further investigation, we traced it back to the Redis client needing to accept the username parameter. The support for that was added in this commit which became version 3.4.0. The current docker image is using an older version of the Redis client
After upgrading to the 2.0.0 image, we saw the following crash in our superset-celery pods:
After some further investigation, we traced it back to the Redis client needing to accept the username parameter. The support for that was added in this commit which became version 3.4.0. The current docker image is using an older version of the Redis client