Skip to content

Commit 23afba2

Browse files
ueshinHyukjinKwon
authored andcommitted
[SPARK-55282][PYTHON][CONNECT] Avoid using worker_util in the Driver-side
### What changes were proposed in this pull request? Avoid using `worker_util` in the Driver-side. ### Why are the changes needed? The `worker_util` module is supposed to be used in the worker processes. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? The existing tests should pass. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #54065 from ueshin/issues/SPARK-55282/avoid_worker_util. Authored-by: Takuya Ueshin <ueshin@databricks.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
1 parent a157725 commit 23afba2

File tree

1 file changed

+1
-3
lines changed
  • python/pyspark/sql/connect/client

1 file changed

+1
-3
lines changed

python/pyspark/sql/connect/client/core.py

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -66,7 +66,7 @@
6666
from google.rpc import error_details_pb2
6767

6868
from pyspark.util import is_remote_only, disable_gc
69-
from pyspark.accumulators import SpecialAccumulatorIds
69+
from pyspark.accumulators import SpecialAccumulatorIds, pickleSer
7070
from pyspark.version import __version__
7171
from pyspark.traceback_utils import CallSite
7272
from pyspark.resource.information import ResourceInformation
@@ -1550,8 +1550,6 @@ def handle_response(
15501550
logger.debug("Received observed metric batch.")
15511551
for observed_metrics in self._build_observed_metrics(b.observed_metrics):
15521552
if observed_metrics.name == "__python_accumulator__":
1553-
from pyspark.worker_util import pickleSer
1554-
15551553
for metric in observed_metrics.metrics:
15561554
(aid, update) = pickleSer.loads(LiteralExpression._to_value(metric))
15571555
if aid == SpecialAccumulatorIds.SQL_UDF_PROFIER:

0 commit comments

Comments
 (0)