Skip to content

[SPARK-51755][INFRA][PYTHON] Set up a scheduled builder for free-threaded Python 3.13#50534

Closed
zhengruifeng wants to merge 6 commits intoapache:masterfrom
zhengruifeng:infra_py313_nogil
Closed

[SPARK-51755][INFRA][PYTHON] Set up a scheduled builder for free-threaded Python 3.13#50534
zhengruifeng wants to merge 6 commits intoapache:masterfrom
zhengruifeng:infra_py313_nogil

Conversation

@zhengruifeng
Copy link
Copy Markdown
Contributor

@zhengruifeng zhengruifeng commented Apr 8, 2025

What changes were proposed in this pull request?

Set up a scheduled (every 3 days) builder for free-threaded Python 3.13

Why are the changes needed?

https://docs.python.org/3/howto/free-threading-python.html

since 3.13, Python supports free-threaded execution, in which GIL is optional.

Test againt this latest feature.

Does this PR introduce any user-facing change?

No

How was this patch tested?

PR builder with

default: '{"PYSPARK_IMAGE_TO_TEST": "python-313-nogil", "PYTHON_TO_TEST": "python3.13t", "PYTHON_GIL": "0" }'

see https://github.com/zhengruifeng/spark/actions/runs/14355716572/job/40244827286

PYTHON_GIL=0 is to forces the GIL off

  • For PySpark Classic, some streaming test and ml tests are skipped;
  • For PySpark Connect, it is not supported at all (one blocker is GRPC which is not compatible), so ALL tests are skipped

Was this patch authored or co-authored using generative AI tooling?

no

@zhengruifeng zhengruifeng force-pushed the infra_py313_nogil branch 2 times, most recently from 3ab3884 to 05965ad Compare April 8, 2025 07:15
@zhengruifeng zhengruifeng force-pushed the infra_py313_nogil branch 6 times, most recently from 0ab1001 to 07292b1 Compare April 9, 2025 05:33
fix

del lxml

upgrade grpcio

del grpcio

PYTHON_GIL=0

PYTHON_GIL=0

fix

fix

fix

fix

fix

fix

fix

fix

fix

fix

fix

fix
fix
@zhengruifeng zhengruifeng changed the title [TEST-ONLY] PySpark with free-threaded Python 3.13 [SPARK-51755][INFRA][PYTHON] Set up a scheduled builder for free-threaded Python 3.13 Apr 9, 2025
@zhengruifeng zhengruifeng marked this pull request as ready for review April 9, 2025 12:57
Copy link
Copy Markdown
Member

@dongjoon-hyun dongjoon-hyun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's nice to see this effort. Thank you, @zhengruifeng .

Copy link
Copy Markdown
Member

@dongjoon-hyun dongjoon-hyun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, LGTM.

@zhengruifeng
Copy link
Copy Markdown
Contributor Author

thanks, merged to master

@zhengruifeng zhengruifeng deleted the infra_py313_nogil branch April 10, 2025 01:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants