Skip to content

[SPARK-55293][PS][TESTS] Avoid using old offset aliases#54077

Closed
ueshin wants to merge 3 commits intoapache:masterfrom
ueshin:issues/SPARK-55293/offset_alias
Closed

[SPARK-55293][PS][TESTS] Avoid using old offset aliases#54077
ueshin wants to merge 3 commits intoapache:masterfrom
ueshin:issues/SPARK-55293/offset_alias

Conversation

@ueshin
Copy link
Copy Markdown
Member

@ueshin ueshin commented Jan 30, 2026

What changes were proposed in this pull request?

Avoid using old offset aliases.

  • "H" -> "h"
  • "T" -> "min"
  • "S" -> "s"
  • "L" -> "ms"
  • "U" -> "us"
  • "Q" -> "QE"
  • "Y" -> "YE"

https://pandas.pydata.org/docs/user_guide/timeseries.html#offset-aliases

Why are the changes needed?

#54018 fixed the offset alias M to ME.

There are other aliases that are deprecated in pandas 2.2 / removed in pandas 3.0.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Fixed the related tests.

Was this patch authored or co-authored using generative AI tooling?

No.

@github-actions
Copy link
Copy Markdown

JIRA Issue Information

=== Sub-task SPARK-55293 ===
Summary: Avoid using old offset aliases
Assignee: None
Status: Open
Affected: ["4.2.0"]


This comment was automatically generated by GitHub Actions

@ueshin
Copy link
Copy Markdown
Member Author

ueshin commented Jan 30, 2026

@ueshin
Copy link
Copy Markdown
Member Author

ueshin commented Jan 30, 2026

? My repo doesn't link the build status back.

https://github.com/ueshin/apache-spark/actions/runs/21533781589

@gaogaotiantian
Copy link
Copy Markdown
Contributor

I believe there is a difference between an offset alias and a period alias. For example, period_range should use period_alias which should be M instead of ME. I'm not 100% sure which one is which in our code base.

I think the difference is that, offset alias means the next point after this much time; whereas period alias means the span of this much time.

@ueshin
Copy link
Copy Markdown
Member Author

ueshin commented Jan 30, 2026

@gaogaotiantian Good catch!
I missed it because actually the test is skipped as it doesn't support Period type.

@ueshin ueshin changed the title [SPARK-55293][PS][TESTS Avoid using old offset aliases [SPARK-55293][PS][TESTS] Avoid using old offset aliases Jan 30, 2026
@HyukjinKwon
Copy link
Copy Markdown
Member

Merged to master.

HyukjinKwon pushed a commit that referenced this pull request Feb 3, 2026
### What changes were proposed in this pull request?

This is a follow-up of #54077.

Avoids more old offset aliases.

### Why are the changes needed?

There are still some places using the old offset aliases.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Fixed the related tests.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #54095 from ueshin/issues/SPARK-55293/ye.

Authored-by: Takuya Ueshin <ueshin@databricks.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants