[SPARK-55076][PYTHON] Fix the type hint issue in ml/mllib and add scipy requirement

[gaogaotiantian]

What changes were proposed in this pull request?

• Pin scipy version to >=1.8.0, which is the first minor version to support 3.10
• Install scipy on lint image so we can find scipy related lint failures
• Add sparray as that's the preferred type for scipy now
• Expand VectorLike to include other vector like types to simplify our code
• Replace some type(x) check with isinstance() because that's the recommended way and mypy understands it
• Fix a few numpy 1 vs 2 related type hints so they can pass with both versions
• Add a few assertions to make mypy happy about attributes

Why are the changes needed?

Currently, local mypy check will fail with a lot of failures due to scipy/numpy because our lint image does not include those stubs. This is bad because it's really hard for people to do mypy check locally - they'll think that their environment setup has issues so mypy result is not to be trusted. We want to make mypy result consistent between CI and local and make it clean.

Does this PR introduce any user-facing change?

It should not. Almost all changes are type annotation related.

How was this patch tested?

CI should pass.

Was this patch authored or co-authored using generative AI tooling?

No

[github-actions]

JIRA Issue Information

=== Improvement SPARK-55076 ===
Summary: Fix type annotation issue for ml/mllib
Assignee: None
Status: Open
Affected: ["4.2.0"]

---

This comment was automatically generated by GitHub Actions

[zhengruifeng]

merged to master