Migrate away from google.com gcp project kubernetes-scale

[spiffxp]

Related:

• Part of umbrella issue to migrate away from google.com gcp projects: Umbrella issue: migrate away from google.com gcp projects #1469
• Part of: Migrate sig-scalability jobs to k8s-infra perf-tests#1898

Umbrella issue for migrating or removing dependence on the kubernetes-scale project. cs.k8s.io results show that:

• it's used for our 5k node scalability tests, specifically
  • ci-kubernetes-e2e-gce-scale-correctness
  • ci-kubernetes-e2e-gce-scale-performance
• these jobs also log to gs://sig-scalability-logs which lives under kubernetes-jenkins

We should:

• stand up an equivalent (singular) project in k8s-infra : infra/gcp/prow: Add e2e project for sig-scalability #2195
• determine what quota is needed for it : Increase quotas for k8s-infra-e2e-scale-5k-project #2225
• request the quota : Increase quotas for k8s-infra-e2e-scale-5k-project #2225
• setup an equivalent bucket for logs in k8s-infra : Re-deploy k8s-infra-scalability-tests-logs to kubernetes-public #2240
• setup and run canary jobs: wg-k8s-infra: canary prowjobs for sig-scalability test-infra#22430
• drop the canary job / switch the live job for ci-kubernetes-e2e-scale-correctness: sig-scalability: migrate e2e-gce-scale-correctness test-infra#24080)
• drop the canary job / switch the live job for ci-kubernetes-e2e-scale-performance: sig-scalability: migrate ci-kubernetes-e2e-gce-scale-performance test-infra#23807

/assign @ameukam
It turns out Arnaud has done a fair amount of this, but I just realized we never had an umbrella issue to tie it to the project we're working to replace. I'll link / edit what's been done so far

/priority important-soon
/wg k8s-infra
/sig scalability
/sig release
since these are release-informing but treated as release-blocking jobs
/sig testing

[spiffxp]

/milestone v1.22
I'm not quite sure it's reasonable to expect we can flip this in time for Test Freeze, but we want this ready by the time the next cycle starts

[ameukam]

Some issues about the canary jobs:

• ci-kubernetes-e2e-gce-scale-correctness-canary requests compute resources superior to what the build clusters can offer. We may need adjust those requests. EDIT: I raised to quotas in Increase quotas for k8s-infra-e2e-scale-5k-project #2225
• I see failures about gce instance access in ci-kubernetes-kubemark-gce-scale-canary, ci-kubernetes-e2e-gce-scale-performance-canary and ci-kubernetes-kubemark-gce-scale-scheduler-canary. I'll enable OS Login for k8s-infra-scale-e2e-5k-project project. EDIT/ os-loginis not needed for the scalability jobs.