EPIC: Implement Solution For Running Jenkins Agent On Solaris with Java 17

[steelhead31]

See: #3741 

A solution will need to be investigated and implemented for running Jenkins agents on Solaris with Java 17.

Since JDK 17 is not supported on solaris, this will need additional work.

Specifically these 4 agents..

• build-siteox-solaris10u11-sparcv9-1
• test-siteox-solaris10u11-sparcv9-1
• build-azure-solaris10-x64-1
• test-azure-solaris10-x64-1

Whatever solution is found, will likely also need to be applied to the TCK solaris nodes.

Investigation an initial prototyping has been done under comments in this issue, the continuation work is in:

• Create new jobs to handle running Solaris builds via a Linux proxy machine temurin-build#4098
• Create new jobs to handle running Solaris tests via a Linux proxy machine temurin-build#4099
• Enable GPG signing of "solaris simple" builds temurin-build#4100

[sxa]

Options:

1. Attempt to rebuild the new LTS agent ourselves such that it will run on JDK11.
2. Attempt to build a JDK17 on Solaris 10 (There are patches to make it work with gcc but I struggled last time I tried to use them)
3. Attempt to build a JDK17 on Solaris 11 and switch to that (also with GCC as most Solaris support was removed in JDK15), while building against a Solaris 10 sysroot (subject to that being feasible in JDK8).
4. Same as the point above, but run a Solaris 10 zone under Solaris 11, similar to what we do with docker on Linux (Would also potentially allow us to satisfy Investigate feasibility of ephemeral build systems for non-Linux platforms temurin-build#3264 (comment)). This would require a bit of 'roll your own' since I doubt there's a docker plugin that can make this easy...
5. Running a non-Solaris JDK17 using qemu on Solaris
6. Running some proxy machine to run the jenkins agent and pass commands to the remote machine (likely hard because of the amount of shell invocations in our jenkins pipelines)
7. Run a second jenkins server running an older version which can communicate with the Solaris node.

Anything else?

Notes:

• All of the "build JDK17" choices would likely force us into building each JDK up from 11 to get there.
• We could probably survive with a 'zero' implementation instead of one with a full JIT if required. I suspect that may be all we could get on SPARC ...

[sxa]

IMHO Options 1 or 3 would be the cleanest ones if we can make them work.

[steelhead31]

1. A quick test of the tribblix Solaris 11 / JDK17 proved it was incompatible with Solaris 10
2. Reached out and confirmed that there is 100% no Solaris 10 / JDK17 package.

[sxa]

Diagram from a discussion this morning:

The diagram in the previous comment is intended to summarise the different ways where builds (also applicable to test, but I'll start with build)
Lines from top to bottom:

1. The normal situation where the jenkins agent runs on a system, and it runs the actual work on the same machine.
2. The 'docker build' situation where the agent runs in an environment (host that can run docker) but the actual work is done in a container managed by the jenkins plugins.
3. Agent runs on a machine, then connects into another environment on the machine which it runs work on over a connection (e.g. ssh). this is what we do with VagrantPlaybook check where the VM is created dynamically
4. Use an isolated old jenkins server to connect to the Solaris agents, so they can run with the older version. This version would not be updated, so needs to be firewalled off from anything that might connect to it (exception for the main jenkins server so it can initiate and copy artifacts back)
5. The jobs are run on a schedule to perform a build and then initiate tests (potentially over ssh to a remote machine to ensure build/test machine isolation) The output can be put in a staging area that can be retrieved by a jenkins job which has the artifacts in a way that 'looks' like the existing pipelines.
6. Run a Solaris 11 host (where it is technically possible to have a JDK17 - third parties have patched it to work) and then use that to run the build (using a shared file system with the host to allow the jenkins agent's file workspace to be shared with the Solaris 10 environment)

Given that above, option 6 is probably the cleanest and give isolation of the Solaris 10 environment from the internet. However this would require updating our machines and adjusting the pipelines to be 'zone aware' (unless there is a jenkins plugin for this as there is for docker, but that seems optimistic!) The simplest solution, which we discussed today, is option 3, where we replace the existing build and test pipelines - for Solaris only - with a time pipeline that will ssh to the Solaris machine and run "git clone temurin-build; run make-adopt-build.farm.sh; copy build artifacts back and artifact them into the jenkins job. We will look at prototyping this.