Almost all nodes are offline with "This node is offline because it uses an old slave.jar"

[richardlau]

From irc:

[20190412 02:10:10] <refack> We updated Jenkins. So it probably got picky

[rvagg]

Yeah .. so the security update of Jenkins brought us onto a newer version, their LTS channel is a bit funky in that it makes us jump with a simple update, no opt-in. The update made our slave.jar's all out of date so I've manually been through all of them and updated slave.jar.

I need @nodejs/platform-aix @nodejs/platform-s390 @nodejs/platform-ppc folks to do these ones though, they're out of my pay grade.:

test-iinthecloud-ibmi72-ppc64_be-1
test-iinthecloud-ibmi72-ppc64_be-2
test-linuxonecc-rhel72-s390x-3
test-marist-zos13-s390x-1
test-marist-zos13-s390x-2
test-osuosl-aix61-ppc64_be-1
test-osuosl-aix61-ppc64_be-2
test-osuosl-aix61-ppc64_be-3
test-osuosl-ubuntu1404-ppc64_le-4

I've disabled https://ci.nodejs.org/job/node-test-commit-aix/ so that can be re-enabled when the AIX ones are online.

[sam-github]

@rvagg I have access to those machines, and would like to help, but I need a hint! I have been learning a bit about ansible lately, but don't know anything about jenkinds

From where do I get the slave.jar? Can I just re-run ansible on them to update, or is this a manual process?

[sam-github]

https://github.com/nodejs/build/blob/master/ansible/playbooks/jenkins/worker/upgrade-jar.yml and https://github.com/nodejs/build/blob/master/ansible/group_vars/infra.yml#L1 makes it look like ansible will do this for me, so we just need to rerun ansible on those hosts?

@mhdawson & @rvagg ^---- shall I do this? Though I only have access to the test machines, not release.

[refack]

Some hints at https://github.com/nodejs/build/blob/master/doc/jenkins-guide.md

tl;dr:

1. On the machines stop the Jenkins worker daemon
2. cd /home/iojs && curl -L https://ci.nodejs.org/jnlpJars/slave.jar -o slave.jar
3. chown iojs.iojs slave.jar (or the platform equivalant)
4. Restart the Jenkins worker daemon
5. Fun & Profit

[refack]

Though I only have access to the test machines, not release.

It seems like the Jenkins version on ci-release is not rejecting old worker remoting jars, so it's less of an issue.

[mhdawson]

Did somebody already do it for the aix machines? It looks like they are online

[mhdawson]

I've updated the IBM platforms in the release CI so if we do a similar upgrade there we should be ok .

[sam-github]

test-linuxonecc-rhel72-s390x-2
test-linuxonecc-rhel72-s390x-3
test-osuosl-ubuntu1404-ppc64_le-1
test-osuosl-ubuntu1404-ppc64_le-2
test-osuosl-ubuntu1404-ppc64_le-4``` are all up-to-date.

The two I updated, test-linuxonecc-rhel72-s390x-3 and test-osuosl-ubuntu1404-ppc64_le-4 are restarted and status says connected.

[mhdawson]

@rvagg do the updates get applied automatically? I went looking for an issue with a heads up that an update to jenkins was going to be applied both in the build repo and in the security repo for some sort of heads up that there was going to be an update. We might want to be proactive in updating the agents or at least have people ready to act if the udpate requires and update.

[miladfarca]

@mhdawson @sam-github I've updated the jar file on test-marist-zos13-s390x-1 and test-marist-zos13-s390x-2. Left the file permissions as the older jar file.