Empty error aborts role but installation continues

[maikknebel]

When i try to install an additional tool the installation returns an error and aborts. The webinterface is listing my installation as running and after a few it has been finished successfully. I guess this is related to #15 and #galaxyproject/galaxy#1628

equests.exceptions.ConnectionError: ('Connection aborted.', BadStatusLine(\"''\",))"

playbook.yml

- hosts: galaxyservers
  become: true
  vars:
      galaxy_restart_handler_enabled: false
      galaxy_tools_base_dir: "{{ galaxy_root }}/var/tmp/"
      galaxy_server_dir: "{{ galaxy_root }}/server"
      galaxy_venv_dir: "{{ galaxy_root }}/venv"
      galaxy_config_file: "{{ galaxy_root }}/config/galaxy.yml"
      galaxy_tools_api_key: 11111111
      galaxy_tools_tool_list_files: [ "files/tools.yml" ]
      galaxy_tools_galaxy_instance_url: http://127.0.0.1:8080/
  roles:
    - galaxyproject.galaxy-tools

tools.yml

tools:
- name: ncbi_blast_plus
  owner: devteam
  tool_panel_section_id: BLAST
  tool_panel_section_label: BLAST

Error

TASK [galaxyproject.galaxy-tools : Install Tool Shed tools] 
failed: [172.18.1.32] (item={u'owner': u'devteam', u'tool_panel_section_id': u'BLAST', u'name': u'ncbi_blast_plus', u'tool_panel_section_label': u'BLAST'}) => {
    "changed": false,
    "cmd": [
        "/srv/galaxy/var/tmp//venv/bin/shed-install",
        "-y",
        "name: ncbi_blast_plus\nowner: devteam\ntool_panel_section_id: BLAST\ntool_panel_section_label: BLAST\n",
        "-a",
        "111111111111111",
        "-g",
        "http://127.0.0.1:8080/"
    ],
    "delta": "0:01:01.338944",
    "end": "2019-05-28 09:31:49.159597",
    "failed_when_result": true,
    "invocation": {
        "module_args": {
            "_raw_params": "/srv/galaxy/var/tmp//venv/bin/shed-install -y \"name: ncbi_blast_plus\nowner: devteam\ntool_panel_section_id: BLAST\ntool_panel_section_label: BLAST\n\" -a \"111111111111111\" -g \"http://127.0.0.1:8080/\"",
            "_uses_shell": false,
            "argv": null,
            "chdir": null,
            "creates": null,
            "executable": null,
            "removes": null,
            "stdin": null,
            "warn": true
        }
    },
    "item": {
        "name": "ncbi_blast_plus",
        "owner": "devteam",
        "tool_panel_section_id": "BLAST",
        "tool_panel_section_label": "BLAST"
    },
    "msg": "non-zero return code",
    "rc": 1,
    "start": "2019-05-28 09:30:47.820653",
    "stderr": "(1/1) Installing tool ncbi_blast_plus from devteam to section \"BLAST\" at revision e25d3acf6e68 (TRT: 0:00:00.893317)\r\nTraceback (most recent call last):\n  File \"/srv/galaxy/var/tmp//venv/bin/shed-install\", line 10, in <module>\n    sys.exit(script_main())\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/ephemeris/shed_install.py\", line 720, in script_main\n    itm.install_tools()\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/ephemeris/shed_install.py\", line 627, in install_tools\n    install_repository_revision(tool, self.tsc)\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/ephemeris/shed_install.py\", line 502, in install_repository_revision\n    response = tsc.install_repository_revision(**tool)\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/bioblend/galaxy/toolshed/__init__.py\", line 146, in install_repository_revision\n    return self._post(url=url, payload=payload)\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/bioblend/galaxy/client.py\", line 152, in _post\n    files_attached=files_attached)\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/bioblend/galaxyclient.py\", line 137, in make_post_request\n    timeout=self.timeout)\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/api.py\", line 116, in post\n    return request('post', url, data=data, json=json, **kwargs)\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/api.py\", line 60, in request\n    return session.request(method=method, url=url, **kwargs)\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/sessions.py\", line 533, in request\n    resp = self.send(prep, **send_kwargs)\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/sessions.py\", line 646, in send\n    r = adapter.send(request, **kwargs)\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/adapters.py\", line 498, in send\n    raise ConnectionError(err, request=request)\nrequests.exceptions.ConnectionError: ('Connection aborted.', BadStatusLine(\"''\",))",
    "stderr_lines": [
        "(1/1) Installing tool ncbi_blast_plus from devteam to section \"BLAST\" at revision e25d3acf6e68 (TRT: 0:00:00.893317)",
        "Traceback (most recent call last):",
        "  File \"/srv/galaxy/var/tmp//venv/bin/shed-install\", line 10, in <module>",
        "    sys.exit(script_main())",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/ephemeris/shed_install.py\", line 720, in script_main",
        "    itm.install_tools()",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/ephemeris/shed_install.py\", line 627, in install_tools",
        "    install_repository_revision(tool, self.tsc)",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/ephemeris/shed_install.py\", line 502, in install_repository_revision",
        "    response = tsc.install_repository_revision(**tool)",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/bioblend/galaxy/toolshed/__init__.py\", line 146, in install_repository_revision",
        "    return self._post(url=url, payload=payload)",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/bioblend/galaxy/client.py\", line 152, in _post",
        "    files_attached=files_attached)",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/bioblend/galaxyclient.py\", line 137, in make_post_request",
        "    timeout=self.timeout)",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/api.py\", line 116, in post",
        "    return request('post', url, data=data, json=json, **kwargs)",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/api.py\", line 60, in request",
        "    return session.request(method=method, url=url, **kwargs)",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/sessions.py\", line 533, in request",
        "    resp = self.send(prep, **send_kwargs)",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/sessions.py\", line 646, in send",
        "    r = adapter.send(request, **kwargs)",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/adapters.py\", line 498, in send",
        "    raise ConnectionError(err, request=request)",
        "requests.exceptions.ConnectionError: ('Connection aborted.', BadStatusLine(\"''\",))"
    ],
    "stdout": "",
    "stdout_lines": []
}
...ignoring

[bernt-matthias]

I observe the same here. When installing via the UI I often observe timeouts from the proxy. We are currently planning to increase the timeout.

[maikknebel]

After a few more tries... the error appears exactly after 60 seconds in the given task-step. The error message above shows that as well.

"start": "2019-05-28 09:30:47.820653"
"end": "2019-05-28 09:31:49.159597",

The gui installation runs fine for me - is requesting the status every few seconds from the api. Just the shed-install seems to get a timeout somewhere when using the ansible role.

Edit:
The error also occurres from the terminal, when i manually activate the virtualenv and call shed-install directly.

[mvdbeek]

Try bumping up the ephemeris version. 0.9.0 should contain galaxyproject/ephemeris@2499ba5

[bernt-matthias]

@maikknebel can you try to change this here:

ansible-galaxy-tools/tasks/tools.yml

Line 6 in 935fc8b

- ephemeris==0.8.0 # Pinned version should make sure ephemeris version matches what the role has been tested with

[bernt-matthias]

Hmm. In my case its actually a 502 Proxy Error.

The proxy server could not handle the request <em><a href=\"/api/tool_shed_repositories/new/install_repository_revision\">POST&nbsp;/api/tool_shed_repositories/new/install_repository_revision</a> 
...
Reason: <strong>Error reading from remote server

Since the runtime in 0:01:01.485219 I guess this is apache's way to signal a timeout..? But might also be a misconfiguration of the Apache.

[maikknebel]

I tried version 0.8.0 and 0.9.0 both give me the following error:

fatal: [172.18.1.32]: FAILED! => {
    "msg": "The conditional check ''installed successfully' in install_result.stderr' failed. The error was: error while evaluating conditional ('installed successfully' in install_result.stderr): Unable to look up a name or access an attribute in template string ({% if 'installed successfully' in install_result.stderr %} True {% else %} False {% endif %}).\nMake sure your variable name does not contain invalid characters like '-': argument of type 'StrictUndefined' is not iterable"
}

Coming from this block in tasks/instal_tool_lists.yml

- name: Install Tool Shed tools
  command: '{{ galaxy_tools_base_dir }}/venv/bin/shed-install -y "{{ item | to_nice_yaml }}" -a "{{ galaxy_tools_api_key
 }}" -g "{{ galaxy_tools_galaxy_instance_url }}"'
  register: install_result
  changed_when: "'installed successfully' in install_result.stderr"
  failed_when: ('Error installing' in install_result.stderr) or ('Missing required' in install_result.stderr) or (instal
l_result.rc != 0)
  ignore_errors: "{{ galaxy_tools_ignore_errors }}"
  notify:
    - restart galaxy
  with_items:
    - "{{ tools }}"

[maikknebel]

I had to replace the command, since it has changed from 0.4.0 to 0.8.0/0.9.0

command: '{{ galaxy_tools_base_dir }}/venv/bin/shed-install -y "{{ item | to_nice_yaml }}" -a "{{ galaxy_tools_api_key }}" -g "{{ galaxy_tools_galaxy_instance_url }}"'

with

command: '{{ galaxy_tools_base_dir }}/venv/bin/shed-tools install -y "{{ item | to_nice_yaml }}" -a "{{ galaxy_tools_api_key }}" -g "{{ galaxy_tools_galaxy_instance_url }}"'

The error stays the same. The role terminates after 60 seconds but the installation continiues.

[mvdbeek]

Is it really the exact same traceback ?
This

requests.exceptions.ConnectionError: ('Connection aborted.', BadStatusLine(\"''\",))

should really not appear anymore in 0.9.0

[maikknebel]

Most likely, some of the scripts in the trace have changed but i guess thats ok due to the changes in ephemeris. The error message and the last calls in the trace are the same.

failed: [172.18.1.32] (item={u'owner': u'devteam', u'tool_panel_section_label': u'BLAST', u'name': u'ncbi_blast_plus'}) => {
    "changed": false,
    "cmd": [
        "/srv/galaxy/var/tmp//venv/bin/shed-tools",
        "install",
        "-y",
        "name: ncbi_blast_plus\nowner: devteam\ntool_panel_section_label: BLAST\n",
        "-a",
        "1111111",
        "-g",
        "http://127.0.0.1:8080/"
    ],
    "delta": "0:01:01.789697",
    "end": "2019-05-28 14:33:59.683804",
    "failed_when_result": true,
    "invocation": {
        "module_args": {
            "_raw_params": "/srv/galaxy/var/tmp//venv/bin/shed-tools install -y \"name: ncbi_blast_plus\nowner: devteam\ntool_panel_section_label: BLAST\n\" -a \"11111111\" -g \"http://127.0.0.1:8080/\"",
            "_uses_shell": false,
            "argv": null,
            "chdir": null,
            "creates": null,
            "executable": null,
            "removes": null,
            "stdin": null,
            "warn": true
        }
    },
    "item": {
        "name": "ncbi_blast_plus",
        "owner": "devteam",
        "tool_panel_section_label": "BLAST"
    },
    "msg": "non-zero return code",
    "rc": 1,
    "start": "2019-05-28 14:32:57.894107",
    "stderr": "Storing log file in: /tmp/ephemeris_tniPFt\r\n(1/1) Installing repository ncbi_blast_plus from devteam to section \"BLAST\" at revision e25d3acf6e68 (TRT: 0:00:00.669282)\r\nTraceback (most recent call last):\n  File \"/srv/galaxy/var/tmp//venv/bin/shed-tools\", line 10, in <module>\n    sys.exit(main())\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/ephemeris/shed_tools.py\", line 937, in main\n    install_tool_manager.install_repositories()\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/ephemeris/shed_tools.py\", line 716, in install_repositories\n    install_repository_revision(repository, self.tsc)\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/ephemeris/shed_tools.py\", line 517, in install_repository_revision\n    response = tool_shed_client.install_repository_revision(**repository)\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/bioblend/galaxy/toolshed/__init__.py\", line 146, in install_repository_revision\n    return self._post(url=url, payload=payload)\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/bioblend/galaxy/client.py\", line 152, in _post\n    files_attached=files_attached)\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/bioblend/galaxyclient.py\", line 137, in make_post_request\n    timeout=self.timeout)\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/api.py\", line 116, in post\n    return request('post', url, data=data, json=json, **kwargs)\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/api.py\", line 60, in request\n    return session.request(method=method, url=url, **kwargs)\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/sessions.py\", line 533, in request\n    resp = self.send(prep, **send_kwargs)\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/sessions.py\", line 646, in send\n    r = adapter.send(request, **kwargs)\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/adapters.py\", line 498, in send\n    raise ConnectionError(err, request=request)\nrequests.exceptions.ConnectionError: ('Connection aborted.', BadStatusLine(\"''\",))",
    "stderr_lines": [
        "Storing log file in: /tmp/ephemeris_tniPFt",
        "(1/1) Installing repository ncbi_blast_plus from devteam to section \"BLAST\" at revision e25d3acf6e68 (TRT: 0:00:00.669282)",
        "Traceback (most recent call last):",
        "  File \"/srv/galaxy/var/tmp//venv/bin/shed-tools\", line 10, in <module>",
        "    sys.exit(main())",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/ephemeris/shed_tools.py\", line 937, in main",
        "    install_tool_manager.install_repositories()",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/ephemeris/shed_tools.py\", line 716, in install_repositories",
        "    install_repository_revision(repository, self.tsc)",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/ephemeris/shed_tools.py\", line 517, in install_repository_revision",
        "    response = tool_shed_client.install_repository_revision(**repository)",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/bioblend/galaxy/toolshed/__init__.py\", line 146, in install_repository_revision",
        "    return self._post(url=url, payload=payload)",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/bioblend/galaxy/client.py\", line 152, in _post",
        "    files_attached=files_attached)",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/bioblend/galaxyclient.py\", line 137, in make_post_request",
        "    timeout=self.timeout)",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/api.py\", line 116, in post",
        "    return request('post', url, data=data, json=json, **kwargs)",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/api.py\", line 60, in request",
        "    return session.request(method=method, url=url, **kwargs)",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/sessions.py\", line 533, in request",
        "    resp = self.send(prep, **send_kwargs)",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/sessions.py\", line 646, in send",
        "    r = adapter.send(request, **kwargs)",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/adapters.py\", line 498, in send",
        "    raise ConnectionError(err, request=request)",
        "requests.exceptions.ConnectionError: ('Connection aborted.', BadStatusLine(\"''\",))"
    ],
    "stdout": "",
    "stdout_lines": []
}
...ignoring

[mvdbeek]

galaxyproject/ephemeris@0.9.0...master ... we released 0.9.0 not so long ago, but we need another release to get that commit ...

[mvdbeek]

There's a new package on pypi now, 0.10.0, that should hopefully work

[maikknebel]

With 0.10.0 its working. Thank you.

Should the playbook in ansible galaxy be updated or the admintraining to directly pull the playbook from git? To reflect the version change.

[bernt-matthias]

I guess so. Maybe just open a PR?

[maikknebel]

I just made a clean install to test everything again... not working :( After 60 seconds in TASK [galaxyproject.galaxy-tools : Install Tool Shed tools], the script fails with the BadStatusLine-Error. My guess is, that it has worked before, because conda has cached the dependency package in the previous tries.

While this error appears, htop shows the following line that with conda installing the dependencies of the package. In the admin interface it says "installing dependencies". The Installation Monitor says, dependency not installed and no error. The dependency overview says that blast is installed. After a few minutes, the conda-task is gone from htop and nothing changed. After a restart of galaxy, the conda task appears again with the same command but taking far longer (10-15 min) than before. Another conda task appeared after that is still running... not sure if thats ok but nothing changed in the installation status. There have been no errors in the galaxy console so far.

Update: After 25 minutes, the second conda task finished and "manage dependencies" lists blast with a red cross, dependency not installed. Manage tools says installed for the first, and installed but missing dependencies for the second and third.

First conda task

12398 galaxy     20   0  613M  505M 12500 D 17.2  8.5  0:58.31 /srv/galaxy/var/dependencies/_conda/bin/python /srv/galaxy/var/dependencies/_conda/bin/conda create -y --quiet --override-channels --channel iuc --channel conda-forge --channel bioconda --channel defaults --name __blast@2.7.1 blast=2.7.1

Second conda task

12746 galaxy     21   1  577M  469M 12428 R  9.0  7.9  0:43.84 /srv/galaxy/var/dependencies/_conda/bin/python /srv/galaxy/var/dependencies/_conda/bin/conda create -y --quiet --override-channels --channel iuc --channel conda-forge --channel bioconda --channel defaults --name __blast@_uv_ blast

TASK [galaxyproject.galaxy-tools : Install Tool Shed tools]
failed: [172.18.1.32] (item={u'owner': u'devteam', u'tool_panel_section_label': u'BLAST', u'name': u'ncbi_blast_plus'}) => {
    "changed": false,
    "cmd": [
        "/srv/galaxy/var/tmp//venv/bin/shed-tools",
        "install",
        "-y",
        "name: ncbi_blast_plus\nowner: devteam\ntool_panel_section_label: BLAST\n",
        "-a",
        "1111111111111111",
        "-g",
        "http://127.0.0.1:8080/"
    ],
    "delta": "0:01:02.685348",
    "end": "2019-06-04 05:45:55.272057",
    "failed_when_result": true,
    "invocation": {
        "module_args": {
            "_raw_params": "/srv/galaxy/var/tmp//venv/bin/shed-tools install -y \"name: ncbi_blast_plus\nowner: devteam\ntool_panel_section_label: BLAST\n\" -a \"11111111111111111111111\" -g \"http://127.0.0.1:8080/\"",
            "_uses_shell": false,
            "argv": null,
            "chdir": null,
            "creates": null,
            "executable": null,
            "removes": null,
            "stdin": null,
            "warn": true
        }
    },
    "item": {
        "name": "ncbi_blast_plus",
        "owner": "devteam",
        "tool_panel_section_label": "BLAST"
    },
    "msg": "non-zero return code",
    "rc": 1,
    "start": "2019-06-04 05:44:52.586709",
    "stderr": "Storing log file in: /tmp/ephemeris_zAoFxM\r\n(1/1) Installing repository ncbi_blast_plus from devteam to section \"BLAST\" at revision e25d3acf6e68 (TRT: 0:00:01.542898)\r\nTraceback (most recent call last):\n  File \"/srv/galaxy/var/tmp//venv/bin/shed-tools\", line 10, in <module>\n    sys.exit(main())\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/ephemeris/shed_tools.py\", line 546, in main\n    **kwargs)\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/ephemeris/shed_tools.py\", line 160, in install_repositories\n    result = self.install_repository_revision(repository, log)\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/ephemeris/shed_tools.py\", line 355, in install_repository_revision\n    response = self.tool_shed_client.install_repository_revision(**repository)\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/bioblend/galaxy/toolshed/__init__.py\", line 146, in install_repository_revision\n    return self._post(url=url, payload=payload)\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/bioblend/galaxy/client.py\", line 160, in _post\n    files_attached=files_attached)\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/bioblend/galaxyclient.py\", line 137, in make_post_request\n    timeout=self.timeout)\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/api.py\", line 116, in post\n    return request('post', url, data=data, json=json, **kwargs)\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/api.py\", line 60, in request\n    return session.request(method=method, url=url, **kwargs)\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/sessions.py\", line 533, in request\n    resp = self.send(prep, **send_kwargs)\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/sessions.py\", line 646, in send\n    r = adapter.send(request, **kwargs)\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/adapters.py\", line 498, in send\n    raise ConnectionError(err, request=request)\nrequests.exceptions.ConnectionError: ('Connection aborted.', BadStatusLine(\"''\",))",
    "stderr_lines": [
        "Storing log file in: /tmp/ephemeris_zAoFxM",
        "(1/1) Installing repository ncbi_blast_plus from devteam to section \"BLAST\" at revision e25d3acf6e68 (TRT: 0:00:01.542898)",
        "Traceback (most recent call last):",
        "  File \"/srv/galaxy/var/tmp//venv/bin/shed-tools\", line 10, in <module>",
        "    sys.exit(main())",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/ephemeris/shed_tools.py\", line 546, in main",
        "    **kwargs)",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/ephemeris/shed_tools.py\", line 160, in install_repositories",
        "    result = self.install_repository_revision(repository, log)",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/ephemeris/shed_tools.py\", line 355, in install_repository_revision",
        "    response = self.tool_shed_client.install_repository_revision(**repository)",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/bioblend/galaxy/toolshed/__init__.py\", line 146, in install_repository_revision",
        "    return self._post(url=url, payload=payload)",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/bioblend/galaxy/client.py\", line 160, in _post",
        "    files_attached=files_attached)",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/bioblend/galaxyclient.py\", line 137, in make_post_request",
        "    timeout=self.timeout)",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/api.py\", line 116, in post",
        "    return request('post', url, data=data, json=json, **kwargs)",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/api.py\", line 60, in request",
        "    return session.request(method=method, url=url, **kwargs)",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/sessions.py\", line 533, in request",
        "    resp = self.send(prep, **send_kwargs)",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/sessions.py\", line 646, in send",
        "    r = adapter.send(request, **kwargs)",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/adapters.py\", line 498, in send",
        "    raise ConnectionError(err, request=request)",
        "requests.exceptions.ConnectionError: ('Connection aborted.', BadStatusLine(\"''\",))"
    ],
    "stdout": "",
    "stdout_lines": []
}
...ignoring

[mvdbeek]

This is very strange, the line numbers in the traceback match 0.10.0, but the exception is explicitly wrapped in

try:
    response = self.tool_shed_client.install_repository_revision(**repository)
    ...
except ConnectionError as e:
    ...
    elif "504" in str(e) or 'Connection aborted' in str(e):
        success = self.wait_for_install(repository=repository, log=log, timeout=3600)

that condition really should evaluate to true in your traceback.

Can you print out the ephemeris version before the install and provide an explicit log file with --log_file ? (/dev/stdout should work to get the log reported in ansible)