[Workaround] Docker-in-Docker for mulled containers

[almahmoud]

This is not meant to be merged, but a useful workaround until we figure out how to automate the process of finding combinations of tools not in biocontainers.
Having a docker-in-docker container will allow the job containers to build mulled containers on-the-fly and use them when the images are not found in biocontainers.

Copy of #58 made as a Draft PR.

[nuwang]

Just noting that this also needs the <build_mulled /> option in container_resolvers_conf.xml, as it has been removed now: 313cef6

[nuwang]

@almahmoud We need to specify the DOCKER_HOST env var for this right? As otherwise, the mulled build process won't know that it should be using the dind container? Where should that be done?

[almahmoud]

I followed this originally: https://medium.com/hootsuite-engineering/building-docker-images-inside-kubernetes-42c6af855f25 so I think the missing piece is:

    env:
    - name: DOCKER_HOST
      value: tcp://localhost:2375

I am not sure how/if the Kubernetes-Docker runtime divorce is going to affect this still working though