Skip to content

Add vg 1.4.0#5086

Closed
johanneskoester wants to merge 26 commits into
masterfrom
vg
Closed

Add vg 1.4.0#5086
johanneskoester wants to merge 26 commits into
masterfrom
vg

Conversation

@johanneskoester
Copy link
Copy Markdown
Contributor

  • I have read the guidelines for bioconda recipes.
  • This PR adds a new recipe.
  • This PR updates an existing recipe.
  • This PR does something else (explain below).

@bioconda-bot
Copy link
Copy Markdown
Collaborator

Linting

Your recipes are in an excellent condition.

1 similar comment
@bioconda-bot
Copy link
Copy Markdown
Collaborator

Linting

Your recipes are in an excellent condition.

@johanneskoester
Copy link
Copy Markdown
Contributor Author

johanneskoester commented Jul 3, 2017

@bgruening this fails because the build requires some git submodules. What is our recommended strategy for such a case (since using the git_url tag is discouraged)?

@druvus
Copy link
Copy Markdown
Member

druvus commented Jul 7, 2017

@bgruening @johanneskoester I wanted to add vg some time ago but put it on hold due to dependancy of +20 submodules. Maybe we can do an exception here and use git_url so it could be added?

@johanneskoester
Copy link
Copy Markdown
Contributor Author

Yes.

@bgruening
Copy link
Copy Markdown
Member

I think the recommended way is to ask for a release tarball if this is not possible use git_url.

@bioconda-bot
Copy link
Copy Markdown
Collaborator

Linting

The following problems have been found

Recipe failed checks
recipes/vg missing_hash
recipes/vg uses_git_url

@bioconda-bot
Copy link
Copy Markdown
Collaborator

Linting

Your recipes are in an excellent condition.

@bioconda-bot
Copy link
Copy Markdown
Collaborator

Linting

The following problems have been found

Recipe failed checks
recipes/vg missing_hash
recipes/vg uses_git_url

@bioconda-bot
Copy link
Copy Markdown
Collaborator

Linting

Your recipes are in an excellent condition.

@bioconda-bot
Copy link
Copy Markdown
Collaborator

Linting

Your recipes are in an excellent condition.

@bioconda-bot
Copy link
Copy Markdown
Collaborator

Linting

Your recipes are in an excellent condition.

@bioconda-bot
Copy link
Copy Markdown
Collaborator

Linting

Your recipes are in an excellent condition.

@bioconda-bot
Copy link
Copy Markdown
Collaborator

Linting

The following problems have been found

Recipe failed checks
recipes/vg missing_hash
recipes/vg uses_git_url

@bioconda-bot
Copy link
Copy Markdown
Collaborator

Linting

Your recipes are in an excellent condition.

@bioconda-bot
Copy link
Copy Markdown
Collaborator

Linting

Your recipes are in an excellent condition.

@johanneskoester
Copy link
Copy Markdown
Contributor Author

johanneskoester commented Jul 18, 2017

The remaining error seems to be a problem with the compiler version: vgteam/vg#751. We need gcc 4.9 for this to work.

@mbargull
Copy link
Copy Markdown
Member

mbargull commented Feb 1, 2018

The error shows they use std::regex, thus gcc >=4.9 would be needed..

@mbargull
Copy link
Copy Markdown
Member

mbargull commented Feb 1, 2018

https://github.com/vgteam/vg/blame/v1.6.0/README.md#L33:

At present, you will need GCC version 4.9 or greater to compile vg. (Check your version with gcc --version.)

@johanneskoester
Copy link
Copy Markdown
Contributor Author

Yeah, I had the hope that those patches in devtoolset-2 gcc would help here :-(.

@mbargull
Copy link
Copy Markdown
Member

mbargull commented Feb 5, 2018

Note that with the new compiler packages you have to make sure CC & co are used. The actual binaries are called, e.g, x86_64-conda_cos6-linux-gnu-gcc instead of just gcc. The activation scripts of the compiler packages take care of setting CC etc. accordingly. https://www.anaconda.com/blog/developer-blog/utilizing-the-new-compilers-in-anaconda-distribution-5/

@mbargull
Copy link
Copy Markdown
Member

mbargull commented Feb 5, 2018

And you have to use the gxx_* packages as gcc_* doesn't contain CXX/*-g++

@johanneskoester
Copy link
Copy Markdown
Contributor Author

Seems like protobuf still wants to use gcc...

@mbargull
Copy link
Copy Markdown
Member

mbargull commented Feb 5, 2018

source_me.sh is still run by Makefile, e.g., https://github.com/vgteam/vg/blob/1f8d1fa24521243546fa12d060b1876cfbf692ab/Makefile#L189.
Hey, but you could patch everything so that vg doesn't vendor "all the things".
... Okay, since that sounds to be no fun potentially/time consuming, you could first try to just remove https://github.com/vgteam/vg/blob/1f8d1fa24521243546fa12d060b1876cfbf692ab/source_me.sh#L8-L9 with a patch.

@johanneskoester
Copy link
Copy Markdown
Contributor Author

Yes, that is what I will do. Yeah removing the dependencies would be nice indeed...

@johanneskoester
Copy link
Copy Markdown
Contributor Author

@mbargull I think I am stuck here. Seems like the error is caused by protobuf still not finding g++: protocolbuffers/protobuf#492. Since you have already familiarized with the new compilers, any idea?

@mbargull
Copy link
Copy Markdown
Member

mbargull commented Feb 20, 2018

protobuf's config.log shows a lot of

$PREFIX/bin/../lib/gcc/x86_64-conda_cos6-linux-gnu/7.2.0/../../../../x86_64-conda_cos6-linux-gnu/bin/ld: cannot find -lgomp

errors -- let's see if adding openmp helps..

EDIT: Specifically, this also failed at

configure:19825: checking for library containing sched_yield
configure:19856: /opt/conda/conda-bld/vg_1519118731750/_b_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/bin/x86_64-conda_cos6-linux-gnu-c++ -o conftest -O3 -fopenmp -std=c++11 -ggdb -g -MMD -MP -fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -I/opt/conda/conda-bld/vg_1519118731750/_b_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/include -fdebug-prefix-map=src=/usr/local/src/conda/vg-1.6.0 -fdebug-prefix-map=/opt/conda/conda-bld/vg_1519118731750/_b_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_=/usr/local/src/conda-prefix -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,-rpath,/opt/conda/conda-bld/vg_1519118731750/_b_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib -L/opt/conda/conda-bld/vg_1519118731750/_b_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib conftest.cpp  >&5
/opt/conda/conda-bld/vg_1519118731750/_b_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/bin/../lib/gcc/x86_64-conda_cos6-linux-gnu/7.2.0/../../../../x86_64-conda_cos6-linux-gnu/bin/ld: cannot find -lgomp
configure:19856: $? = 1
configure: failed program was:
| /* confdefs.h */
| #define PACKAGE_NAME "Protocol Buffers"
| #define PACKAGE_TARNAME "protobuf"
| #define PACKAGE_VERSION "3.4.0"
| #define PACKAGE_STRING "Protocol Buffers 3.4.0"
| #define PACKAGE_BUGREPORT "protobuf@googlegroups.com"
| #define PACKAGE_URL ""
| #define PACKAGE "protobuf"
| #define VERSION "3.4.0"
| #define HAVE_SYS_TYPES_H 1
| #define HAVE_SYS_STAT_H 1
| #define HAVE_STDLIB_H 1
| #define HAVE_STRING_H 1
| #define HAVE_MEMORY_H 1
| #define HAVE_STRINGS_H 1
| #define HAVE_INTTYPES_H 1
| #define HAVE_STDINT_H 1
| #define HAVE_UNISTD_H 1
| #define __EXTENSIONS__ 1
| #define _ALL_SOURCE 1
| #define _GNU_SOURCE 1
| #define _POSIX_PTHREAD_SEMANTICS 1
| #define _TANDEM_SOURCE 1
| #define HAVE_DLFCN_H 1
| #define LT_OBJDIR ".libs/"
| #define HAVE_FCNTL_H 1
| #define HAVE_INTTYPES_H 1
| #define HAVE_LIMITS_H 1
| #define HAVE_STDLIB_H 1
| #define HAVE_UNISTD_H 1
| #define HAVE_PTHREAD 1
| #define HAVE_HASH_MAP 1
| #define HAVE_HASH_SET 1
| #define HASH_MAP_H <unordered_map>
| #define HASH_SET_H <unordered_set>
| #define HASH_NAMESPACE std
| #define HASH_MAP_CLASS unordered_map
| #define HASH_SET_CLASS unordered_set
| /* end confdefs.h.  */
| 
| /* Override any GCC internal prototype to avoid an error.
|    Use char because int might match the return type of a GCC
|    builtin and then its argument prototype would still apply.  */
| #ifdef __cplusplus
| extern "C"
| #endif
| char sched_yield ();
| int
| main ()
| {
| return sched_yield ();
|   ;
|   return 0;
| }
configure:19856: /opt/conda/conda-bld/vg_1519118731750/_b_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/bin/x86_64-conda_cos6-linux-gnu-c++ -o conftest -O3 -fopenmp -std=c++11 -ggdb -g -MMD -MP -fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -I/opt/conda/conda-bld/vg_1519118731750/_b_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/include -fdebug-prefix-map=src=/usr/local/src/conda/vg-1.6.0 -fdebug-prefix-map=/opt/conda/conda-bld/vg_1519118731750/_b_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_=/usr/local/src/conda-prefix -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,-rpath,/opt/conda/conda-bld/vg_1519118731750/_b_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib -L/opt/conda/conda-bld/vg_1519118731750/_b_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib conftest.cpp -lrt   >&5
/opt/conda/conda-bld/vg_1519118731750/_b_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/bin/../lib/gcc/x86_64-conda_cos6-linux-gnu/7.2.0/../../../../x86_64-conda_cos6-linux-gnu/bin/ld: cannot find -lgomp
configure:19856: $? = 1
configure: failed program was:
| /* confdefs.h */
| #define PACKAGE_NAME "Protocol Buffers"
| #define PACKAGE_TARNAME "protobuf"
| #define PACKAGE_VERSION "3.4.0"
| #define PACKAGE_STRING "Protocol Buffers 3.4.0"
| #define PACKAGE_BUGREPORT "protobuf@googlegroups.com"
| #define PACKAGE_URL ""
| #define PACKAGE "protobuf"
| #define VERSION "3.4.0"
| #define HAVE_SYS_TYPES_H 1
| #define HAVE_SYS_STAT_H 1
| #define HAVE_STDLIB_H 1
| #define HAVE_STRING_H 1
| #define HAVE_MEMORY_H 1
| #define HAVE_STRINGS_H 1
| #define HAVE_INTTYPES_H 1
| #define HAVE_STDINT_H 1
| #define HAVE_UNISTD_H 1
| #define __EXTENSIONS__ 1
| #define _ALL_SOURCE 1
| #define _GNU_SOURCE 1
| #define _POSIX_PTHREAD_SEMANTICS 1
| #define _TANDEM_SOURCE 1
| #define HAVE_DLFCN_H 1
| #define LT_OBJDIR ".libs/"
| #define HAVE_FCNTL_H 1
| #define HAVE_INTTYPES_H 1
| #define HAVE_LIMITS_H 1
| #define HAVE_STDLIB_H 1
| #define HAVE_UNISTD_H 1
| #define HAVE_PTHREAD 1
| #define HAVE_HASH_MAP 1
| #define HAVE_HASH_SET 1
| #define HASH_MAP_H <unordered_map>
| #define HASH_SET_H <unordered_set>
| #define HASH_NAMESPACE std
| #define HASH_MAP_CLASS unordered_map
| #define HASH_SET_CLASS unordered_set
| /* end confdefs.h.  */
| 
| /* Override any GCC internal prototype to avoid an error.
|    Use char because int might match the return type of a GCC
|    builtin and then its argument prototype would still apply.  */
| #ifdef __cplusplus
| extern "C"
| #endif
| char sched_yield ();
| int
| main ()
| {
| return sched_yield ();
|   ;
|   return 0;
| }
configure:19873: result: no
configure:19880: error: in `/opt/conda/conda-bld/vg_1519118731750/work/deps/protobuf':
configure:19882: error: sched_yield was not found on your system
See `config.log' for more details

@mbargull
Copy link
Copy Markdown
Member

Hm, conda-forge's recipe for the same version doesn't mention OpenMP, though: https://github.com/conda-forge/protobuf-feedstock/tree/4a47177307485d1ce97cd1af7fb35ccdf6297468/recipe
¯\_(ツ)_/¯

@mbargull
Copy link
Copy Markdown
Member

conda-forge's recipe made me curious: looks like they just use the system gcc:
https://github.com/conda-forge/protobuf-feedstock/blob/4a47177307485d1ce97cd1af7fb35ccdf6297468/recipe/build.sh#L15-L16
https://github.com/conda-forge/protobuf-feedstock/blob/4a47177307485d1ce97cd1af7fb35ccdf6297468/recipe/meta.yaml#L24-L36
(double-checked with the built package's info/recipe/meta.yaml: no gcc was included in the rendered build section). So maybe this means with the Docker image's devtoolset-2/gcc we always have OpenMP?


Why would this be interesting? => If were to change some recipes to explicitly use gcc or {{ compiler("c") }}/{{ compiler("cxx") }}, we now know we have to be on the lookout for ld: cannot find -lgomp errors!

@mbargull
Copy link
Copy Markdown
Member

Yay, that worked perfectly.
Well, but unsurprisingly another of the plethora of vendored libraries failed to compile:

configure: creating ./config.status
config.status: creating Makefile
config.status: creating src/gperftools/tcmalloc.h
config.status: creating src/windows/gperftools/tcmalloc.h
config.status: creating src/config.h
config.status: executing depfiles commands
config.status: executing libtool commands
configure: WARNING: No frame pointers and no libunwind. Using experimental backtrace capturing via libgcc
make[1]: Entering directory `/opt/conda/conda-bld/vg_1519120737998/work/deps/gperftools'
/bin/sh ./libtool  --tag=CXX   --mode=compile /opt/conda/conda-bld/vg_1519120737998/_b_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/bin/x86_64-conda_cos6-linux-gnu-c++ -DHAVE_CONFIG_H -I. -I./src  -I./src  -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -DNO_TCMALLOC_SAMPLES -pthread -DNDEBUG -Wall -Wwrite-strings -Woverloaded-virtual -Wno-sign-compare -fno-builtin-malloc -fno-builtin-free -fno-builtin-realloc -fno-builtin-calloc -fno-builtin-cfree -fno-builtin-memalign -fno-builtin-posix_memalign -fno-builtin-valloc -fno-builtin-pvalloc  -Wno-unused-result -fsized-deallocation  -DNO_FRAME_POINTER -O3 -fopenmp -std=c++11 -ggdb -g -MMD -MP -fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -I/opt/conda/conda-bld/vg_1519120737998/_b_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/include -fdebug-prefix-map=src=/usr/local/src/conda/vg-1.6.0 -fdebug-prefix-map=/opt/conda/conda-bld/vg_1519120737998/_b_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_=/usr/local/src/conda-prefix -MT src/libtcmalloc_minimal_la-tcmalloc.lo -MD -MP -MF src/.deps/libtcmalloc_minimal_la-tcmalloc.Tpo -c -o src/libtcmalloc_minimal_la-tcmalloc.lo `test -f 'src/tcmalloc.cc' || echo './'`src/tcmalloc.cc
libtool: compile:  /opt/conda/conda-bld/vg_1519120737998/_b_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/bin/x86_64-conda_cos6-linux-gnu-c++ -DHAVE_CONFIG_H -I. -I./src -I./src -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -DNO_TCMALLOC_SAMPLES -pthread -DNDEBUG -Wall -Wwrite-strings -Woverloaded-virtual -Wno-sign-compare -fno-builtin-malloc -fno-builtin-free -fno-builtin-realloc -fno-builtin-calloc -fno-builtin-cfree -fno-builtin-memalign -fno-builtin-posix_memalign -fno-builtin-valloc -fno-builtin-pvalloc -Wno-unused-result -fsized-deallocation -DNO_FRAME_POINTER -O3 -fopenmp -std=c++11 -ggdb -g -MMD -MP -fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -I/opt/conda/conda-bld/vg_1519120737998/_b_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/include -fdebug-prefix-map=src=/usr/local/src/conda/vg-1.6.0 -fdebug-prefix-map=/opt/conda/conda-bld/vg_1519120737998/_b_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_=/usr/local/src/conda-prefix -MT src/libtcmalloc_minimal_la-tcmalloc.lo -MD -MP -MF src/.deps/libtcmalloc_minimal_la-tcmalloc.Tpo -c src/tcmalloc.cc  -fPIC -DPIC -o src/.libs/libtcmalloc_minimal_la-tcmalloc.o
In file included from src/libc_override_glibc.h:62:0,
                 from src/libc_override.h:79,
                 from src/tcmalloc.cc:156:
src/libc_override_gcc_and_weak.h:60:33: error: ISO C++1z does not allow dynamic exception specifications
 void* operator new(size_t size) throw (std::bad_alloc)
                                 ^~~~~
src/libc_override_gcc_and_weak.h:64:35: error: ISO C++1z does not allow dynamic exception specifications
 void* operator new[](size_t size) throw (std::bad_alloc)
                                   ^~~~~
make[1]: *** [src/libtcmalloc_minimal_la-tcmalloc.lo] Error 1
make[1]: Leaving directory `/opt/conda/conda-bld/vg_1519120737998/work/deps/gperftools'
make: *** [lib/libtcmalloc_minimal.a] Error 2

See ClickHouse/ClickHouse#1002 (comment) for fixes.

@mbargull
Copy link
Copy Markdown
Member

So regarding that gperftools compile error, I guess your options are

  • "de-vendor" gperftools,
  • make sure -std=c++11 or -std=c++14 (whatever is applicable) but not -std=c++17 is used,
  • try to replace the vendored one with gperftools >=2.6.1,
  • patch the vendored one.

@tseemann
Copy link
Copy Markdown
Contributor

tseemann commented Oct 3, 2019

Any suggestions @ekg ?

@bgruening
Copy link
Copy Markdown
Member

@druvus
Copy link
Copy Markdown
Member

druvus commented Oct 21, 2020

Added in another PR

@druvus druvus closed this Oct 21, 2020
@druvus druvus deleted the vg branch October 21, 2020 21:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants