GPU build fails for lightgbm target

[StrikerRUS]

Error message

[ 95%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/tree_learner.cpp.o
[ 96%] Linking CXX executable ../lightgbm
CMakeFiles/lightgbm.dir/src/treelearner/gpu_tree_learner.cpp.o: In function `boost::compute::command_queue::command_queue(boost::compute::context const&, boost::compute::device const&, unsigned long)':
/home/travis/build/Microsoft/LightGBM/src/treelearner/gpu_tree_learner.cpp:(.text._ZN5boost7compute13command_queueC2ERKNS0_7contextERKNS0_6deviceEm[_ZN5boost7compute13command_queueC2ERKNS0_7contextERKNS0_6deviceEm]+0xee): undefined reference to `clCreateCommandQueueWithProperties'
CMakeFiles/lightgbm.dir/src/treelearner/gpu_tree_learner.cpp.o: In function `boost::compute::kernel::set_arg_svm_ptr(unsigned long, void*)':
/home/travis/build/Microsoft/LightGBM/src/treelearner/gpu_tree_learner.cpp:(.text._ZN5boost7compute6kernel15set_arg_svm_ptrEmPv[_ZN5boost7compute6kernel15set_arg_svm_ptrEmPv]+0xb): undefined reference to `clSetKernelArgSVMPointer'
clang-5.0: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [../lightgbm] Error 1
make[1]: *** [CMakeFiles/lightgbm.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[ 98%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/voting_parallel_tree_learner.cpp.o
[100%] Linking CXX shared library ../lib_lightgbm.so
[100%] Built target _lightgbm
make: *** [all] Error 2

Full logs:

• https://travis-ci.org/Microsoft/LightGBM/jobs/523007683,
• https://dev.azure.com/lightgbm-ci/lightgbm-ci/_build/results?buildId=1859&view=logs&jobId=9ce42e5d-f31a-544a-a6de-dc42f8c013ed&taskId=b696def5-78ae-590c-4234-04332342ba2d&lineStart=281&lineEnd=282&colStart=1&colEnd=1.

Reproducible examples

Remove _lightgbm at this line:

https://github.com/Microsoft/LightGBM/blob/4be53a5a1f18892692138dfd3085190124bc1a3e/.ci/test.sh#L127

If we try to build all targets (_lightgbm and lightgbm), then GPU build fails with the same error as in #458 and #1047.

@huanzhang12 Do you have any ideas why it happens?

[StrikerRUS]

Gently ping @huanzhang12

[StrikerRUS]

With 25d149d cannot reproduce at Travis anymore. Still the problem at Azure Pipelines:

2020-02-19T18:58:20.1589111Z -- The C compiler identification is GNU 4.8.4
2020-02-19T18:58:20.3498936Z -- The CXX compiler identification is GNU 4.8.4
2020-02-19T18:58:20.3620194Z -- Check for working C compiler: /usr/bin/cc
2020-02-19T18:58:20.4144976Z -- Check for working C compiler: /usr/bin/cc -- works
2020-02-19T18:58:20.4175518Z -- Detecting C compiler ABI info
2020-02-19T18:58:20.4632491Z -- Detecting C compiler ABI info - done
2020-02-19T18:58:20.4780074Z -- Detecting C compile features
2020-02-19T18:58:20.4789457Z -- Detecting C compile features - done
2020-02-19T18:58:20.4852644Z -- Check for working CXX compiler: /usr/bin/c++
2020-02-19T18:58:20.5419638Z -- Check for working CXX compiler: /usr/bin/c++ -- works
2020-02-19T18:58:20.5437637Z -- Detecting CXX compiler ABI info
2020-02-19T18:58:20.6024449Z -- Detecting CXX compiler ABI info - done
2020-02-19T18:58:20.6172078Z -- Detecting CXX compile features
2020-02-19T18:58:20.6798086Z -- Detecting CXX compile features - done
2020-02-19T18:58:20.8772960Z -- Found OpenMP_C: -fopenmp (found version "3.1") 
2020-02-19T18:58:20.9378296Z -- Found OpenMP_CXX: -fopenmp (found version "3.1") 
2020-02-19T18:58:20.9386117Z -- Found OpenMP: TRUE (found version "3.1")  
2020-02-19T18:58:20.9415414Z -- Looking for CL_VERSION_2_2
2020-02-19T18:58:20.9893714Z -- Looking for CL_VERSION_2_2 - not found
2020-02-19T18:58:20.9899455Z -- Looking for CL_VERSION_2_1
2020-02-19T18:58:21.0339676Z -- Looking for CL_VERSION_2_1 - not found
2020-02-19T18:58:21.0351164Z -- Looking for CL_VERSION_2_0
2020-02-19T18:58:21.1026495Z -- Looking for CL_VERSION_2_0 - found
2020-02-19T18:58:21.1091822Z -- Found OpenCL: /usr/lib/x86_64-linux-gnu/libOpenCL.so (found version "2.0") 
2020-02-19T18:58:21.1105772Z -- OpenCL include directory: /__w/1/s/AMDAPPSDK/include
2020-02-19T18:58:21.1467548Z -- Found Boost: /usr/include (found suitable version "1.70.0", minimum required is "1.56.0") found components: filesystem system 
2020-02-19T18:58:21.1490507Z -- Performing Test MM_PREFETCH
2020-02-19T18:58:21.2314822Z -- Performing Test MM_PREFETCH - Success
2020-02-19T18:58:21.2320066Z -- Using _mm_prefetch
2020-02-19T18:58:21.2324721Z -- Performing Test MM_MALLOC
2020-02-19T18:58:21.3051095Z -- Performing Test MM_MALLOC - Success
2020-02-19T18:58:21.3055710Z -- Using _mm_malloc
2020-02-19T18:58:21.3074704Z -- Configuring done
2020-02-19T18:58:21.3280877Z -- Generating done
2020-02-19T18:58:21.3293530Z -- Build files have been written to: /__w/1/s/build
2020-02-19T18:58:21.3876808Z Scanning dependencies of target lightgbm
2020-02-19T18:58:21.3900774Z Scanning dependencies of target _lightgbm
2020-02-19T18:58:21.3972862Z [  1%] Building CXX object CMakeFiles/lightgbm.dir/src/main.cpp.o
2020-02-19T18:58:21.3997117Z [  3%] Building CXX object CMakeFiles/lightgbm.dir/src/boosting/boosting.cpp.o
2020-02-19T18:58:21.4010946Z [  4%] Building CXX object CMakeFiles/lightgbm.dir/src/application/application.cpp.o
2020-02-19T18:58:21.4033660Z [  6%] Building CXX object CMakeFiles/_lightgbm.dir/src/c_api.cpp.o
2020-02-19T18:58:23.4142324Z [  7%] Building CXX object CMakeFiles/_lightgbm.dir/src/lightgbm_R.cpp.o
2020-02-19T18:58:27.2551631Z [  9%] Building CXX object CMakeFiles/_lightgbm.dir/src/application/application.cpp.o
2020-02-19T18:58:30.0863224Z [ 11%] Building CXX object CMakeFiles/lightgbm.dir/src/boosting/gbdt.cpp.o
2020-02-19T18:58:31.2869716Z [ 12%] Building CXX object CMakeFiles/lightgbm.dir/src/boosting/gbdt_model_text.cpp.o
2020-02-19T18:58:36.1021548Z [ 14%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/boosting.cpp.o
2020-02-19T18:58:37.0184486Z [ 15%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/gbdt.cpp.o
2020-02-19T18:58:38.5579545Z [ 17%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/gbdt_model_text.cpp.o
2020-02-19T18:58:38.9526760Z [ 19%] Building CXX object CMakeFiles/lightgbm.dir/src/boosting/gbdt_prediction.cpp.o
2020-02-19T18:58:42.0806517Z [ 20%] Building CXX object CMakeFiles/lightgbm.dir/src/boosting/prediction_early_stop.cpp.o
2020-02-19T18:58:43.2301369Z [ 22%] Building CXX object CMakeFiles/lightgbm.dir/src/io/bin.cpp.o
2020-02-19T18:58:44.6182562Z [ 23%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/gbdt_prediction.cpp.o
2020-02-19T18:58:45.5712001Z [ 25%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/prediction_early_stop.cpp.o
2020-02-19T18:58:46.2470518Z [ 26%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/bin.cpp.o
2020-02-19T18:58:46.6550588Z [ 28%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/config.cpp.o
2020-02-19T18:58:47.7589285Z [ 30%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/config_auto.cpp.o
2020-02-19T18:58:53.2809895Z [ 31%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/dataset.cpp.o
2020-02-19T18:58:53.9951478Z [ 33%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/dataset_loader.cpp.o
2020-02-19T18:59:04.0509723Z [ 34%] Building CXX object CMakeFiles/lightgbm.dir/src/io/config.cpp.o
2020-02-19T18:59:04.0904073Z [ 36%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/file_io.cpp.o
2020-02-19T18:59:04.7379051Z [ 38%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/json11.cpp.o
2020-02-19T18:59:05.2942680Z [ 39%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/metadata.cpp.o
2020-02-19T18:59:06.9869227Z [ 41%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/parser.cpp.o
2020-02-19T18:59:09.5598575Z [ 42%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/tree.cpp.o
2020-02-19T18:59:10.1512752Z [ 44%] Building CXX object CMakeFiles/lightgbm.dir/src/io/config_auto.cpp.o
2020-02-19T18:59:11.3061582Z [ 46%] Building CXX object CMakeFiles/_lightgbm.dir/src/metric/dcg_calculator.cpp.o
2020-02-19T18:59:11.3622544Z [ 47%] Building CXX object CMakeFiles/_lightgbm.dir/src/metric/metric.cpp.o
2020-02-19T18:59:15.2781241Z [ 49%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/linker_topo.cpp.o
2020-02-19T18:59:16.1252497Z [ 50%] Building CXX object CMakeFiles/lightgbm.dir/src/io/dataset.cpp.o
2020-02-19T18:59:16.5431815Z [ 52%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/linkers_mpi.cpp.o
2020-02-19T18:59:16.6103455Z [ 53%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/linkers_socket.cpp.o
2020-02-19T18:59:17.5022447Z [ 55%] Building CXX object CMakeFiles/_lightgbm.dir/src/network/network.cpp.o
2020-02-19T18:59:21.1010597Z [ 57%] Building CXX object CMakeFiles/_lightgbm.dir/src/objective/objective_function.cpp.o
2020-02-19T18:59:21.9783186Z [ 58%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/data_parallel_tree_learner.cpp.o
2020-02-19T18:59:27.0542374Z [ 60%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/feature_parallel_tree_learner.cpp.o
2020-02-19T18:59:27.4383586Z [ 61%] Building CXX object CMakeFiles/lightgbm.dir/src/io/dataset_loader.cpp.o
2020-02-19T18:59:35.9462300Z [ 63%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/gpu_tree_learner.cpp.o
2020-02-19T18:59:37.6992433Z [ 65%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/serial_tree_learner.cpp.o
2020-02-19T18:59:38.4583154Z [ 66%] Building CXX object CMakeFiles/lightgbm.dir/src/io/file_io.cpp.o
2020-02-19T18:59:39.6067666Z [ 68%] Building CXX object CMakeFiles/lightgbm.dir/src/io/json11.cpp.o
2020-02-19T18:59:40.4432994Z [ 69%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/tree_learner.cpp.o
2020-02-19T18:59:44.5588354Z [ 71%] Building CXX object CMakeFiles/lightgbm.dir/src/io/metadata.cpp.o
2020-02-19T18:59:49.3076751Z [ 73%] Building CXX object CMakeFiles/_lightgbm.dir/src/treelearner/voting_parallel_tree_learner.cpp.o
2020-02-19T18:59:50.7749811Z [ 74%] Building CXX object CMakeFiles/lightgbm.dir/src/io/parser.cpp.o
2020-02-19T18:59:53.2702724Z [ 76%] Building CXX object CMakeFiles/lightgbm.dir/src/io/tree.cpp.o
2020-02-19T18:59:55.2271130Z [ 77%] Building CXX object CMakeFiles/lightgbm.dir/src/metric/dcg_calculator.cpp.o
2020-02-19T18:59:59.1462297Z [ 79%] Building CXX object CMakeFiles/lightgbm.dir/src/metric/metric.cpp.o
2020-02-19T18:59:59.8861161Z [ 80%] Building CXX object CMakeFiles/lightgbm.dir/src/network/linker_topo.cpp.o
2020-02-19T19:00:00.0863227Z [ 82%] Building CXX object CMakeFiles/lightgbm.dir/src/network/linkers_mpi.cpp.o
2020-02-19T19:00:00.1541975Z [ 84%] Building CXX object CMakeFiles/lightgbm.dir/src/network/linkers_socket.cpp.o
2020-02-19T19:00:01.9193131Z [ 85%] Building CXX object CMakeFiles/lightgbm.dir/src/network/network.cpp.o
2020-02-19T19:00:05.5262348Z [ 87%] Building CXX object CMakeFiles/lightgbm.dir/src/objective/objective_function.cpp.o
2020-02-19T19:00:05.6350398Z [ 88%] Building CXX object CMakeFiles/lightgbm.dir/src/treelearner/data_parallel_tree_learner.cpp.o
2020-02-19T19:00:09.7744165Z [ 90%] Linking CXX shared library ../lib_lightgbm.so
2020-02-19T19:00:10.2257709Z [ 90%] Built target _lightgbm
2020-02-19T19:00:10.2424289Z [ 92%] Building CXX object CMakeFiles/lightgbm.dir/src/treelearner/feature_parallel_tree_learner.cpp.o
2020-02-19T19:00:14.9223100Z [ 93%] Building CXX object CMakeFiles/lightgbm.dir/src/treelearner/gpu_tree_learner.cpp.o
2020-02-19T19:00:20.6221279Z [ 95%] Building CXX object CMakeFiles/lightgbm.dir/src/treelearner/serial_tree_learner.cpp.o
2020-02-19T19:00:21.0822342Z [ 96%] Building CXX object CMakeFiles/lightgbm.dir/src/treelearner/tree_learner.cpp.o
2020-02-19T19:00:24.1582373Z [ 98%] Building CXX object CMakeFiles/lightgbm.dir/src/treelearner/voting_parallel_tree_learner.cpp.o
2020-02-19T19:00:38.6584628Z [100%] Linking CXX executable ../lightgbm
2020-02-19T19:00:38.8299334Z CMakeFiles/lightgbm.dir/src/treelearner/gpu_tree_learner.cpp.o: In function `LightGBM::GPUTreeLearner::InitGPU(int, int)':
2020-02-19T19:00:38.8301269Z gpu_tree_learner.cpp:(.text+0xa56b): undefined reference to `clCreateCommandQueueWithProperties'
2020-02-19T19:00:38.8302549Z CMakeFiles/lightgbm.dir/src/treelearner/gpu_tree_learner.cpp.o: In function `boost::compute::detail::meta_kernel::compile(boost::compute::context const&, std::string const&)':
2020-02-19T19:00:38.8304023Z gpu_tree_learner.cpp:(.text._ZN5boost7compute6detail11meta_kernel7compileERKNS0_7contextERKSs[_ZN5boost7compute6detail11meta_kernel7compileERKNS0_7contextERKSs]+0x4d5): undefined reference to `clSetKernelArgSVMPointer'
2020-02-19T19:00:38.8305716Z collect2: error: ld returned 1 exit status
2020-02-19T19:00:38.8309097Z make[2]: *** [../lightgbm] Error 1
2020-02-19T19:00:38.8313105Z make[1]: *** [CMakeFiles/lightgbm.dir/all] Error 2
2020-02-19T19:00:38.8317359Z make: *** [all] Error 2

https://dev.azure.com/lightgbm-ci/lightgbm-ci/_build/results?buildId=4747&view=logs&j=9ce42e5d-f31a-544a-a6de-dc42f8c013ed&t=b696def5-78ae-590c-4234-04332342ba2d