Using third party library depends on Thrust in DGL

[yofufufufu]

Hello everyone! I want to use a third party library called stdgpu in DGL to implement some custom function in CUDA/C++ for experimental purpose.
So I clone the repository to the dgl/third_party directory and add

add_subdirectory(third_party/stdgpu)
list(APPEND DGL_LINKER_LIBS stdgpu::stdgpu)

in CMakeList.
However, this library depends on c++17 and Thrust. When I just simply include the library like:

#include <stdgpu/queue.cuh>

and try to rebuild DGL from source, there will be a lot of errors…
So I changed c++14 to c++17 in CMakeList and CUDA.cmake, for example in CMakeLists:

# change to c++17
# set(CMAKE_CXX_FLAGS "-O2 -Wall -fPIC -std=c++14 ${CMAKE_CXX_FLAGS}")
set(CMAKE_CXX_FLAGS "-O2 -Wall -fPIC -std=c++17 ${CMAKE_CXX_FLAGS}")

# change to c++17
# set_property(TARGET dgl PROPERTY CXX_STANDARD 14)
set_property(TARGET dgl PROPERTY CXX_STANDARD 17)

and in CUDA.cmake:

# change to c++17
#  list(APPEND CUDA_NVCC_FLAGS "-std=c++14")
list(APPEND CUDA_NVCC_FLAGS "-std=c++17")

Now, when I rebuild DGL from source, the number of errors has decreased and the main error include:

namespace stdgpu
{
template <typename T1, typename T2>
# error: identifier "thrust" is undefined
using pair = thrust::pair<T1, T2>;

// ... ...
}

I guess the reason is in CMakeList:

  # see https://github.com/NVIDIA/thrust/issues/1401
  add_definitions(-DTHRUST_CUB_WRAPPED_NAMESPACE=dgl)
  include(cmake/modules/CUDA.cmake)
  message(STATUS "Use external CUB/Thrust library for a consistent API and performance.")
  cuda_include_directories(BEFORE "${CMAKE_SOURCE_DIR}/third_party/thrust")
  cuda_include_directories(BEFORE "${CMAKE_SOURCE_DIR}/third_party/thrust/dependencies/cub")

Because if I chage the code above like:

namespace dgl{
namespace stdgpu
{
template <typename T1, typename T2>
# error: identifier "thrust" is undefined
using pair = thrust::pair<T1, T2>;

// ... ...
}
}

The error of this file is resolved. But the library is strongly depends on Thrust, so it is hard for me to do such kind of change everywhere the library uses Thrust…
What should I do so I can use this library without change too much? Thanks in advance!

[czkkkkkk]

@yofufufufu , did you try to remove add_definitions(-DTHRUST_CUB_WRAPPED_NAMESPACE=dgl)?

[yofufufufu]

Thanks for your reply!
Yes，but it is not allowed：

dmlc/dgl/blob/7b1639f1aacb006fa65ef8cef09c49f5219bd5c1/src/array/cuda/dgl_cub.cuh#L10-L13

Any other suggestion？

[czkkkkkk]

I see. I feel it is hard to address this problem if you compile dgl and stdgpu together. Can you compile you application and dgl independently? You may first build dgl and link you application against the shared library of DGL.

[yofufufufu]

Sorry, I could not understand what you mean . I want to implement some custom functions based on DGL FFI, how can I compile independently?

[czkkkkkk]

Oh, FFI is a problem. It is hard to compile them independently if you are utilizing DGL FFI. You can try a tricky solution by removing thrust dependency in DGL. I found thrust is only used in src/array/cuda/spmat_op_impl_csr.cu and src/array/cuda/labor_sampling.cu. You can comment out these two files and the previous static_assert. Then you can freely remove THRUST_CUB_WRAPPED_NAMESPACE.

[yofufufufu]

If commemt out those files, files that depends on function implemented in those files will not work, then error occur.

[czkkkkkk]

To make the code pass compilation, you may need some minor changes. For example, int a = thrust::foo(b); can be replaced by int a;.

[yofufufufu]

Thanks for your reply!
I add

#undef THRUST_CUB_WRAPPED_NAMESPACE

to one stdgpu header file(called utility.h), and then comment out the static_assert. Surprisingly, the code pass compilation.
What really confuse me is that I only need to add undef to one stdgpu header file(called utility.h), not all header files that depends on thrust .
Just now, I commemt out the static_assert and the add_definitions(-DTHRUST_CUB_WRAPPED_NAMESPACE=dgl) in CMakeList without changing anything else. The code pass compilation again, makes me more confused…
In conclusion, now I can use the third party library(stdgpu). But I think there should be a better way to solve this problem.

[czkkkkkk]

Glad to see it works.

Yes, a hacky solution is totally not elegant. A better way is to build DGL library (libdgl.so) independently and link you application to DGL. Then you don’t bother to think about how DGL link to thrust. You may check DGL sparse (dgl_sparse) to see how it works.