Hi, when I used MultiLayerFullNeighborSampler to train MoNet with built-in GMMConv, I encountered following errors:
Traceback (most recent call last):
File "/home/Wuyinwei/Desktop/Distill_GCN_Experiment/test_monet.py", line 110, in
File "/home/Wuyinwei/anaconda3/envs/ldgl/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/Wuyinwei/Desktop/Distill_GCN_Experiment/test_monet.py", line 45, in forward
us, vs = blocks[i].edges(order='eid')
File "/home/Wuyinwei/anaconda3/envs/ldgl/lib/python3.8/site-packages/dgl/heterograph.py", line 3435, in in_degrees
deg = self._graph.in_degrees(etid, v_tensor)
File "/home/Wuyinwei/anaconda3/envs/ldgl/lib/python3.8/site-packages/dgl/heterograph_index.py", line 669, in in_degrees
_CAPI_DGLHeteroInDegrees(self, int(etype), F.to_dgl_nd(v))
File "dgl/_ffi/_cython/./function.pxi", line 295, in dgl._ffi._cy3.core.FunctionBase.**call**
File "dgl/_ffi/_cython/./function.pxi", line 227, in dgl._ffi._cy3.core.FuncCall
File "dgl/_ffi/_cython/./function.pxi", line 217, in dgl._ffi._cy3.core.FuncCall3
dgl._ffi.base.DGLError: [19:26:06] /opt/dgl/src/array/cuda/spmat_op_impl_csr.cu:163: Check failed: e == cudaSuccess || e == cudaErrorCudartUnloading: CUDA kernel launch error: an illegal memory access was encountered
Stack trace:
[bt] (0) /home/Wuyinwei/anaconda3/envs/ldgl/lib/python3.8/site-packages/dgl/libdgl.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x4f) [0x7fd85ef1b30f]
[bt] (1) /home/Wuyinwei/anaconda3/envs/ldgl/lib/python3.8/site-packages/dgl/libdgl.so(dgl::runtime::NDArray dgl::aten::impl::CSRGetRowNNZ<(DGLDeviceType)2, long>(dgl::aten::CSRMatrix, dgl::runtime::NDArray)+0x1f2) [0x7fd85fd4c122]
[bt] (2) /home/Wuyinwei/anaconda3/envs/ldgl/lib/python3.8/site-packages/dgl/libdgl.so(dgl::aten::CSRGetRowNNZ(dgl::aten::CSRMatrix, dgl::runtime::NDArray)+0x368) [0x7fd85ef001e8]
[bt] (3) /home/Wuyinwei/anaconda3/envs/ldgl/lib/python3.8/site-packages/dgl/libdgl.so(dgl::UnitGraph::CSR::OutDegrees(unsigned long, dgl::runtime::NDArray) const+0xb4) [0x7fd85f3a9a84]
[bt] (4) /home/Wuyinwei/anaconda3/envs/ldgl/lib/python3.8/site-packages/dgl/libdgl.so(dgl::UnitGraph::InDegrees(unsigned long, dgl::runtime::NDArray) const+0xe2) [0x7fd85f3a23a2]
[bt] (5) /home/Wuyinwei/anaconda3/envs/ldgl/lib/python3.8/site-packages/dgl/libdgl.so(dgl::HeteroGraph::InDegrees(unsigned long, dgl::runtime::NDArray) const+0x46) [0x7fd85f29f246]
[bt] (6) /home/Wuyinwei/anaconda3/envs/ldgl/lib/python3.8/site-packages/dgl/libdgl.so(+0x735309) [0x7fd85f2a9309]
[bt] (7) /home/Wuyinwei/anaconda3/envs/ldgl/lib/python3.8/site-packages/dgl/libdgl.so(DGLFuncCall+0x48) [0x7fd85f22ee58]
[bt] (8) /home/Wuyinwei/anaconda3/envs/ldgl/lib/python3.8/site-packages/dgl/_ffi/_cy3/core.cpython-38-x86_64-linux-gnu.so(+0x163fc) [0x7fd97ba043fc]
I have raised an issue in GITHUB but got no further response. Could anyone help me with that?
To Reproduce
The code I ran is at:
Env
- DGL Version:
dgl 1.0.2+cu116
dglgo 0.0.2 - Backend Library & Version:
torch 1.12.1+cu116
torch-geometric 2.1.0 - OS:
linux ubuntu 20 - How you installed DGL:
pip install dgl -f https://data.dgl.ai/wheels/cu116/repo.html
pip install dglgo -f https://data.dgl.ai/wheels-test/repo.html - Python version:
3.8 - CUDA/cuDNN version (if applicable):
11.3 - GPU models and configuration:
RTX3090 - Any other relevant information: