ModuleNotFoundError in DGL with PyTorch 2.2 on Python 3.12 Docker Image

Questions
#4

No, I have a different problem, this is the error:

################################################################################
WARNING!
The 'datapipes', 'dataloader2' modules are deprecated and will be removed in a
future torchdata release! Please see https://github.com/pytorch/data/issues/1196
to learn more and leave feedback.
################################################################################

  deprecation_warning()
Traceback (most recent call last):
  File "/var/task/evaluate_model.py", line 10, in <module>
    import dgl
  File "/var/task/dgl/__init__.py", line 16, in <module>
    from . import (
  File "/var/task/dgl/dataloading/__init__.py", line 13, in <module>
    from .dataloader import *
  File "/var/task/dgl/dataloading/dataloader.py", line 27, in <module>
    from ..distributed import DistGraph
  File "/var/task/dgl/distributed/__init__.py", line 5, in <module>
    from .dist_graph import DistGraph, DistGraphServer, edge_split, node_split
  File "/var/task/dgl/distributed/dist_graph.py", line 11, in <module>
    from .. import backend as F, graphbolt as gb, heterograph_index
  File "/var/task/dgl/graphbolt/__init__.py", line 55, in <module>
    load_graphbolt()
  File "/var/task/dgl/graphbolt/__init__.py", line 45, in load_graphbolt
    raise FileNotFoundError(
FileNotFoundError: Cannot find DGL C++ graphbolt library at /var/task/dgl/graphbolt/libgraphbolt_pytorch_2.3.0.so

Looking at the libraries I can see that on the image is installed graphbolt up to pythorch 2.2, that’s the reason why I was using an earlier version.

#5

How did you install DGL when you got this error?

If you have the CPU version of torch 2.3, then you need to use this link to install DGL: https://data.dgl.ai/wheels/torch-2.3/repo.html

#6

What about GPU version? Thanks.

#7

Both of the lines below are only for torch 2.3 versions.

pip install dgl -f https://data.dgl.ai/wheels/torch-2.3/cu121/repo.html

For CUDA 12.1 and

pip install dgl -f https://data.dgl.ai/wheels/torch-2.3/cu118/repo.html

for CUDA 11.8.

#8

Thank you, then I got exactly the same error as the previous user posted.

#9

I can further report that, after install, import dgl gets the following error:
ImportError: Cannot load Graphbolt C++ library
File /local_disk0/.ephemeral_nfs/envs/pythonEnv-242fa303-4a84-4d3b-aec6-3a2e19f10b8c/lib/python3.10/site-packages/dgl/graphbolt/init.py:31, in load_graphbolt()
30 try:
—> 31 torch.classes.load_library(path)
32 except Exception: # pylint: disable=W0703

I have checked the torch and cuda version should match the installation command you recommended:
%pip install torch==2.3.0
%pip install dgl -f https://data.dgl.ai/wheels/torch-2.3/cu121/repo.html

Any advise? Thank you.

#10

are sure you’re isntalling cuda version of torch? could you try to install with pip install torch==2.3.0 torchvision==0.18.0 torchaudio==2.3.0 --index-url https://download.pytorch.org/whl/cu121 and check with torch.cuda.is_available()?

#11

After running:
%pip install torch==2.3.0 torchvision==0.18.0 torchaudio==2.3.0 --index-url https://download.pytorch.org/whl/cu121
%pip install dgl -f https://data.dgl.ai/wheels/torch-2.3/cu121/repo.html
torch.cuda.is_available() still returns false somehow…

#12

I just ran pip install -r requirements.txt, using the same requirements I posted originally but replacing 2.3 for pytorch and DGL

#13

DGL 2.4 is out and it has support for torch 2.4.1. Would you like to try out this combination?

#14

sure, if not too much trouble, what is the command to install dgl 2.4 if we use torch 2.4.1 and cuda 12.1?

#15

CUDA: pip install dgl -f https://data.dgl.ai/wheels/torch-2.4/cu124/repo.html

CPU: pip install dgl -f https://data.dgl.ai/wheels/torch-2.4/repo.html

#16

Thank you, could we use cu121 instead?

#17

Yes. If you go to the DGL webpage to the getting started section, you can pick cuda version, torch version and conda/pip etc.

https://www.dgl.ai/pages/start.html

#18

Thank you. I installed dgl 2.4 and torch 2.4 But

import dgl

still gives error:
FileNotFoundError: Cannot find DGL C++ graphbolt library at /local_disk0/.ephemeral_nfs/envs/pythonEnv-525aecaa-dec3-4609-a10d-dbd14eef7889/lib/python3.10/site-packages/dgl/graphbolt/libgraphbolt_pytorch_2.4.0.so

#19

I downloaded the wheel and the file is there:

image
image1187×581 43.2 KB

#20

I recommend going to the installation directory and verifying if the .so files can be found there.

#21

unfortunately I’m using databrick where I cannot access the installed directory, hence it might not be possible to check if the file is there or not.

#22

For future reference, if someone is using databrick system, dgl works when downgrade to torch 2.1.0+cuda 11.8.

1 Like
#23

I managed to resolve my issue. Basically (and I don’t know why) if we just use the requirements.txt to tell pip the packages to install it installs the wrong versions. Here is my final Dockerfile:

FROM public.ecr.aws/lambda/python:3.12
ARG PRIVATE_PIP_INDEX_URL
ARG PROD_REGION
ARG PROD_ACCESS_KEY
ARG PROD_SECRET_ACCESS_KEY
ENV DGLBACKEND=pytorch

# Install the dependencies
COPY requirements.txt .
RUN pip install torch==2.3.1 \
    --index-url https://download.pytorch.org/whl/cpu

RUN pip install dgl -f https://data.dgl.ai/wheels/torch-2.3/repo.html
RUN pip3 install \
    --index-url  https://pypi.org/simple \
    --extra-index-url "${PRIVATE_PIP_INDEX_URL}" \
    -r requirements.txt --target "${LAMBDA_TASK_ROOT}"

# Copy our code
COPY *.py ${LAMBDA_TASK_ROOT}


# Set the handler function
CMD [ "evaluate_model.handler" ]%
1 Like