How does DGL deal with out-of-memory graphs in general

Questions
#1

Hi,

I’m trying to understand how DGL deals with out-of-memory graphs. edges file and features file are usually the ones that may cause the problem. For small graphs like products, I see everything is read and stored in memory beforehand (like before the training starts). However, for larger graphs, I am wondering if there is any specific way that DGL uses to avoid OOM. Specifically, does DGL support features that don’t fit in memory? How about edges?
(I saw in other posts uva is mentioned but it’s like sharing DRAM and GPU memory?) Thanks in advance :slight_smile:

#2

You can look at the OnDiskDataset class introduced in DGL 2.0: Composing OnDiskDataset from raw data — DGL 2.0.0 documentation. It supports loading features from disk.

Currently we do not support out-of-core graphs. We have plans to support it in the future.

1 Like
#3

Thank you for the pointer.

It shows “For homogeneous graph, we just need to save edges(namely node pairs) into CSV file”. Does that mean DGL only supports CSV files when reading edges?

#4

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.