How to create Massive Heterograph in DGL?

Hi all, I have some credit card transaction data that I’m trying to create a Heterograph out of. This heterograph will have 3 kinds of nodes (cardholders, merchants and transactions) each having 80-dimensional node feature . The eventual goal is to train a multi-GPU R-GCN on this Heteorograph to do node classification. My graph has 1B+ nodes and 10B+ edges so I’m wondering what would be the best way to create the Heterograph and then perform multi-GPU R-GCN training. I know there’s Heterograph function in DGL but it’s data_dict arg requires Pytorch tensors loaded into memory of 1 machine right? That doesn’t seem feasible for the size of my graph. Any workarounds or alternative ways of creating a large heterograph of this scale?

Hi,

Could you save the graph without features by in-memory process? Possibly you may need to save the features separately now. We’ll consider support this feature in the future

do you like to use multiple machines or just a single machine with multiple GPUs?

For a start I’d like to be able to use NVIDIA DGX A100 which has eight A100 GPUs (either 40GB or 80GB) but because of the graph size it may require multiple DGX A100s (multi-node)

i don’t understand. it seems you try to avoid using one machine in your first post. but you like to start with one machine?

Ideally I’d like to be able to create the graph on 2 (or more) DGX A100s and then be able to train a GNN on 16 (or more ) A100 GPUs.

Hi can you please give an example of how I would do that? Currently, this is how I create the Heterograph

data_dict = {('cardholder', 'transaction', 'merchant') : (card_nodes_th, merch_nodes_th)}
g = dgl.heterograph(data_dict)

Here card_nodes_th and merch_nodes_th are Pytorch tensors for my cardholder and merchant nodes and I have to load them all from disk into CPU memory.

And then I add cardholder node features like this:

g.nodes['cardholder'].data['features'] = card_feats_th 

where card_feats_th is the cardholder feats pytorch tensor