Partition graph with deprecated reshuffle = False

Questions
#1

Hi there, I noticed that the comments of function partition_graph in python/dgl/distributed/partition.py say that:
“293 If reshuffle=False, node IDs and edge IDs of a partition do not fall into contiguous
294 ID ranges. In this case, DGL stores node/edge mappings (from
295 node/edge IDs to partition IDs) in separate files (node_map.npy and edge_map.npy).
296 The node/edge mappings are stored in numpy files.
298 … warning::
299 this format is deprecated and will not be supported by the next release.”

In our graph, keeping the original node/edge ids as opposed to maintaining a mapping between the local contiguous ids and global ids would reduce much code change, so can I use the deprecated flag (reshuffle=False) for the partitioning? BTW, what were the reasons behind the scenes to introduce local contiguous ids by additionally maintaining a map? Thanks

Save heterogeneous graphs into distributed DGL
#2

Are the below statements still true? Say in DistGraph, “For heterogeneous graphs, users need to convert them into DGL graphs with one node type and one edge type” ?

Note
    ----
    ``DistGraph`` currently only supports graphs with only one node type and one edge type.
    For heterogeneous graphs, users need to convert them into DGL graphs with one node type and
    one edge type and store the actual node types and edge types as node data and edge data.
#3

One more question from me was posted in here, thanks!

Save heterogeneous graphs into distributed DGL
#4

I don’t think reshuffle=False can still be used in the latest version. The whole reason that we want to shuffle node/edge IDs is to avoid maintaining vectors of N elements (N is the number of nodes/edges in the global graph) in each trainer process that map between node/edge IDs and the partition IDs as well as between node/edge IDs and the local node/edge IDs. This will be memory consuming for a large graph.

#5

can you show me where the statement is? in the latest version, DistGraph supports heterogeneous graph. We need to update it.

#6

Here you go: dgl/dist_graph.py at master · dmlc/dgl · GitHub
There are also asserts as follows, can you explain? Thanks
assert len(self.ntypes) == 1, “ndata only works for a graph with one node type.”
dgl/dist_graph.py at master · dmlc/dgl · GitHub
dgl/dist_graph.py at master · dmlc/dgl · GitHub
dgl/dist_graph.py at master · dmlc/dgl · GitHub

#7

Can the above asserts be removed if not valid anymore? @zhengda1936 Thanks

#9

Is this the page that explains how DistGraph now supports heterogeneous graphs, or does it need to be updated too?:

https://docs.dgl.ai/en/0.6.x/guide/distributed-hetero.html#guide-distributed-hetero

It seems very similar to the heterogeneous-to-homogeneous conversion workaround explained in the note. Either way, it looks like the end result is a homogeneous graph. Is that true? Or is there a way to do distributed learning without converting to homogeneous?

#10

Thanks for pointing me to the out-of-date messages.
The asserts are still valid because ndata and edata can only be used on homogeneous graphs. DGLGraph has the same assert.

#11

For distributed training, we still use homogeneous graph formats to store a heterogeneous graph. however, the Python class DistGraph now supports the APIs designed for heterogeneous graphs. For example, you now can access node data of a particular type as follows:
g.nodes['node_type'].data['h']. this is useful if the heterogeneous graph has different node data and edge data. Previous version doesn’t support.

#12

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.