Convert ParMetis partition to dgl graph

Questions
#1

I’m following the instructions to convert the ParMetis format to dgl graph but I fail on the third steps: dgl/examples/pytorch/rgcn/experimental at master · dmlc/dgl · GitHub

  1. dgl/tools/convert_partition.py This file doesn’t exist. dgl/tools/distpartitioning/convert_partition.py is a module. Is this the right file to call: dgl/parmetis_postprocess.py at master · dmlc/dgl · GitHub?

  2. I also notice that the parmetis_wrapper calls pm_dglpart3 but all the other documentation calls pm_dglpart dgl/parmetis_wrapper.py at f635e2a11eb4d74618594fc99aadbc3b3d3f4111 · dmlc/dgl · GitHub Which one should we call?

#2

Hi,
The README is outdated as we’re working on a new distributed partition pipeline which involves the modules and programs we were using before such as pm_dglpart, convert_partition. We’ll update the doc page once it’s fully ready.

Does python3 partition_graph.py --dataset ogbn-mag --num_parts 4 --balance_train --balance_edges work for you?

#3

For mag, the partition_graph.py works for me. I’m looking for example/instruction for using ParMetis. I’m trying to rerun the experiment of this paper that describes the new design of DGL: Distributed Hybrid CPU and GPU training for Graph Neural Networks on Billion-Scale Heterogeneous Graphs.

It seems the rgcn is the only one with step by step examples for ParMetis. I want to partition ogb-paper100M. It’s in the partition_graph.py of the graphsage example. I get OOM when I run the script.

#4

As I mentioned here: ogb-papers100M requires 400+GB in RAM.

I think one workaround is checking out to 0.8.x: dgl/tools at 0.8.x · dmlc/dgl · GitHub which is supposed to work well as the README in rgcn example. After you successfully obtaining the partitioned graphs, you could turn to latest DGL for train.

#5

Thanks for the link. I can try the pipeline in the master.

#6

By the way, has the design mentioned in the paper merged in the latest DGL repo?

#7

DistDGLv2 has not been implemented in DGL yet.

#8

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.