Failed to partition FraudAmazonDataset with dgl.transform.metis_partition

When partitioning the train subgraph of FraudAmazonDataset (g_train below), dgl.transform.metis_partition terminates abnormally and prompts “*** Error in `python’: free(): corrupted unsorted chunks: 0x000055d7ef0ebf20 ***”.

Other datasets are successfully partitioned with metis_paritition(). Then I use metis to partition the same graph (g_train), it works well.
It seems the program aborted in metis_partition_assignment (line 270: converting the graph to biparted graph)

The testing code is listed as follow:

import dgl
import torch
from dgl.data.fraud import FraudDataset
from dgl.transform import metis_partition
import metis
import networkx as nx


def dgl_metis():
    # load dataset: FraudAmazon
    data = FraudDataset('amazon', raw_dir='./data')
    g_raw = data[0]
    train_nid = torch.nonzero(g_raw.ndata['train_mask'], as_tuple=True)[0]
    # convert to homogeneous graph
    g_homo = dgl.to_homogeneous(g_raw)
    # build train graph
    g_train = g_homo.subgraph(train_nid)
    parts = metis_partition(g_train, 10)  # "free(): corrupted unsorted chunks"
    print(parts)


def metis_lib_test():
    # load dataset: FraudAmazon
    data = FraudDataset('amazon', raw_dir='./data')
    g_raw = data[0]
    train_nid = torch.nonzero(g_raw.ndata['train_mask'], as_tuple=True)[0]
    # convert to homogeneous graph
    g_homo = dgl.to_homogeneous(g_raw)
    # build train graph
    g_train = g_homo.subgraph(train_nid)
    adj_mat = g_train.adj(scipy_fmt='csr')
    G = nx.from_scipy_sparse_matrix(adj_mat)
    edge_cuts, parts = metis.part_graph(G, 10)
    print(edge_cuts)    # 1178265



dgl_metis()
metis_lib_test()

I found that the homo_graph in Amazon.mat has 8,796,784 edges while g_homo has 9,557,648 edges (sum of all three relationships). I suppose redundant edges cause partition failure.

U-P-U: 351,216
U-S-U: 7,132,958
U-V-U: 2,073,474

    data = FraudDataset('amazon', raw_dir='./fraud')
    g = data[0]
    train_nid = torch.nonzero(g.ndata['train_mask'], as_tuple=True)[0]
    amazon = loadmat('../data/Amazon.mat')
    amazon_homo = amazon['homo']
    new_g = dgl.from_scipy(amazon_homo)
    new_in_degrees = new_g.in_degrees().numpy()
    tr_g = new_g.subgraph(train_nid)
    pnids = dgl.transform.metis_partition(tr_g, k=10)
    print(pnids)

Bug confirmed. Need some time to investigate

1 Like

Yes. I found this is due to the duplicate edges in the graph. If you add g_train=dgl.to_simple(g_train), which eliminates the duplicate edges, then it works well

I created issue at Metis partition fails if graph has duplicate edges · Issue #3224 · dmlc/dgl · GitHub

Thanks a lot for your response and verification. :+1:

The problem was discovered by accident because other graphs have no duplicate edges after call dgl.to_homogeneous. For example, FraudYelpDataset has 7,693,958 edges in homo_graph and the sum of all three relations is also 7,693,958.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.