Unsupervised GraphSAGE loss not being minimised

[davidireland3]

I am trying to implement the loss function found in the GraphSAGE paper (equation 1). Here is my code I am using:

def loss_fn(graph, embeddings, length, node_ids, size_neg_graph):
    # get positive scores first
    pos_ids = dgl.sampling.random_walk(graph, node_ids, length=length)[0][:, length - 1] # gets the node ids of nodes that occur on RW 
    pos_score = torch.sum(embeddings * embeddings[pos_ids], dim=1) # performs inner product
    pos_score = torch.nn.functional.logsigmoid(pos_score) # log sigmoid

    # get negative scores
    neg_graph = construct_negative_graph(graph, size_neg_graph) # constructs a negative graph by corrupting destination nodes by sampling randomly from all nodes in graph
    neg_score = dp(neg_graph, embeddings) # dot product function to compute dot prod along all edges of the embeddings
    neg_score = torch.nn.functional.logsigmoid(-neg_score) 

    # return loss
    loss = -torch.sum(pos_score) - torch.sum(neg_score)
    return loss / len(node_ids)

When I run this, the loss never minimises. It always gets stuck at a relatively high number. Is there something wrong with my code?

The DP function and Negative graph sampling function are defined as follows:

class DotProductPredictor(nn.Module):
    def forward(self, graph, h):
        # h contains the node representations computed from the GNN
        with graph.local_scope():
            graph.ndata['h'] = h
            graph.apply_edges(fn.u_dot_v('h', 'h', 'score'))
            return graph.edata['score']

def construct_negative_graph(graph, k):
    src, dst = graph.edges()

    neg_src = src.repeat_interleave(k)
    neg_dst = torch.randint(0, graph.number_of_nodes(), (len(src) * k,))
    return dgl.graph((neg_src, neg_dst), num_nodes=graph.number_of_nodes())

[minjie]

Hi, the code looks correct to me. For the loss issue, it is usually very case-specific so I can only provide some high-level suggestions:

• Try tune the hyperparameters (e.g., learning rate, batch size, walk length in your case). If you set walk length to one, it should become a normal loss function defined on edges.
• Pay attention to the parameter initializer. Sometimes a different initializer will work much better.
• Try print out the gradient of each layer to check whether there are any gradient explosion problem.
• Try debug with simple models first. For example, if the encoder is only an MLP rather than a GNN, does the problem still exist?

You could search for similar questions on the forum to see whether they give you the answer.

Help! Cannot get GCN model to decrease

I was training on a self-collected dataset using GCN model. Specifically, I constructed many small graphs batched together during training, with batch_size=100. For each node, I extracted Resnet 2048-dim feature, the goal is to do binary classification for each graph. Loss: BCELoss GCN model: Connection: bidirectional, fully connected edges for each graph Model: 3 layers: 2048->128->1, all are GCN layer MLP model: For verification, I trained MLP (2048->128->128->1, ReLU in between) on th…

Problem to reproduce the result in DGL gcn_concat

That means you should use DGL instead . I haven’t got time to dig into your model, but would like to share some of my experiences in matching the performance: Verify data preprocessing. Make sure that the data preprocess is exactly the same as the one in DGL. For example, make sure the features are correctly normalized, the self-loops are added and the train/val/test split is the same. According to this paper, GCN is actually not very stable when the split changes so make sure t…

[davidireland3]

Thanks! I’m glad I was able to get the message passing right.

I figured my loss might not actually be that bad. As the loss function is essentially Q * the expectation of the negative score, my loss always hang around a number lower than Q, so its actually not too bad. For instance, I was training with 10 negative samples and my loss hangs around 6-7, which isn’t too bad at all as it means that the expected value of the negative scores is less than 1 (because the positive scores end up contribution 0 to the loss).

[darpitdave]

Hi @davidireland3 and @minjie : I am trying to compute embeddings for my graph in an unsupervised approach. I am trying to use the loss function here. I have a concern for computing the DorProductPredictor(). Unlike for positive score, why do we do the dot product with itself because as in equation it should be between z_u and z_v. Is it specific to the edge prediction problem or this loss function above can be used for extracting node-level embeddings. I have a homogenous graph.

[minjie]

@darpitdave

why do we do the dot product with itself because as in equation it should be between z_u and z_v.

If you are referring to u_dot_v('h', 'h', 'score'), then it is actually doing what you are suggesting. The API name u_dot_v means it will get the first feature from source nodes (‘u’) and the second feature from destination nodes (‘v’). Since they are both ‘h’, it conducts exactly h_u dot-product h_v.