Cosine Embedding Loss

[ogggcar]

Hi.

I’m trying to build a link prediction model using Cosine Embedding Loss from Torch.
I split original graph (350.000 edges) 90% edges for training and 10% for testing.

My model looks like this:

  neg_g = negative_sampler(train_g, negative_samples, etype)
  user_feats = train_g.nodes['user'].data['feats'].to(device)
  post_feats = train_g.nodes['post'].data['feats'].to(device)
  node_features = {'user': user_feats, 'post': post_feats}
  model = model.to(device)
  train_g = train_g.to(device)
  neg_g = neg_g.to(device)
  feats = model(train_g, node_features)
  loss = cosine_loss(train_g, neg_g, feats, etype, device)
  opt.zero_grad()
  loss.backward()
  opt.step()

Cosine loss function does the following. First I add start nodes (‘posts’) in out1 and the end nodes (‘users’) in out2, both for positve (g) and negatie (neg_g) examples:

criterion = nn.CosineEmbeddingLoss()

out1 = []
out2 = []

src, dst = g.edges(etype=etype)
  
src_list = src.tolist()
dst_list = dst.tolist()

for i, node in enumerate(src_list):
    out1.append(feats['post'][node])
    out2.append(feats['user'][dst_list[i]])

y = torch.tensor([1]*len(dst_list))
  
neg_src, neg_dst = neg_g.edges(etype=etype)
neg_src_list = neg_src.tolist()
neg_dst_list = neg_dst.tolist()

ceros = torch.tensor([-0.5]*len(neg_dst_list))

for i, node in enumerate(neg_src_list):
    out1.append(feats['post'][node])
    out2.append(feats['user'][neg_dst_list[i]])

y = torch.cat((y, ceros), 0)
y = y.to(device)

out1= torch.stack(out1)
out2 = torch.stack(out2)
loss = criterion(out1, out2, y)
return loss

Results look good: `

Epoch 1 Train loss: 0.482245 Test eval similarity: 0.773184
Epoch 2 Train loss: 0.112204 Test eval similarity: 0.909741
Epoch 3 Train loss: 0.044195 Test eval similarity: 0.944503
Epoch 4 Train loss: 0.026975 Test eval similarity: 0.959659
Epoch 5 Train loss: 0.019582 Test eval similarity: 0.968107

That eval similarity is the mean of the cosine similarity of real edges. in a test graph (same nodes,but 10% of edges). Problema is that, when printing that same score for the negative graph, that mean similarity is algo high and tends to reach the positive edges similarity after some epochs.

Epoch 1: Positive edge mean: (tensor(0.7760), Negative edges mean: tensor(0.7740))
Epoch 2: Positive edge mean: (tensor(0.9035), Negative edges mean:tensor(0.9022))
Epoch 3: Positive edge mean: (tensor(0.9453), Negative edges mean:tensor(0.9444))

Any idea of why negative edges have that high similarity, since loss seems to be working and it takes negative examples into account? Could the problem be my negative sampler?

This does’t happen if a use margin loss. In those cases positive similarity isn’t that high, but its way higher than the negative one (it reaches +0.5)

Thanks.