Always the same predictions

Questions
#1

Hi. I’m trying to build a recsys with a R-GCN.

My dataset looks like this:

Graph(num_nodes={'post': 11911, 'user': 9482},
      num_edges={('post', 'liked', 'user'): 363984, ('user', 'like', 'post'): 363984},
      metagraph=[('post', 'user', 'liked'), ('user', 'post', 'like')])

I add the reverse link so there aren’t nodes with no incoming edges. Since the graph is not too big, I dont use batches. I use cosine similarity during training and testing. Training looks good. Loss dicreases properly and for every epoch I evaluate the model on 10% of those edges (not seen during trainig), comparing positive edges and negative ones, and achieve almost +0.6 mean cosine score for positve examples. I train just on one direction: (post, liked, user).

After that, I want to use that model for inference. I pass the graph trough the model to update node embddings and I use 10 random users and generate top 20 most similar posts to them. The issue is that predicted posts looks almost the same for all the users, so I must be doing something wrong. Where could be the problem? My only clue is the direction of the training edge, but I don’t know.

Thanks.

#2

Usually for link prediction, people used MRR or Hits@N as the metric. Could you try these metrics? Not only top1 matters in link prediction tasks. Also for the post predicted, I would suggest checking the degrees of it.

#3

I will.

In order to prefer lower degree nodes?

#4

Hi, @VoVAllen , do I need to hardcode it or are they implemented anywhere?

Thanks.

#5

You need to implement those metric functions yourself. You may find some implementations to borrow from DGL or OGB’s github repos.

1 Like
#6

Another question. Could using another reduce operation during traing help me with this inference problem? I’m using ‘mean’ right now. Thanks.

#7

that predicted posts looks almost the same for all the users

Have you checked the cosine similarity scores? Are they also the same?

#8

Yes. Given, let’s say two different posts. its top k users has this cosines (user IDs are the same but cannot share them, sorry):

[(tensor(0.0883, device='cuda:0'), 
(tensor(0.0628, device='cuda:0'),
(tensor(0.0441, device='cuda:0'), 
(tensor(0.0319, device='cuda:0'), 
(tensor(0.0017, device='cuda:0'), 
(tensor(-0.0043, device='cuda:0'), 
(tensor(-0.0099, device='cuda:0'), 
(tensor(-0.0111, device='cuda:0'), 
(tensor(-0.0176, device='cuda:0'),
tensor(-0.0185, device='cuda:0'), 
(tensor(-0.0190, device='cuda:0'),

and

[(tensor(0.0883, device='cuda:0'), 
(tensor(0.0628, device='cuda:0'),
(tensor(0.0441, device='cuda:0'), 
(tensor(0.0319, device='cuda:0'), 
(tensor(0.0017, device='cuda:0'), 
(tensor(-0.0043, device='cuda:0'), 
(tensor(-0.0099, device='cuda:0'), 
(tensor(-0.0111, device='cuda:0'), 
(tensor(-0.0176, device='cuda:0'),
tensor(-0.0185, device='cuda:0'), 
(tensor(-0.0190, device='cuda:0'),

I’m afraid this just happens with isolated nodes. Nodes with no incoming edges end up with almost similar embeddings among them. Aren’t those nodes suppose to maintain its own embeddings, since they don’t get any message? Why could the look alike after going through the model?

#9

Have you added self-loops or residual connections for these isolated nodes?

#10

Both during training and inference?

Since I have this two edges: ('post', 'liked', 'user'), ('user', 'like', 'post') , how should I have those for isolated nodes? An edge with a random node of the other type?

#11

Then, you can simply use residual connections by modifying your GNN module.

#12

Sorry, but how could I do that?

#13

You can follow the implementation in built-in modules like this.

1 Like
#14

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.