Hi, I am working on link prediction task using edgedataloding for minibatch training, how to calculate MRR and Hits after I get pos_score and neg_score?
MRR and Hit rate are ranking-based metrics that measures how well the relevant items are retrieved among a set of items. So for each source node, you will have a set of target nodes to predict scores and rank them to compute MRR and Hit rate.
You can probably just use torchmetrics.RetrievalMRR and torchmetrics.RetrievalRecall (which is the same as hit rate). See Module metrics — PyTorch-Metrics 0.5.0 documentation for usages.
thank you!
In link prediction task, we iterater over the edges in a graph, sample negtive samples for each positive edge, there is no ranking accroding to source nodes right? only one positive target and n negtive targets, so the Hits is not suitable for this situation? only hit@1 suits.
Or for a source node a, there are N true targets, and we sample M false target nodes for this source node, then we can rank the score of all the N+M edges, and count the positive edges in top k.
This one is correct. MRR (shorthand for Mean Reciprocal Rank) and Hit Rate are information retrieval metrics. Typically you start with a query and a set of candidate items, and you pick a few from those candidate items, ranking from the most relevant to less relevant as model predicted. This translates to what you said above when it comes to link prediction.
Also, even if the number of positive targets is one, you can still do say HITS@10 (or equivalently Recall@10), because the number 10 refers to how many candidates you pick. As long as the relevant items are in the candidates you picked then you are good. For a list of information retrieval metrics and their meaning you can take a look at Evaluation measures (information retrieval) - Wikipedia.
，Thank you so much!!!
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.