How to define accuracy function in R-GCN?

Questions
#1

Hi,
I am working on edge prediction task and for that I am using r-gcn model. I have few questions related to r-gcn model :

What is MRR (raw) , Hits (raw ) in r-gcn?
How to get accuracy score in r-gcn during training and during evaluating?

Thank you !

#2

Hi,

They are common metrics used for link prediction. For example, from distmult paper section 4:

Mean Reciprocal Rank (MRR) (an average of the reciprocal rank of an answered entity over all test triplets), HITS@10 (top-10 accuracy)

The function here will calculate both raw MRR and HITS. And you can follow this line to calculate the metrics.

#3

Hi,
Thanks for the reply. I am training the network on one dataset but Hits@10 not increasing it’s going down , results looks something like this :

Epoch 0200 | Loss 0.3042 | Best MRR 0.0260 | Forward 10.5518s | Backward 15.6663s

start eval

batch 0 / 13

batch 1 / 13

batch 2 / 13

…

MRR (raw): 0.007128

Hits (raw) @ 1: 0.000000

Hits (raw) @ 3: 0.000000

Hits (raw) @ 10: 0.000786

training done

Mean forward time: 137.697712s

Mean Backward time: 26.457217s

Is it bad or what’s the reason?

#4

@lingfan Any idea on that?

#5

@Abhinav43, are you running dgl’s example code? And you only showed results for epoch 200 without comparison to earlier epochs. So I can’t see that MRR or Hits are getting worse.

I reran the RGCN link prediction code for 100 epochs and at epoch 50, I got:
MRR (raw): 0.024565
Hits (raw) @ 1: 0.009559
Hits (raw) @ 3: 0.021594
Hits (raw) @ 10: 0.050031

And at epoch 100, I got:
MRR (raw): 0.094404
Hits (raw) @ 1: 0.050375
Hits (raw) @ 3: 0.093219
Hits (raw) @ 10: 0.177051

So, all four metrics are getting better.