About GAT accuracy on Cora

Hi, I noticed that the accuracy of GAT implemented with DGL on Cora dataset is higher than the original implementation. Could you please share what differences resulted in this increased accuracy? I searched a bit in this forum and did not find answers to my question.

Hi,
The Cora/Citeseer/Pubmed dataset are very small, the performance of GNN models on these datasets are not stable and sensitive to hyperparameters like random seed. So I don’t think you need to worry about the performance difference, it might just because of a better random seed.

If you are talking about the result of just result from one round of training, random seed could be an explanation. However in the DGL paper (https://rlgm.github.io/papers/49.pdf) the result on Cora is 83.98 ± 0.52, which is higher than the original paper’s 83±0.7. If the mean of the accuracy distribution is higher, I don’t think random seed is a valid explanation.

Sorry I did not notice that you were mentioning the result in our workshop paper.

A possible explanation is that: we did not use early stopping in our GAT example when the paper was written, you may find that the result we reported on citeseer and pubmed is lower than reported in GAT paper (https://github.com/dmlc/dgl/issues/743).

Later on we adopted the early stopping algorithm (https://github.com/dmlc/dgl/pull/750), it improves the average accuracy on citeseer and pubmed but harms the accuracy on cora: the result on cora with latest GAT codebase is 82.81(±0.68) in 10 runs.