Hi all,
Just wanted to know, are you are aware of any model for heterogeneous graphs that implements attention linearly? I have been using HGT and linear transformers have been around for a while, but I couldnt find any implementation fro graphs.
thanks!
Hi,
What’s linear attention? Could you post some references here?
Hi @VoVAllen, I am sorry I didnt answer before. I missed the notification.
Sure! Are you familiar with attention mechanisms of transformers? Attention has a quadratic time/memory, because it compares every node with every node.
Linear attention tries to replicate the same mechanism, but with linear time/memory. This paper is relatively old, but it compares many transformers with different kinds of attentions. If you want to see their implementations, you can find most in lucidrains git.
You can find an interesting discussion here
Thanks. I haven’t seen any paper about mixing linear attention with graph yet. It seems not hard to move it to graph?
Probably not…definitely a nice paper for whoever does it.
Google released yesterday a new linear transformer called [H-transformer](https://arxiv.org/pdf/2107.11906.pdf.
Check the results:
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.
