Implemented Caching Schemes in Distributed Training

Questions
#1

Hello! For context I am working in the multi-CPU and multi-GPU setting for large graphs in which node features are partitioned across machines and mini-batches sent to GPUs for compute. I wanted to follow up on the second question in this post from a year ago about whether node features that have not been partitioned onto a machine can be additionally cached for a period of time, or are node features always disjoint across machines? The answer a year ago was no and I have not found anything in the code/roadmaps/docs saying that has changed or will change but would like to confirm. And perhaps more generally, if the system provides caching at any level that has a notable effect on performance (besides the k-hop neighborhoods of the partitioned graph structure on each machine which I believe is sometimes referred to as extra_cached_hops) it would be great if I could be pointed to some documentation or quick set of high level notes that I could reference while reading the codebase.

Thank you!

#2

Unfortunately, the answer is still NO.

#3

BTW, do you want to cache other info/data besides data like features of current graph’s nodes and neighbor nodes? namely, something extra_cached_hops cannot achieve?

#4

I was looking to cache features of nodes that had not been originally partitioned onto the machine. I believe extra_cached_hops only caches the graph structure, not features?

#5

Yes, you’re right. Caching features is an interesting feature and we need more time to discuss internally on how to achieve this. Could you help open a feature request in DGL’s github so that we could track this?

#6

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.