If the whole graph is being loaded on the GPU at the beginning of the 1st epoch why do I see H2D happening before every step while going through dataloader, i.e., for every batch of input I see H2D. Can someone explain to me the reason behind this?
Which example were you running?
If you are training GraphSAGE in sample-based setting (instead of full-graph training), then H2D happening before each step is reasonable. Currently we only support graph sampling on CPU, and the sampled graph need to be copied to GPU via a CudaMemcpy from host to device.
1 Like
Can you elaborate, what is sample based setting and full graph training?
Full-graph training is essentially performing message passing on the entire graph. Such graph together with node/edge features can reside on GPU so that there is no H2D at all.
Sample-based training by contrast works on graphs too big to fit into the memory. Essentially, it copies from CPU to GPU only the features of the nodes/edges necessary for computing the predictions of the given minibatch. Because computing labels of a small number of nodes with GraphSAGE with neighbor sampling requires input features of far more number of nodes, we will need much more H2D to copy the input features from CPU to GPU compared to, say training images with CNN.
1 Like