Fetching halo node's features using RPC

[sark777]

Hi,

This question is similar to Calculate bytes sent and received with communication time per machine in distributed training, but I didn’t find a straightforward answer.

1. What is the best way to measure the time spent on rpc per minibatch per trainer?

2. As far as my current understanding goes, g.ndata["features"][input_nodes] also pulls features of the local nodes so some time will also be spent on retrieving the local nodes’ features from the local KVstore. Is this overlapped with rpc?

3. How do I measure how many rpc calls were made per minibatch per trainer?

4. In this post When and how to fetch features on the remote machine, you mention -

In short, local ids will be converted to global ids and send request to target machines

However, doesn’t the input_nodes returned by the dataloader for step, (input_nodes, seeds, blocks) in enumerate(dataloader) already represent global_ids?

Thank you for your time!

[minjie]

1. I don’t think there is straightforward way to do that besides digging into the codebase and time the remote and local operations separately.
2. Yes, it is overlapped with rpc.
3. Similar answer to 1.
4. Yes, the ID returned by enumerator is always global IDs. The local IDs are internal implementation details.