How to encode the associated image of each node?

[meettyj]

Suppose each node in the graph is associate with an image, how can we encode the image during training? instead of encoding them when building the graph? I am thinking of something like this:

class model(nn.Module):
    def __init__(self):
        self.cnn = cnn()
        self.gcn = SAGEConv()

    def forward(self, nodeID)
        image = get_image(nodeID)
        image_feature = self.cnn(image)
        node_embedding = self.gcn(graph, image_feature)

Is this doable? If so, how should I define the data loader to generate the nodeID? Can I use the NodeDataLoader/EdgeDataLoader that dgl provides?

Thank you in advance!

[Rhett-Ying]

graph and related feature is usually passed into def forward(self,...) just like the example here: 5.1 Node Classification/Regression — DGL 0.8 documentation. Why not encode the images for each node in advance and attach into graph.ndata['x']? Encoding is required to be changing and trained with CNN?

NodeDataLoader is used with sampler for mini-batch training. I don’t think you need it in your case necessarily.

As for the proposal you mentioned, I think you could generate nodeIDs via g.nodes() and treat it as node features, attach to g.ndata['x']. when training, just pass such feature into model like the example I pasted. In the body of forward, you could encode images via the nodeID feature and pass such encoded feature into gcn. pls note the feature shape in forward: NN Modules (PyTorch) — DGL 0.8 documentation.

[meettyj]

Thank you for your suggestions. This is really helpful! Yes the reason why I want to encode the image during training instead of attaching them to graph.ndata[‘x’], is because I want the parameters of CNN to be trained end to end. I am assuming if I attach the images into graph.ndata[‘x’], then the CNN cannot be trained simultaneously right? Thanks again.