How to encode the associated image of each node?

Suppose each node in the graph is associate with an image, how can we encode the image during training? instead of encoding them when building the graph? I am thinking of something like this:

class model(nn.Module):
    def __init__(self):
        self.cnn = cnn()
        self.gcn = SAGEConv()

    def forward(self, nodeID)
        image = get_image(nodeID)
        image_feature = self.cnn(image)
        node_embedding = self.gcn(graph, image_feature)

Is this doable? If so, how should I define the data loader to generate the nodeID? Can I use the NodeDataLoader/EdgeDataLoader that dgl provides?

Thank you in advance!

graph and related feature is usually passed into def forward(self,...) just like the example here: 5.1 Node Classification/Regression — DGL 0.8 documentation. Why not encode the images for each node in advance and attach into graph.ndata['x']? Encoding is required to be changing and trained with CNN?

NodeDataLoader is used with sampler for mini-batch training. I don’t think you need it in your case necessarily.

As for the proposal you mentioned, I think you could generate nodeIDs via g.nodes() and treat it as node features, attach to g.ndata['x']. when training, just pass such feature into model like the example I pasted. In the body of forward, you could encode images via the nodeID feature and pass such encoded feature into gcn. pls note the feature shape in forward: NN Modules (PyTorch) — DGL 0.8 documentation.

Thank you for your suggestions. This is really helpful! Yes the reason why I want to encode the image during training instead of attaching them to graph.ndata[‘x’], is because I want the parameters of CNN to be trained end to end. I am assuming if I attach the images into graph.ndata[‘x’], then the CNN cannot be trained simultaneously right? Thanks again.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.