I am working on a project where I try to implement RL with observations that are encoded as graphs. In my original RL algorithm I fill a rollout-buffer with observations (tensor shaped), which is then used for training a neural network.
I now want to move to a graph neural network. My rollout-buffer should again be filled with observations - which are now graphs with different topologies, nodes and features - to again be used for training over a minibatch. However, I am struggling with finding an efficient way to store these observations. Maybe some of you might have some ideas that could help me!
Two ideas/options I could maybe consider:
- Store the observations as tensors and somehow make a batched graph of these batched tensors before applying my GNN?
- First make the graph encodings from the original observation and storing the actual graphs in the rollout-buffer and then batching them before applying my GNN? Would this be efficient to store the actual graphs?
Any help, ideas or suggestions would be greatly appreciated!