Implementing non-negative defense in GCN graph classification to train a robust model?

Questions
#1

I wanted to implement the idea in this paper in my GCN model :

the general idea of paper is this :

As the name suggests, Non-Negative MalConv was constrained during training to have non-negative weight matrices. The point of doing this is to prevent trivial attacks like those created against MalConv. When done properly, the non-negative weights make binary classifiers monotonic; meaning that the addition of new content can only increase the malicious score. This would make evading the model very difficult, because most evasion attacks do require adding content to the file. Fortunately for me, this implementation of Non-Negative MalConv has a subtle but critical flaw.

Right now my model is very similar to this : https://docs.dgl.ai/en/0.4.x/tutorials/basics/4_batch.html

so how can i implement this idea in this code? basically i want my GCN to learn to only decide based on malicious features in nodes, and adding benign features in a node (i don’t have node labels, by benign i mean features that adding a lot of them will cause a model to converge to labeling the graph as benign) will not cause a model to give higher probability for the graph of being benign, and only adding malicious features will cause the graph classification probability to change

for example if each node has 50 features, and in the vanila GCN, the adding to the value of second feature will cause the graph classification to converge to being benign, in this non-negative GCN adding this feature will not affect the model.

any idea how can i implement this in GCN? is it possible?

#2

One possibility is to follow SGC. Basically, it removes all nonlinear activation functions between GCN layers. As a result, you can implement the model as follows:

  1. Pre-compute message passing A^{k}X, where A is the adjacency matrix and X is input node features
  2. Pass the result of A^{k}X to a logistic regression model.

To implement non-negative weight, you can simply use the non-negative variant of logistic regression.

1 Like
#3

Thank you, so what is the most efficient way of changing the SGConv code of dgl to do this? sorry I’m new to pytorch and I’m not that familiar with the functions pytorch provides to make this easier.

class SGConv(nn.Module):

    def __init__(self,
                 in_feats,
                 out_feats,
                 k=1,
                 cached=False,
                 bias=True,
                 norm=None):
        super(SGConv, self).__init__()
        self.fc = nn.Linear(in_feats, out_feats, bias=bias)
        self._cached = cached
        self._cached_h = None
        self._k = k
        self.norm = norm
        self.reset_parameters()

    def reset_parameters(self):
        nn.init.xavier_uniform_(self.fc.weight)
        if self.fc.bias is not None:
            nn.init.zeros_(self.fc.bias)

    def forward(self, graph, feat):

        graph = graph.local_var()
        if self._cached_h is not None:
            feat = self._cached_h
        else:
            # compute normalization
            degs = graph.in_degrees().float().clamp(min=1)
            norm = th.pow(degs, -0.5)
            norm = norm.to(feat.device).unsqueeze(1)
            # compute (D^-1 A^k D)^k X
            for _ in range(self._k):
                feat = feat * norm
                graph.ndata['h'] = feat
                graph.update_all(fn.copy_u('h', 'm'),
                                 fn.sum('m', 'h'))
                feat = graph.ndata.pop('h')
                feat = feat * norm

            if self.norm is not None:
                feat = self.norm(feat)

            # cache feature
            if self._cached:
                self._cached_h = feat
        return self.fc(feat)
#4

You just need to remove the part related to self.fc and pass the output of SGConv to a non-negative logistic regression.

1 Like
#5

Sorry for rookie question but how can i implement “non negative” logistic regression in pytorch?

all the tutorial for pytorch’s logistic regression i found implemented something like this :

class LogisticRegression(torch.nn.Module):
    def __init__(self, input_dim, output_dim):
        super(LogisticRegression, self).__init__()
        self.linear = torch.nn.Linear(input_dim, output_dim)

    def forward(self, x):
        outputs = self.linear(x)
        return outputs

but i couldn’t find anything regarding a non negative version of logistic regression?

also what do you mean regarding removing the parts related to self.fc? do you mean i should remove any usage of self.fc in the SGconv class like this ? :

    class SGConv(nn.Module):

        def __init__(self,
                     in_feats,
                     out_feats,
                     k=1,
                     cached=False,
                     bias=True,
                     norm=None):
            super(SGConv, self).__init__()
            self._cached = cached
            self._cached_h = None
            self._k = k
            self.norm = norm
            self.reset_parameters()

        def reset_parameters(self):
            pass

        def forward(self, graph, feat):

            graph = graph.local_var()
            if self._cached_h is not None:
                feat = self._cached_h
            else:
                # compute normalization
                degs = graph.in_degrees().float().clamp(min=1)
                norm = th.pow(degs, -0.5)
                norm = norm.to(feat.device).unsqueeze(1)
                # compute (D^-1 A^k D)^k X
                for _ in range(self._k):
                    feat = feat * norm
                    graph.ndata['h'] = feat
                    graph.update_all(fn.copy_u('h', 'm'),
                                     fn.sum('m', 'h'))
                    feat = graph.ndata.pop('h')
                    feat = feat * norm

                if self.norm is not None:
                    feat = self.norm(feat)

                # cache feature
                if self._cached:
                    self._cached_h = feat
            return feat
#6

Also it seems like in keras it is possible to constraint the weights to be positive using

tf.keras.constraints.NonNeg()

but i couldn’t find the Equivelent of this in pytorch

#7

Basically, you just need to project the weights after each gradient descent and make them nonzero.

model = LogisticRegression(...)
# gradient descent
model.linear.weight.data = model.linear.weight.data.clamp(min=0)

The clamping operation above is equivalent to tf.keras.constraints.NonNeg().

For self.fc, you need to remove it by LogisticRegression.

1 Like
#8

Thanks for answer

how can i force the weights of GCN (GraphConv) to be positive? is it possible?

#9

It’s the same. Just clamp all the weights after a gradient update.

1 Like
#10