Which ports need to be opened in firewall for DGL Distributed?

shadowforce · October 23, 2022, 11:12pm

DGL distributed doesn’t run when firewall is enabled on hosts even though I have opened port 30050 on all hosts. I also tried specifying port in ip_config.txt file and opened that port in firewall along with port 30050 on all hosts but no joy:

192.168.1.100 10020
192.168.1.101 10020
...

Should the port in the ip_config.txt file be enclosed in brackets like this?

192.168.1.100 [10020]
192.168.1.101 [10020]
...

The training works fine when I disable firewall on all hosts (without specifying port in ip_config.txt), but it doesn’t work with firewall enabled.

Which ports need to be opened in firewall for distributed training?

Rhett-Ying · October 24, 2022, 1:45am

when the firewall is enabled, does ssh 192.168.1.100 work? and what’s the error log?

shadowforce · October 24, 2022, 7:49am

yes, ssh works, I have setup passwordless ssh.
Where can I get the error log?

does dgl uses only port 30050 or other ports too?

Rhett-Ying · October 24, 2022, 8:34am

if any error/exception are thrown, you’re supposed to see them on the terminal where you launch the train job.

30050 is used for servers and free ports will be used for clients via:

github.com

dmlc/dgl/blob/e682fa74bad92449e86cc01e368d504b7fd5665e/python/dgl/distributed/rpc_client.py#L86-L107

      
        
            def get_local_usable_addr(probe_addr):
                """Get local usable IP and port
            
            
    Returns
                -------
                str
                    IP address, e.g., '192.168.8.12:50051'
                """
                sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
                try:
                    # should get the address on the same subnet as probe_addr's
                    sock.connect((probe_addr, 1))
                    ip_addr = sock.getsockname()[0]
                except ValueError:
                    ip_addr = "127.0.0.1"
                finally:
                    sock.close()
                sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
                sock.bind(("", 0))
                sock.listen(1)

This file has been truncated. show original

So I think you need to disable firewall or clients cannot obtain any free port that servers could access if firewall blocks any ports in default.

Could you share the launch cmd and log too?

system · November 23, 2022, 8:35am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.