How to debug distributed mod?

[hbsun2113]

I only have one worker and want to debug distributed mod. I export BYTEPS_FORCE_DISTRIBUTED=1 like this but failed. What others should I set?

[hbsun2113]

nohup.txt

[hbsun2113]

[bobzhuyb]

Does it run normally in non-distributed mode? Did you start scheduler and server? We need your complete starting commands so that we can really help you.

[hbsun2113]

Does it run normally in non-distributed mode? Did you start scheduler and server? We need your complete starting commands so that we can really help you.

It runs normally in non-distributed mode. I did not start scheduler and server. Can you give me a guide about running distributed mode with only one worker like this?

[hbsun2113]

@bobzhuyb If I set the BYTEPS_FORCE_DISTRIBUTED as following, should I also start scheduler and server?

[bobzhuyb]

@bobzhuyb If I set the BYTEPS_FORCE_DISTRIBUTED as following, should I also start scheduler and server?

If you set BYTEPS_FORCE_DISTRIBUTED, you must start at least one server and one scheduler, just like you are starting distributed training job. You can start the server and scheduler on the same machine as worker, and set DMLC_PS_ROOT_URI to be your local IP or even 127.0.0.1

[hbsun2113]

@bobzhuyb
Hi, I follow the steps to start scheduler:

docker pull bytepsimage/byteps_server
docker run -it --net=host bytepsimage/byteps_server bash
export DMLC_NUM_WORKER=1
export DMLC_ROLE=scheduler 
export DMLC_NUM_SERVER=1 
export DMLC_PS_ROOT_URI=127.0.0.1
export DMLC_PS_ROOT_PORT=3034  

but meet the following problem, could you help me?

[hbsun2113]

@bobzhuyb

[bobzhuyb]

Why did you overwrite the BYTEPS_SERVER_MXNET_PATH? I don't think you should overwrite it.

Anyways, we'll get rid of the MXNet part in the BytePS soon, so you won't have this kind of problem anymore. See the server branch. We will merge it in a week or two, and update the docker image.

[hbsun2113]

@bobzhuyb If I do not overwrite the BYTEPS_SERVER_MXNET_PATH, there will be a problem:

Is there a method that I can start the scheduler now?

[bobzhuyb]

@bobzhuyb If I do not overwrite the BYTEPS_SERVER_MXNET_PATH, there will be a problem:

Is there a method that I can start the scheduler now?

This is very strange. The environmental variable should be there. See
https://github.com/bytedance/byteps/blob/master/docker/Dockerfile.server#L29
or
https://github.com/bytedance/byteps/blob/master/docker/Dockerfile.mix.mxnet15#L119

If you don't have it, set it like the dockerfile

[hbsun2113]

@bobzhuyb
Now I can start scheduler successfully with export DMLC_PS_ROOT_URI=127.0.0.1:

But if I change it to export DMLC_PS_ROOT_URI=xx.xx.xx.xx where xx.xx.xx.xx means a public ip, it failed as following:

I think it is a bug for ps-lite, do you notice it?

[bobzhuyb]

Is xx.xx.xx.xx an IP that belongs to the machine and visible to the docker? Is the DMLC_PS_ROOT_PORT port available?... What value did you set for DMLC_PS_ROOT_PORT ?

[hbsun2113]

Is xx.xx.xx.xx an IP that belongs to the machine and visible to the docker? Is the DMLC_PS_ROOT_PORT port available?... What value did you set for DMLC_PS_ROOT_PORT ?

xx.xx.xx.xx is a public IP that belongs to the machine and visible to the docker.
DMLC_PS_ROOT_PORT is 3034, and I think it is available. The following is how to test the port:
I python3 -m http.server in the docker and can visit xx.xx.xx.xx:3034 by my chrome:

what the docker shows:

My env is:

[bobzhuyb]

Can you set PS_VERBOSE=2 when you run the scheduler? This will output more logs.

[hbsun2113]

Can you set PS_VERBOSE=2 when you run the scheduler? This will output more logs.

[bobzhuyb]

Let's clarify a few things..

1. Can you bind your HTTP server to the 162.x.x.x IP and 3034 port?

2. When you say 162.x.x.x "belongs to the machine", do you mean the public IP is assigned by OpenStack/K8S/other cloud, or do you mean that it actually shows up in ifconfig?

I feel that 162.x.x.x is not an IP that can be bound locally.

[hbsun2113]

Let's clarify a few things.

1. Can you bind your HTTP server to the 162.x.x.x IP and 3034 port?
2. When you say 162.x.x.x "belongs to the machine", do you mean the public IP is assigned by OpenStack/K8S/other cloud, or do you mean that it actually shows up in ifconfig?

I feel that 162.x.x.x is not an IP that can be bound locally.

You are right! When I change the IP to what ifconfig shows, the scheduler can start successfully.
I want to know can I change the port of server or worker because my machine only has several available ports, and my worker runs failed.

worker:

scheduler:

server:

[bobzhuyb]

What do you mean by available ports? Are you referring to the ports allowed by the cloud's security group? The 172.x.x.x IP is an internal IP within a subnet, which should not be affected by security groups... For now I don't think you can specify the worker port. It will bind to a random port. This is ps-lite's logic.

From your screenshot, it does not look like IP binding/connection problem..

[hbsun2113]

What do you mean by available ports? Are you referring to the ports allowed by the cloud's security group? The 172.x.x.x IP is an internal IP within a subnet, which should not be affected by security groups... For now I don't think you can specify the worker port. It will bind to a random port. This is ps-lite's logic.

From your screenshot, it does not look like IP binding/connection problem..

Agree with you that it is not the port problem. So why did worker crash?

[hbsun2113]

@bobzhuyb

[bobzhuyb]

Are you using the docker image provided by us or built from code by yourself? Can you confirm that the same build work okay in non-distributed mode?

There is a knob here https://github.com/bytedance/byteps/blob/master/launcher/launch.py#L37 that allows you to run the program with gdb. Hopefully it will give us more info..

[hbsun2113]

Are you using the docker image provided by us or built from code by yourself? Can you confirm that the same build work okay in non-distributed mode?

There is a knob here https://github.com/bytedance/byteps/blob/master/launcher/launch.py#L37 that allows you to run the program with gdb. Hopefully it will give us more info..

1. I built from code by myself.

2. The same build work okay in non-distributed mode:
   
   and the env is:
   
   I also notice that:
   

3. but if I export BYTEPS_FORCE_DISTRIBUTED=1:
   

4. GDB: Sorry I do not have the authority to restart docker container with gdb:
   

[bobzhuyb]

The output you have is just a warning that you can probably ignore. The problem is that byteps launcher uses gdb to start the shell script example/pytorch/start_pytorch_byteps.sh.. Can you avoid using example/pytorch/start_pytorch_byteps.sh and just use the launcher to start a python script in hbsun_run.sh ?