Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

(111: Connection refused) while connecting to upstream, client: 127.0.0.1, server #23

Open
orena1 opened this issue Aug 17, 2023 · 3 comments
Open

(111: Connection refused) while connecting to upstream, client: 127.0.0.1, server #23

orena1 opened this issue Aug 17, 2023 · 3 comments

Comments

@orena1
Copy link

orena1 commented Aug 17, 2023

Hi,
I've followed the instructions here https://elephant-track.github.io/#/?id=setting-up-with-singularity

git clone https://github.com/elephant-track/elephant-server
cd elephant-server
make singularity-build
make singularity-launch

But I get this error:

base) oamsalem@colab00:~/Downloads/elephant-server$ make singularity-launch
singularity instance start --nv --bind /home/oamsalem/.elephant_binds/var/lib:/var/lib,/home/oamsalem/.elephant_binds/var/log:/var/log,/home/oamsalem/.elephant_binds/var/run:/var/run,/home/oamsalem/Downloads/elephant-server/workspace:/workspace elephant.sif elephant
INFO:    instance started successfully
if [ all = all ]; then \
        singularity exec instance://elephant /start.sh; \
else \
        SINGULARITYENV_CUDA_VISIBLE_DEVICES=all singularity exec instance://elephant /start.sh; \
fi 

Checking for script in /app/prestart.sh
Running script /app/prestart.sh
set RUN_ON_FLASK
2023-08-17 15:43:46,197 INFO Included extra file "/etc/supervisor/conf.d/supervisord.conf" during parsing
Unlinking stale socket /var/run/supervisor.sock
2023-08-17 15:43:46,521 INFO RPC interface 'supervisor' initialized
2023-08-17 15:43:46,521 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2023-08-17 15:43:46,521 INFO supervisord started with pid 19
2023-08-17 15:43:47,525 INFO spawned: 'rabbitmq-server' with pid 31
2023-08-17 15:43:47,528 INFO spawned: 'redis-server' with pid 32
2023-08-17 15:43:47,531 INFO spawned: 'uwsgi' with pid 33
2023-08-17 15:43:47,534 INFO spawned: 'celery' with pid 34
2023-08-17 15:43:47,536 INFO spawned: 'nginx' with pid 36
32:C 17 Aug 15:43:47.724 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
32:C 17 Aug 15:43:47.724 # Redis version=4.0.9, bits=64, commit=00000000, modified=0, pid=32, just started
32:C 17 Aug 15:43:47.724 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
32:M 17 Aug 15:43:47.725 * Increased maximum number of open files to 10032 (it was originally set to 1024).
32:M 17 Aug 15:43:47.726 * Running mode=standalone, port=6379.
32:M 17 Aug 15:43:47.726 # Server initialized
32:M 17 Aug 15:43:47.726 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
32:M 17 Aug 15:43:47.726 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
32:M 17 Aug 15:43:47.726 * Ready to accept connections
2023/08/17 15:43:47 [warn] 36#36: the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /etc/nginx/nginx.conf:1
nginx: [warn] the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /etc/nginx/nginx.conf:1
[uWSGI] getting INI configuration from /etc/uwsgi/uwsgi.ini
[uWSGI] getting INI configuration from /app/uwsgi.ini

;uWSGI instance configuration
[uwsgi]
chown-socket = 1001:1001
uid = 1001
gid = 1001
ini = /etc/uwsgi/uwsgi.ini
ini = /app/uwsgi.ini
module = main
callable = app
cheaper = 2
processes = 16
socket = /tmp/uwsgi.sock
chown-socket = nginx:nginx
chmod-socket = 664
hook-master-start = unix_signal:15 gracefully_kill_them_all
need-app = true
die-on-term = true
show-config = true
route = ^/state/gpus* donotlog:
;end of configuration

*** Starting uWSGI 2.0.18 (64bit) on [Thu Aug 17 15:43:48 2023] ***
compiled with version: 7.3.0 on 22 July 2019 16:35:30
os: Linux-5.15.0-46-generic #49~20.04.1-Ubuntu SMP Thu Aug 4 19:15:44 UTC 2022
nodename: colab00
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 40
current working directory: /home/oamsalem/Downloads/elephant-server
detected binary path: /opt/conda/bin/uwsgi
*** dumping internal routing table ***
[rule: 0] subject: path_info regexp: ^/state/gpus* action: donotlog:
*** end of the internal routing table ***
your processes number limit is 1159763
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
chown(): Operation not permitted [core/utils.c line 2649]
2023-08-17 15:43:48,198 INFO exited: uwsgi (exit status 1; not expected)
2023/08/17 15:43:48 [error] 52#52: *2 connect() to unix:///tmp/uwsgi.sock failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: , request: "GET /state/gpus HTTP/1.1", upstream: "uwsgi://unix:///tmp/uwsgi.sock:", host: "localhost:8080"
2023-08-17 15:43:48,740 INFO success: rabbitmq-server entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-08-17 15:43:48,740 INFO success: redis-server entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-08-17 15:43:48,741 INFO success: celery entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-08-17 15:43:48,741 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2023-08-17 15:43:49,744 INFO spawned: 'uwsgi' with pid 263
2023/08/17 15:43:49 [error] 52#52: *2 connect() to unix:///tmp/uwsgi.sock failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: , request: "GET /state/gpus HTTP/1.1", upstream: "uwsgi://unix:///tmp/uwsgi.sock:", host: "localhost:8080"
[uWSGI] getting INI configuration from /etc/uwsgi/uwsgi.ini
[uWSGI] getting INI configuration from /app/uwsgi.ini

;uWSGI instance configuration
[uwsgi]
chown-socket = 1001:1001
uid = 1001
gid = 1001
ini = /etc/uwsgi/uwsgi.ini
ini = /app/uwsgi.ini
module = main
callable = app
cheaper = 2
processes = 16
socket = /tmp/uwsgi.sock
chown-socket = nginx:nginx
chmod-socket = 664
hook-master-start = unix_signal:15 gracefully_kill_them_all
need-app = true
die-on-term = true
show-config = true
route = ^/state/gpus* donotlog:
;end of configuration

*** Starting uWSGI 2.0.18 (64bit) on [Thu Aug 17 15:43:49 2023] ***
compiled with version: 7.3.0 on 22 July 2019 16:35:30
os: Linux-5.15.0-46-generic #49~20.04.1-Ubuntu SMP Thu Aug 4 19:15:44 UTC 2022
nodename: colab00
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 40
current working directory: /home/oamsalem/Downloads/elephant-server
detected binary path: /opt/conda/bin/uwsgi
*** dumping internal routing table ***
[rule: 0] subject: path_info regexp: ^/state/gpus* action: donotlog:
*** end of the internal routing table ***
your processes number limit is 1159763
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
chown(): Operation not permitted [core/utils.c line 2649]
2023-08-17 15:43:49,757 INFO exited: uwsgi (exit status 1; not expected)
2023/08/17 15:43:50 [error] 52#52: *2 connect() to unix:///tmp/uwsgi.sock failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: , request: "GET /state/gpus HTTP/1.1", upstream: "uwsgi://unix:///tmp/uwsgi.sock:", host: "localhost:8080"
2023/08/17 15:43:51 [error] 52#52: *2 connect() to unix:///tmp/uwsgi.sock failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: , request: "GET /state/gpus HTTP/1.1", upstream: "uwsgi://unix:///tmp/uwsgi.sock:", host: "localhost:8080"
2023-08-17 15:43:52.551472-04:00 [notice] <0.44.0> Application syslog exited with reason: stopped
2023-08-17 15:43:52.562684-04:00 [notice] <0.230.0> Logging: switching to configured handler(s); following messages may not be visible in this log output
2023-08-17 15:43:52,565 INFO spawned: 'uwsgi' with pid 264
[uWSGI] getting INI configuration from /etc/uwsgi/uwsgi.ini
[uWSGI] getting INI configuration from /app/uwsgi.ini

;uWSGI instance configuration
[uwsgi]
chown-socket = 1001:1001
uid = 1001
gid = 1001
ini = /etc/uwsgi/uwsgi.ini
ini = /app/uwsgi.ini
module = main
callable = app
cheaper = 2
processes = 16
socket = /tmp/uwsgi.sock
chown-socket = nginx:nginx
chmod-socket = 664
hook-master-start = unix_signal:15 gracefully_kill_them_all
need-app = true
die-on-term = true
show-config = true
route = ^/state/gpus* donotlog:
;end of configuration

*** Starting uWSGI 2.0.18 (64bit) on [Thu Aug 17 15:43:52 2023] ***
compiled with version: 7.3.0 on 22 July 2019 16:35:30
os: Linux-5.15.0-46-generic #49~20.04.1-Ubuntu SMP Thu Aug 4 19:15:44 UTC 2022
nodename: colab00
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 40
current working directory: /home/oamsalem/Downloads/elephant-server
detected binary path: /opt/conda/bin/uwsgi
*** dumping internal routing table ***
[rule: 0] subject: path_info regexp: ^/state/gpus* action: donotlog:
*** end of the internal routing table ***
your processes number limit is 1159763
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
chown(): Operation not permitted [core/utils.c line 2649]
2023-08-17 15:43:52,581 INFO exited: uwsgi (exit status 1; not expected)
2023/08/17 15:43:52 [error] 52#52: *2 connect() to unix:///tmp/uwsgi.sock failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: , request: "GET /state/gpus HTTP/1.1", upstream: "uwsgi://unix:///tmp/uwsgi.sock:", host: "localhost:8080"

  ##  ##      RabbitMQ 3.12.3
  ##  ##
  ##########  Copyright (c) 2007-2023 VMware, Inc. or its affiliates.
  ######  ##
  ##########  Licensed under the MPL 2.0. Website: https://rabbitmq.com

  Erlang:      25.3.2.5 [jit]
  TLS Library: OpenSSL - OpenSSL 1.1.1  11 Sep 2018
  Release series support status: supported

  Doc guides:  https://rabbitmq.com/documentation.html
  Support:     https://rabbitmq.com/contact.html
  Tutorials:   https://rabbitmq.com/getstarted.html
  Monitoring:  https://rabbitmq.com/monitoring.html

  Logs: /var/log/rabbitmq/[email protected]
        <stdout>

  Config file(s): (none)

  Starting broker...2023/08/17 15:43:53 [error] 52#52: *2 connect() to unix:///tmp/uwsgi.sock failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: , request: "GET /state/gpus HTTP/1.1", upstream: "uwsgi://unix:///tmp/uwsgi.sock:", host: "localhost:8080"
2023/08/17 15:43:54 [error] 52#52: *2 connect() to unix:///tmp/uwsgi.sock failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: , request: "GET /state/gpus HTTP/1.1", upstream: "uwsgi://unix:///tmp/uwsgi.sock:", host: "localhost:8080"
2023-08-17 15:43:55,758 INFO spawned: 'uwsgi' with pid 265
2023/08/17 15:43:55 [error] 52#52: *2 connect() to unix:///tmp/uwsgi.sock failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: , request: "GET /state/gpus HTTP/1.1", upstream: "uwsgi://unix:///tmp/uwsgi.sock:", host: "localhost:8080"
[uWSGI] getting INI configuration from /etc/uwsgi/uwsgi.ini
[uWSGI] getting INI configuration from /app/uwsgi.ini

;uWSGI instance configuration
[uwsgi]
chown-socket = 1001:1001
uid = 1001
gid = 1001
ini = /etc/uwsgi/uwsgi.ini
ini = /app/uwsgi.ini
module = main
callable = app
cheaper = 2
processes = 16
socket = /tmp/uwsgi.sock
chown-socket = nginx:nginx
chmod-socket = 664
hook-master-start = unix_signal:15 gracefully_kill_them_all
need-app = true
die-on-term = true
show-config = true
route = ^/state/gpus* donotlog:
;end of configuration

*** Starting uWSGI 2.0.18 (64bit) on [Thu Aug 17 15:43:55 2023] ***
compiled with version: 7.3.0 on 22 July 2019 16:35:30
os: Linux-5.15.0-46-generic #49~20.04.1-Ubuntu SMP Thu Aug 4 19:15:44 UTC 2022
nodename: colab00
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 40
current working directory: /home/oamsalem/Downloads/elephant-server
detected binary path: /opt/conda/bin/uwsgi
*** dumping internal routing table ***
[rule: 0] subject: path_info regexp: ^/state/gpus* action: donotlog:
*** end of the internal routing table ***
your processes number limit is 1159763
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
chown(): Operation not permitted [core/utils.c line 2649]
2023-08-17 15:43:55,778 INFO exited: uwsgi (exit status 1; not expected)
2023/08/17 15:43:56 [error] 52#52: *2 connect() to unix:///tmp/uwsgi.sock failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: , request: "GET /state/gpus HTTP/1.1", upstream: "uwsgi://unix:///tmp/uwsgi.sock:", host: "localhost:8080"
2023-08-17 15:43:56,759 INFO gave up: uwsgi entered FATAL state, too many start retries too quickly
 completed with 0 plugins.
2023/08/17 15:43:57 [error] 52#52: *2 connect() to unix:///tmp/uwsgi.sock failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: , request: "GET /state/gpus HTTP/1.1", upstream: "uwsgi://unix:///tmp/uwsgi.sock:", host: "localhost:8080"
 
 -------------- celery@colab00 v5.0.5 (singularity)
--- ***** ----- 
-- ******* ---- Linux-5.15.0-46-generic-x86_64-with-debian-buster-sid 2023-08-17 15:43:58
- *** --- * --- 
- ** ---------- [config]
- ** ---------- .> app:         main:0x7f66f63a4090
- ** ---------- .> transport:   redis://localhost:6379//
- ** ---------- .> results:     redis://localhost:6379/
- *** --- * --- .> concurrency: 40 (thread)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** ----- 
 -------------- [queues]
                .> celery           exchange=celery(direct) key=celery
                

2023/08/17 15:43:58 [error] 52#52: *2 connect() to unix:///tmp/uwsgi.sock failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: , request: "GET /state/gpus HTTP/1.1", upstream: "uwsgi://unix:///tmp/uwsgi.sock:", host: "localhost:8080"
2023/08/17 15:43:59 [error] 52#52: *2 connect() to unix:///tmp/uwsgi.sock failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: , request: "GET /state/gpus HTTP/1.1", upstream: "uwsgi://unix:///tmp/uwsgi.sock:", host: "localhost:8080"
2023/08/17 15:44:00 [error] 52#52: *2 connect() to unix:///tmp/uwsgi.sock failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: , request: "GET /state/gpus HTTP/1.1", upstream: "uwsgi://unix:///tmp/uwsgi.sock:", host: "localhost:8080"
2023/08/17 15:44:01 [error] 52#52: *2 connect() to unix:///tmp/uwsgi.sock failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: , request: "GET /state/gpus HTTP/1.1", upstream: "uwsgi://unix:///tmp/uwsgi.sock:", host: "localhost:8080"
2023/08/17 15:44:02 [error] 52#52: *2 connect() to unix:///tmp/uwsgi.sock failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: , request: "GET /state/gpus HTTP/1.1", upstream: "uwsgi://unix:///tmp/uwsgi.sock:", host: "localhost:8080"
2023/08/17 15:44:03 [error] 52#52: *2 connect() to unix:///tmp/uwsgi.sock failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: , request: "GET /state/gpus HTTP/1.1", upstream: "uwsgi://unix:///tmp/uwsgi.sock:", host: "localhost:8080"
^C2023-08-17 15:44:03,921 WARN received SIGINT indicating exit request

Any suggestions?
Thanks,
Oren
@mestia
Copy link

mestia commented Sep 8, 2023 •
edited
Loading

I could work around this problem by using network instead of a unix socket:

docker/nginx.conf
uwsgi_pass  127.0.0.1:10002;

docker/uwsgi.ini
socket = 127.0.0.1:10002

@orena1
Copy link
Author

orena1 commented Sep 29, 2023

Thanks @mestia so what should I do?

@mestia
Copy link

mestia commented Oct 3, 2023

You need to clone the repository, modify the 2 mentioned above files and re-build the singularity image with make singularity-build
After that one should be able to start the singularity instance and access the server from a browser.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mestia @orena1