Skip to content
This repository has been archived by the owner on Sep 13, 2022. It is now read-only.

System stops working after a few weeks #39

Open
xadcoh opened this issue Jul 18, 2016 · 1 comment
Open

System stops working after a few weeks #39

xadcoh opened this issue Jul 18, 2016 · 1 comment

Comments

@xadcoh
Copy link

xadcoh commented Jul 18, 2016

Hi.

I've deloyed amsterdam 1.0 and it runs good for a while, but after a few weeks it's just stop responding (i mean web interface shows no activity). Evebox loading endlessly, scirius shows "Fetching data". Elasticsearch become red. Eve Insert rate is about 20 (but when it works good its about 100) Some log:

> elasticsearch_1  | [2016-07-18 17:52:42,687][DEBUG][action.admin.cluster.node.stats] [Dum-Dum Dugan] failed to execute on node [peHmChnxQ5qjNYNvCJkjhw]
> elasticsearch_1  | ReceiveTimeoutTransportException[[Dum-Dum Dugan][172.17.0.3:9300][cluster:monitor/nodes/stats[n]] request_id [1456381] timed out after [17231ms]]
> elasticsearch_1  |      at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:679)
> elasticsearch_1  |      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> elasticsearch_1  |      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> elasticsearch_1  |      at java.lang.Thread.run(Thread.java:745)
> elasticsearch_1  | [2016-07-18 17:52:47,052][WARN ][transport                ] [Dum-Dum Dugan] Received response for a request that has timed out, sent [21596ms] ago, timed out [4365ms] ago, action [cluster:monitor/nodes/stats[n]], node [{Dum-Dum Dugan}{peHmChnxQ5qjNYNvCJkjhw}{172.17.0.3}{172.17.0.3:9300}], id [1456381]
> kibana_1         | {"type":"log","@timestamp":"2016-07-18T17:52:59+00:00","tags":["status","plugin:elasticsearch","error"],"pid":11,"name":"plugin:elasticsearch","state":"red","message":"Status changed from green to red - Request Timeout after 30000ms","prevState":"green","prevMsg":"Kibana index ready"}
> kibana_1         | {"type":"log","@timestamp":"2016-07-18T17:53:04+00:00","tags":["status","plugin:elasticsearch","info"],"pid":11,"name":"plugin:elasticsearch","state":"green","message":"Status changed from red to green - Kibana index ready","prevState":"red","prevMsg":"Request Timeout after 30000ms"}

Configuration is:
SW:
Debian Jessie 8.5 x86_64
HW:
core i3-3220
16Gb RAM
2x500Gb SATA 7200RPM @ RAID1
eth - 1Gbps (real traffic is about 30-40Mbps)

It there performance bottle neck? or maybe it's configuration problem?

Thank you

@infiniteproject
Copy link
Contributor

Increase ES_HEAP_SIZE in docker-compose.yml.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants