Pipeline Using a lot of RAM, Crashing

[camwin]

I am running into an issue using Pipeline where the process crashes after a while of collecting telemetry data from about 12 Cisco ASR 9006, 9001, and 9901 routers. These routers are streaming 7 different sensor-paths (uptime, cpu, node summary, optics-info, process-info, and generic-counters) every 10 seconds.

Pipeline normally uses about 5-250MB of RAM and has a "scrape duration" of about 12ms. When Pipeline crashes, RAM usage goes to >1GB, and scrape duration goes >1 second. I can either restart the Pipeline process or reboot the host to get Pipeline working again.

Pipeline.log does show some meaningful log messages. I can post that here if it'll help.

Has anyone ran into this sort of issue?

[remingtonc]

What is your datachanneldepth set to?

[tsamanis]

same here...I used datachanneldepth 1000 and 10000 respectively but nothing changed in behaviour.