Co-Created with @bennwitt
@bennwitt and I were in need of a toolset that could analyze disparate logs from multiple sources. We built this tool mainly to analyze video streaming logs from multipel Windows servers (IIS logs) and evolved the tool to include Limelight CDN log analysis, Conviva API logs, as well as AppNeta logs to analyze website and video streaming behaviors under load.
the setup that follows is the installation portion of this tool using Elastic Search, Kibana, and Logstash (ELK).
- Elastic signs all of its packages with the Elasticsearch PGP Signing Key. Add this key to your server.
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
- As the packages will be retrieved via HTTPS, you will need to install the apt-transport-https package.
sudo apt install apt-transport-https
- You will next need to add the Elastic repository.
There are different repositories for the standard distribution (X-Pack pre-bundled) and the Apache 2.0 licensed distribution. You will use the standard distribution in order to take advantage of the Security features of the X-Pack Basic tier.
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
- You will now need to update the local cache of available packages and their versions.
sudo apt update
- You can now install Elasticsearch.
sudo apt install elasticsearch
- Configure JVM Heap.
Java Virtual Machine is included with Elastic 7.x, so for Elastic and Kibana, there is no need to download Java. (Logstash will however require it, so we handle that later). If a JVM is started with unequal initial and max heap sizes, it may pause as the JVM heap is resized during system usage. For this reason it’s best to start the JVM with the initial and maximum heap sizes set to equal values. This file can also be found in the /etc/ folder
Edit /etc/elasticsearch/jvm.options and set -Xms and -Xmx to about one third of the system memory, but do not exceed 31g. For example...
# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space
-Xms8g
-Xmx8g
- Increase System Limits.
You should specify system limits in a systemd configuration file for the elasticsearch service. This file is included in the /etc/ folder
sudo mkdir /etc/systemd/system/elasticsearch.service.d
Copy the provided file etc/systemd/system/elasticsearch.service.d/elasticsearch.conf to /etc/systemd/system/elasticsearch.service.d
Additionally, copy the provided file etc/sysctl.d/70-elasticsearch.conf to /etc/sysctl.d. Reboot the system for these changes to take effect.
- Modify the Elasticsearch configuration.
Replace the default Elasticsearch configuration file with the provided configuration by copying etc/elasticsearch/elasticsearch.yml to /etc/elasticsearch
Modify this configuration as may be appropriate for your environment.
- Start Elasticsearch.
sudo systemctl daemon-reload
sudo systemctl start elasticsearch.service
- Enable Elasticsearch to start automatically when the system is started.
sudo systemctl enable elasticsearch.service
- Set passwords for the Elasticsaerch, Kibana, Logstash, Filebeats application:
sudo /usr/share/elasticsearch/bin/elasticsearch-setup-passwords interactive
-
Test those passwords Browse to the Elasticsearch Application from browser http://server.company.com:9200. when prompted, enter in "elastic" then the passwrod you entered in for Step #11
-
Test a query on Elasticsearch
curl -X GET "localhost:9200/?pretty"
- Install Kibana.
sudo apt install kibana
- Modify the Kibana configuration.
Replace the default Kibana configuration file with the provided configuration by copying etc/kibana/kibana.yml to /etc/kibana
- Starting Kibana the Very first time:
please be patient, to load the kibana.yml config we just copied into the Kibana, Kibana's first load must be "optimized" for the installation.
start kibana to "test" the config and if test is successful, optimize Kibana
sudo /usr/hare/kibana/bin/kibana --allow-root
PLEASE BE PATIENT: this may take 10 minutes to load the first time.
- Start Kibana.
the step above, Kibana is running from the CMD line, not as a service. Load Kibana as a service below.
sudo systemctl daemon-reload
sudo systemctl start kibana.service
Be patient. It may take a moment the first time as Kibana optimizes enabled applications.
- Enable Kibana to start automatically when the system is started.
sudo systemctl enable kibana.service
- Install nginx proxy so Kibana uses port 80 instead of 5601
apt install nginx apache2-utils -y
- COnfigure Nginx
open and edit /etc/nginx/sites-available/kibana Insert this:
server {
listen 80;
server_name elk_server;
server_name elk_server.company.net;
location / {
proxy_pass http://localhost:5601;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}
- enable this new Kibana profile for nginx
ln -s /etc/nginx/sites-available/kibana /etc/nginx/sites-enabled/
and test the config
nginx -t
- Enable NGinx to run as a service
systemctl enable nginx
and restart nginx
systemctl restart nginx
- Install Java
apt install default-jre
- edit /etc/profile add this text to the end of the document
JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64
PATH=$PATH:$HOME/bin:$JAVA_HOME/bin
export JAVA_HOME
export JRE_HOME
export PATH
- Set JAVA_HOME
export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64
verify java home
echo $JAVA_HOME
- Install Logstash
apt-get update && sudo apt-get install logstash
apt install software-properties-common
We use SAMBA to drop files onto the ELK server directly. to do this, we need SAMBA
- install samaba
apt-get install samba -y
- mkdir of the shared folder and give everyone full access to that folder.
mkdir -m777 inbound
- create local SAMBA user
smbpasswd -a {{sambauser}}
password
changeme
- Change SMB config
nano /etc/samba/smb.conf
change these lines
workgroup = {{elkgroup}}
server string = %h
change message body for "[printers]" to:
[inbound]
comment = inbound
browseable = yes
path = /index/inbound/
guest ok = yes
ready only = no
create mask = 0777
remove next print section altogether
- restart SMB to that the sahre is active
/etc/init.d/smbd restart
- Test SMB storage location go to WIndows machine that test that \elk_server\inbound\ can be reached and if its available.
-
create a symbolic link for logstash to run in your folder
ln -s /usr/share/logstash/bin/logstash logstash -
run it from the command line
./logstash -w 1 -f /index/inbound/config/maincookie.conf
