Setting up Logstash - the basics

Setting up Logstash, the basics


Installing Logstash on the server (done with Ubuntu 16.04 LTS).
    server1(admin) ~$ wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
    server1(admin) ~$ sudo apt-get install apt-transport-https
    server1(admin) ~$ echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list
    server1(admin) ~$    sudo apt-get update && sudo apt-get install logstash


You also need to have Java installed. Only Java 8 is supported with Logstash (do not use Java 9 yet).
You can used the official Java Distribution or an Open Source distribution like OpenJDK

To install OpenJDK:
    server1(admin) ~$ sudo apt-get install openjdk-8-jre-headless
    
To install the official Java JDK from Oracle:
    Go to http://www.oracle.com/technetwork/java/javase/downloads/index.html
    Download the last version of the Java SE (Standard Edition) 8 JDK
    Extract the content of the downloaded archive into one directory, eg /opt/, this will result in a directory called /opt/jdk1.8.8_<update version>
    As root create a symbolic link from this directory to /opt/java, so if you do an upgrade later you just have to change this link and the rest remain unchanged.
        server1(admin) ~$    sudo ln -s /opt/jdk1.8.0_101 /opt/java
    As root, created a symbolic link for the java binary:
        server1(admin) ~$    sudo ln -s /opt/java/bin/java /usr/bin/java
    Test the version and the availability of the executable by doing:
        server1(admin) ~$    java -version
    java version "1.8.0_101"
    Java(TM) SE Runtime Environment (build 1.8.0_101-b13)
    Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode)


The Logstash configuration is organized into 3 main blocks: input, filter and output.
# at the beginning of a line starts a comment
{ } are used to enclose instruction belonging to the same blocks
Values assigned to the parameters can be:
number: value => 0
boolean: value => true
string: value => "string"
array: value => [ "string 1", "string 2" ]
object: value => { param1 = "string", param2 = number }

The detailed documentation on the Elastic web site will give you a detailed overview of all possible parameters and value types per configuration elements.

A basic configuration can be:
input {
    file {
        path => [ "/var/log/messages" ]
    }
}
filter {}
output {
    elasticsearch {
        hosts => [ "es1:9200","es2:9200" ]
    }
}  
 
This basic configuration will send all new lines appearing into the /var/log/messages file (like "tail" can do at the command line) to an ElasticSearch cluster for which 2 nodes address are given.

Before launching Logstash, or reloading it with a new configuration, you can test your newly made configuration at the command line like this:
    server1(admin) ~$ /usr/share/logstash/bin/logstash -f test.conf -d

The Ubuntu startup script will load all files, regardless their extension, present in the /etc/logstash/conf.d directory at startup. Logstash merge all file together to form a single configuration, therefore what you define in one file exists also for another.
In other words, if you define two different outputs in two different configuration files for two different input, because everything is merged together, all messages entering each input will be sent to both output.
If you define a fixed field in one input of one configuration file, you can define actions in another file using its value.

To avoid sending each input to each output, Logstash configuration gives you the ability to create logical structures in your configuration file with "if ... then ... else" clauses.
This will be useful to apply different filtering on events originating from different sources. Because you can add your own fieldnames and values via the configuration, you are able to add a field "type" with the value "tweet" in your Twitter source configuration.
Later on, you perform a test on the value of the "type" field, like in this extract of configuration:
<...>
  type => "tweet"
<...>
  if [type] == "tweet" {
        (some actions definition)
    }
<...>

These logical structures can be incorporated into any of the 3 main sections (input, filter and output). You cannot place a logical structure inside the configuration element you plan to use.
In other words, if you want to store the data into two different ElasticSearch indexes, you cannot do:
elasticsearch {
    if [type] == "tweet" {
        name = "tweets"
    } else {
        name = "others"
    }
}


but you will need to repeat the ElasticSearch configuration block inside the "if" clause:
if [type] == "tweet" {
    elasticsearch {
        name = "tweets"
    }
} else {
    elasticsearch {
        name = "others"
    }
}