ELK enterprise application - elk quick build - logstash

 

ELK enterprise application - elk quick build - logstash

 

1, install JDK
elasticsearch, the operation of logstash depends on the java environment.
Download and unzip the jdk binary package.

tar xf jdk-8u144-linux-x64.tar.gz -C /usr/local
mv /usr/local/jdk1.8.0_144 /usr/local/java
cd ~

Configure the java environment variable.
Add the following at the end of the ~/.bashrc file:

export JAVA_HOME=/usr/local/java
export JRE_HOME=$JAVA_HOME/jre
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/bin/tools.jar:$JRE_HOME/lib
export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH

Make the configuration take effect.

source ~/.bashrc

2, install Logstash

It is recommended that the Linux class server download the rmp package installation.
2.1. Download the logstash installation package

touch /etc/default/logstash
ln -s /usr/local/java/bin/java /usr/bin/java
rpm -ivh logstash-6.2.4.rpm
cd ~

2.2. Configure systemd to start

When installing rpm, the configuration file for creating the startup script is /etc/logstash/startup.options

/usr/share/logstash/bin/system-install /etc/logstash/startup.options systemd

Note: When the script fails to start, you can create your own startup script.

[root@l ~]# cat /etc/systemd/system/logstash.service 
[Unit]
Description=logstash
​
[Service]
Type=simple
ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash"
ExecStop=/bin/kill -s QUIT $MAINPID
ExecReload=/bin/kill -s HUP $MAINPID
WorkingDirectory=/usr/share/logstash/bin
​
[Install]
WantedBy=multi-user.target
​
[root@l ~]# systemctl daemon-reload #####Update
[root@l ~]# 
[root@l ~]# systemctl list-unit-files |grep logstash
logstash.service                              disabled
[root@l ~]# 
 [root@l ~]# systemctl restart logstash.service #### Restart

 

2.3. Errors encountered

[root@l opt]# /usr/share/logstash/bin/system-install /etc/logstash/startup.options systemd
Using provided startup.options file: /etc/logstash/startup.options
Manually creating startup for specified platform: systemd
/usr/share/logstash/vendor/jruby/bin/jruby: Line 401: /usr/bin/java: No such file or directory
Unable to install system startup script for Logstash.

Solution

ln -s /usr/local/java/bin/java /usr/bin/java
/usr/share/logstash/bin/system-install /etc/logstash/startup.options systemd

3, configuration

cd /etc/logstash/conf.d/
chown -R logstash /etc/logstash/conf.d
mkdir /opt/logstash
touch /opt/logstash/messages
chown -R logstash /opt/logstash
chown -R logstash /opt/logstash/messages
chown -R logstash /var/log/messages

Shipper configuration file (logstash_shipper.conf)

vim logstash_shipper.conf
###########################################3
input{
   file{
       type => "messages"
       path => "/var/log/messages"
       start_position => "beginning"
       sincedb_path => "/dev/null"
   }
}
 
 
output{
    if [type] == "messages"{
       redis{
           host => "10.0.0.132"
           data_type => "list"
           key => "messages"
           port => 6379
           db => 2
           password => "123456"
       }
   }
}

Indexer configuration file (logstash_indexer.conf) Note: This configuration file must be re-node node, otherwise the two output will repeat the output log, plus the redis cache will be infinite output.

vim logstash_indexer.conf
######################################
input{
   redis{
       host => "10.0.0.132"
       data_type => "list"
       key => "messages"
       password => "123456"
       db => 2
   }
}
 
output{
    if [type] == "messages" {
       elasticsearch{
           hosts => ["10.0.0.130"]
           index => "messages-%{+YYYY-MM-dd}"
       }
   }
}

4, test

cd /usr/share/logstash/bin/
./logstash --path.settings /etc/logstash/ -r /etc/logstash/conf.d/ --config.test_and_exit
[root@l bin]# ./logstash --path.settings /etc/logstash/ -r /etc/logstash/conf.d/ --config.test_and_exit
Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
Configuration OK

5, start

systemctl start logstash.service
systemctl enable logstash.service