前言
查看日志的傳統方法是：登錄操作系統，使用命令工具如cat、tail、sed、awk、grep等等進行過濾輸出后分析，處理少量日志還好，日志量大處理效率就沒那么高了。而且很多情況下開發人員需要查看并分析日志進行排錯，但他們對Linux命令又不是太熟悉，而且有時候又不能賦予他們服務器權限，更多時候是運維把日志文件導出來發給開發人員，這無疑會給我們增加工作量。ELK（Elasticsearch+Logstash+Kibana）架構就是專門為采集、分析、存儲日志所設計的：
Elasticsearch：基于Lucenne的搜索服務器，提供一個分布式多用戶的全文搜索引擎，能過做到實時搜索。
Logstash：可以對日志進行采集、過濾、輸出。
Kibana：可以匯總、分析、搜索日志數據并提供友好的web界面。
工作流程：logstash agent監控并過濾日志，為了保證日志的完整性先將日志內容輸出到RabbitMQ進行存儲；logstash
indexer再把RabbitMQ上的日志隊列收集后發送給全文搜索服務器Elasticsearch，然后可以用Elasticsearch進行自定
義搜索，再通過Kibana來結合自定義搜索進行頁面展示。
ELK架構圖
從官網下載軟件logstash、elasticsearch、kibana以及JRE，分別在相應主機上安裝
wget https://download.elastic.co/kibana/kibana/kibana-4.5.3-1.x86_64.rpm wget https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/rpm/elasticsearch/2.3.4/elasticsearch-2.3.4.rpm wget https://download.elastic.co/logstash/logstash/packages/centos/logstash-2.3.4-1.noarch.rpm wget http://download.oracle.com/otn-pub/java/jdk/8u101-b13/jre-8u101-linux-x64.rpm
RabbitMQ配置
安裝RabbitMQ
yum install rabbitmq-server
啟用RabbitMQ的web管理功能
/usr/lib/rabbitmq/bin/rabbitmq-plugins enable rabbitmq_management /usr/lib/rabbitmq/bin/rabbitmq-plugins list
下載并安裝命令管理工具rabbitmqadmin
wget http://rabbitmq-server:15672/cli/rabbitmqadmin mv rabbitmqadmin /usr/local/bin chmod +x /usr/local/bin/rabbitmqadmin
給rabbitmqadmin工具準備配置文件
# vim /etc/mqadmin.conf [default] hostname = localhost port = 55672 username = liang password = liang123
創建一個vhost和user并賦權
rabbitmqctl add_user liang liang123 rabbitmqctl add_vhost elk rabbitmqctl set_permissions -p elk liang ".*" ".*" ".*" rabbitmqctl set_user_tags liang administrator rabbitmqctl list_permissions -p elk
創建一個exchange
rabbitmqadmin -c /etc/mqadmin.conf declare exchange --vhost=elk name=elk_exchange type=direct
創建一個queue
rabbitmqadmin -c /etc/mqadmin.conf declare queue --vhost=elk name=elk_queue durable=true
創建一個binding，綁定之前創建的exchange和queue并設置一個routing_key
rabbitmqadmin -c /etc/mqadmin.conf --vhost=elk declare binding source="elk_exchange" destination="elk_queue" routing_key="elk_key"
以上關于RabbitMQ的配置均可以通過登錄web控制臺進行操作，更簡單方便，這里就不再演示了。
Elasticsearch配置
給elasticsearch安裝shield插件，用于權限控制，此插件是收費產品，可免費使用30天，到期后可降級使用，只是關于集群的一些功能將不可用。
cd /usr/share/elasticsearch bin/plugin install license bin/plugin install shield
如有需要修改elasticsearch.yml，配置服務監聽地址，默認監聽在127.0.0.1上，端口是9200和9300；配置日志數據的存儲路徑，默認保存在/var/lib/elasticsearch下
path.data: /data/elastic_data network.host: 192.168.X.X
配置shield，修改shield的權限控制文件roles.yml，修改默認角色logstash的權限，指定可創建的索引文件；并添加一個readonly的角色dashboard，用于控制用戶在kibana上的權限。
logstash: cluster: - manage_index_templates indices: - names: [ 'logstash-*','nginx-*','tomcat-*' ] privileges: - write - delete - create_index dashboard: indices: 'tomcat-*': - indices:admin/mappings/fields/get - indices:admin/validate/query - indices:data/read/search - indices:data/read/msearch - indices:data/read/field_stats - indices:admin/get '.kibana': - indices:admin/exists - indices:admin/mappings/fields/get - indices:admin/refresh - indices:admin/validate/query - indices:data/read/get - indices:data/read/mget - indices:data/read/search
配置shield，創建用戶logstash、kibana、readuser、admin并指定相應角色
cd /usr/share/elasticsearch/bin/shield ./esusers useradd logstash -p 123456 -r logstash ./esusers useradd kibana -p 123456 -r kibana4_server ./esusers useradd readuser -p 123456 -r dashboard ./esusers useradd admin -p 123456 -r admin
啟動Elasticsearch
service elasticsearch start
Logstash配置
配置nginx服務器輸出json格式日志
log_format json '{"@timestamp":"$time_iso8601",'
'"host":"$server_addr",'
'"clientip":"$remote_addr",' '"size":$body_bytes_sent,'
'"responsetime":$request_time,'
'"upstreamtime":"$upstream_response_time",'
'"upstreamhost":"$upstream_addr",'
'"http_host":"$host",'
'"url":"$uri",'
'"xff":"$http_x_forwarded_for",'
'"referer":"$http_referer",'
'"agent":"$http_user_agent",'
'"status":"$status"}';
access_log /usr/local/nginx/logs/api_json.log json;
配置logstash agent采集nginx日志并輸出到RabbitMQ；為了排錯，同時輸出一份日志到本地。
# vim /etc/logstash/conf.d/ngx_log.conf
input {
file {
path => "/usr/local/nginx/logs/api_json.log"
codec => "json"
type => "nginx"
}
}
output {
rabbitmq {
host => "RabbitMQ_server"
port => "5672"
vhost => "elk"
exchange => "elk_exchange"
exchange_type => "direct"
key => "elk_key"
user => "liang"
password => "liang123"
}
stdout { codec => rubydebug }
}
配置tomcat服務器輸出json格式日志，修改工程的logback.xml配置文件，添加如下配置
<appender name="LOGSTASH" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>${catalina.base}/logs/tomcat_json.log</file>
<encoder class="net.logstash.logback.encoder.LogstashEncoder">
<charset>utf8</charset>
</encoder>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>${catalina.base}/logs/tomcat_json-%d{yyyy-MM-dd}.log</fileNamePattern>
</rollingPolicy>
</appender>
<root level="info">
<appender-ref ref="LOGSTASH" />
</root>
下載依賴的jar包logstash-logback-encoder到{CATALINA_BASE}/lib
wget http://central.maven.org/maven2/net/logstash/logback/logstash-logback-encoder/4.4/logstash-logback-encoder-4.4.jar
配置logstash agent采集tomcat日志并輸出到RabbitMQ
# vim /etc/logstash/conf.d/tomcat_log.conf
input {
file {
path => "/usr/local/tomcat/logs/tomcat_json.log"
codec => "json"
type => "tomcat"
}
}
output {
rabbitmq {
host => "RabbitMQ_server"
port => "5672"
vhost => "elk"
exchange => "elk_exchange"
exchange_type => "direct"
key => "elk_key"
user => "liang"
password => "liang123"
}
stdout { codec => rubydebug }
}
配置logstash indexer把日志從RabbitMQ輸出到Elasticsearch
# vim /etc/logstash/conf.d/rabbitmq.conf
input {
rabbitmq {
host => "127.0.0.1"
subscription_retry_interval_seconds => "5"
vhost => "elk"
exchange => "elk_exchange"
queue => "elk_queue"
durable => "true"
key => "elk_key"
user => "liang"
password => "liang123"
}
}
output {
if [type] == "nginx" {
elasticsearch {
hosts => "Elasticsearch_server:9200"
user => "logstash"
password => "123456"
index => "nginx-%{+YYYY.MM.dd}"
}
}
else if [type] == "tomcat" {
elasticsearch {
hosts => "Elasticsearch_server:9200"
user => "logstash"
password => "123456"
index => "tomcat-%{+YYYY.MM.dd}"
}
}
else {
file {
path => "/var/log/logstash/unknown_messages.log"
}
}
stdout { codec => rubydebug }
}
啟動logstash服務
service logstash start
在RabbitMQ服務器上查看是否接收到日志消息，登錄RabbitMQ的web控制臺查看詳細信息。
rabbitmqctl list_queues -p elk
kibana配置
給kibana安裝shield插件，用于權限控制
cd /opt/kibana/bin ./kibana plugin --install kibana/shield/2.3.4
修改kibana的主配置文件kibana.yml，似乎啟用權限控制后強制使用https，shield的加密key可以隨便指定，會話超時時間默認是30分鐘，超時時間的單位為毫秒
elasticsearch.username: "kibana" elasticsearch.password: "123456" elasticsearch.url: "http://elasticsearch_server:9200" server.ssl.cert: /opt/kibana/ssl/kibana.crt server.ssl.key: /opt/kibana/ssl/kibana.key shield.encryptionKey: "abc123" shield.sessionTimeout: 600000
啟動kibana
service kibana start
kibana啟用后，就可以通過https://server:5601進行訪問了。如有需要配置Apache做個反向代理
NameVirtualHost *:443 <VirtualHost *:443> ProxyRequests on SSLEngine on SSLProxyEngine on SSLCertificateFile /opt/kibana/ssl/kibana.crt SSLCertificateKeyFile /opt/kibana/ssl/kibana.key ProxyPass / https://127.0.0.1:5601/ ProxyPassReverse / https://127.0.0.1:5601/ </VirtualHost>
登錄界面。
填入之前定義的索引文件，就可以處理日志了。
看到kibana有收到日志就算成功了。
至此，整個架構部署完畢，如有錯誤或不足之處，歡迎指正。
原創文章，作者：秦，如若轉載，請注明出處：http://www.www58058.com/26776
只完成了第一步，只將日志成功輸入到ES里，要是能加上grok規律規則就好了。將日志每個字段分別取出。
通過kibana創建各種圖表，進行統計分析。