作為運維人員，經常會需要會對日志中的某些重要信息進行篩選，比如說ip等參數。
案例一：篩選出IP地址信息
日志信息如下：
[root@C67-X64-A1 hanghang]# cat test.txt Jul 13 08:13:09 localhost sshd[14678]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=61.152.95.172 Jul 13 08:13:09 localhost sshd[14679]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=222.73.173.143 user=root Jul 13 08:13:11 localhost sshd[14691]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=61.152.95.172 user=admin Jul 13 08:13:11 localhost sshd[14692]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=222.73.173.143 Jul 13 08:13:14 localhost sshd[14707]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=61.152.95.172 Jul 13 08:13:14 localhost sshd[14711]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=222.73.173.143 user=root Jul 13 08:13:17 localhost sshd[14722]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=61.152.95.172 Jul 13 08:13:17 localhost sshd[14724]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=222.73.173.143 Jul 13 08:13:20 localhost sshd[14739]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=61.152.95.172 user=root Jul 13 08:13:23 localhost sshd[14753]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=61.152.95.172 user=root Jul 13 08:13:26 localhost sshd[14767]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=61.152.95.172 Jul 13 08:13:29 localhost sshd[14781]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=61.152.95.172 Jul 13 08:13:32 localhost sshd[14795]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=61.152.95.172 Jul 13 08:13:35 localhost sshd[14809]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=61.152.95.172 Jul 13 08:13:38 localhost sshd[14823]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=61.152.95.172 Jul 13 08:13:41 localhost sshd[14837]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=61.152.95.172 user=apache Jul 13 08:13:44 localhost sshd[14851]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=61.152.95.172 Jul 13 08:13:47 localhost sshd[14865]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=61.152.95.172 Jul 13 08:13:49 localhost sshd[14876]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=61.152.95.172 Jul 13 08:13:53 localhost sshd[14895]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=61.152.95.172
方法1：利用awk命令進行篩選
[root@C67-X64-A1 hanghang]# awk -F "rhost=" '{print $NF}' test.txt |awk '{print $1'}|sort -r|uniq
61.152.95.172
222.73.173.143
方法2：利用grep的擴展命令egrep進行篩選
[root@C67-X64-A1 hanghang]# egrep -o '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}' test.txt |sort -r|uniq
61.152.95.172
222.73.173.143
方法3：利用sed命令進行篩選
[root@C67-X64-A1 hanghang]# sed -nr 's/.*[^0-9](([0-9]+\.){3}[0-9]+).*/\1/p' test.txt |sort -r|uniq
61.152.95.172
222.73.173.143
[root@C67-X64-A1 hanghang]# sed -nr 's/(^|.*[^0-9])(([0-9]+\.){3}[0-9]+).*/\2/p' test.txt |sort -r|uniq
61.152.95.172
222.73.173.143
案例二：根據需求對日志信息進行篩選
需求：
最近需要處理下網站日志：
例如
A 1.1.1.1 用戶訪問 有index.html 和a.jpg 的日志
B 20.20.20.20 用戶訪問 有index.html 的日志 沒其他文件記錄的日志
現在需要提取B的IP 不需要A的IP
日志信息如下：
[root@C67-X64-A1 hanghang]# cat files 1.1.1.1 - - [19/Jul/2013:15:01:39 +0800] "GET /index.html HTTP/1.1 10.10.10.10 - - [19/Jul/2013:15:01:39 +0800] "GET /index.html HTTP/1.1 10.10.10.10 - - [19/Jul/2013:15:01:39 +0800] "GET /logo.jpg HTTP/1.1 10.10.10.10 - - [19/Jul/2013:15:01:39 +0800] "GET /a.js HTTP/1.1 3.3.3.3 - - [19/Jul/2013:15:01:39 +0800] "GET /index.html HTTP/1.1 20.20.20.20 - - [19/Jul/2013:15:01:39 +0800] "GET /index.html HTTP/1.1 20.20.20.20 - - [19/Jul/2013:15:01:39 +0800] "GET /logo.jpg HTTP/1.1 20.20.20.20 - - [19/Jul/2013:15:01:39 +0800] "GET /a.js HTTP/1.1 30.30.30.30 - - [19/Jul/2013:15:01:39 +0800] "GET /index.html HTTP/1.1 30.30.30.30 - - [19/Jul/2013:15:01:39 +0800] "GET /logo.jpg HTTP/1.1 30.30.30.30 - - [19/Jul/2013:15:01:39 +0800] "GET /a.js HTTP/1.1 4.4.4.4 - - [19/Jul/2013:15:01:39 +0800] "GET /index.html HTTP/1.1 5.5.5.5 - - [19/Jul/2013:15:01:39 +0800] "GET /index.html HTTP/1.1 1.1.1.1 - - [20/Jul/2013:15:01:39 +0800] "GET /index.html HTTP/1.1 2.2.2.2 - - [21/Jul/2013:15:01:39 +0800] "GET /index.html HTTP/1.1 3.3.3.3 - - [21/Jul/2013:15:01:55 +0800] "GET /index.html HTTP/1.1 4.4.4.4 - - [21/Jul/2013:16:01:55 +0800] "GET /index.html HTTP/1.1 5.5.5.5 - - [21/Jul/2013:17:02:55 +0800] "GET /index.html HTTP/1.1
Shell腳本實現：
#!/bin/bash
#author molewan
for i in `grep -v "/index.html" files | awk '{print $1}' | uniq`;do
echo "| grep -v "$i" " >> tmp_title
done
M=`cat tmp_title | tr "\n" " " | sed 's#^#cat files | sort -r | uniq#'`
echo $M | bash | awk '{print $1}'
rm -rf tmp_title
Python腳本實現：
假設日志信息是放在文件log.dat里面的：
#! /usr/bin/env python
import re
Dip_reso = {}
pattern = re.compile('(\d+\.\d+\.\d+\.\d+).*GET /(.*) .*')
f = open('log.dat')
for line in f:
resource = re.match(pattern, line)
key = resource.group(1)
value = resource.group(2)
if key in Dip_reso:
if value not in Dip_reso[key]:
Dip_reso[key].append(value)
else:
continue
else:
Dip_reso[key] = []
Dip_reso[key].append(value)
f.close()
for k in Dip_reso:
if len(Dip_reso[k]) == 1 and cmp(Dip_reso[k][0], 'index.html') == 1:
print k
#如果你要搜集數據，可以這樣
# ip_data = [ip for ip in Dip_reso if len(Dip_reso[ip]) == 1 and cmp(Dip_reso[ip][0], 'index.html') == 1]
這樣，ip_data就是所有的ip了。
原創文章，作者：Net21-冰凍vs西瓜，如若轉載，請注明出處：http://www.www58058.com/24749
