Python小程序
需求：统计每分钟有多少行；及23:59有多少行，00:00有多少行，文件过大每个文件2G，共5个文件
文件：access_log.2017-11-06-00_00_00文件内容如下：
21.123.42.233 - - [05/Nov/2017:23:59:59 +0800] "POST /IPPSRemote/remoting/RemoteCfcaQryCert4softCertInvoker HTTP/1.1" 200 TIME: 11637 1388 21.123.42.233 - - [06/Nov/2017:00:00:00 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1VerifySignMsgInvoker HTTP/1.1" 200 TIME: 7223 150 21.123.42.235 - - [05/Nov/2017:23:59:59 +0800] "POST /IPPSRemote/remoting/RemoteIps35062Invoker HTTP/1.1" 200 TIME: 185121 135 21.123.42.235 - - [06/Nov/2017:00:00:00 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1SignMsgInvoker HTTP/1.1" 200 TIME: 7998 330 21.123.42.232 - - [06/Nov/2017:00:00:00 +0800] "POST /IPPSRemote/remoting/RemoteCfcaQryCert4softCertInvoker HTTP/1.1" 200 TIME: 29833 1388 21.123.42.232 - - [06/Nov/2017:00:00:00 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1VerifySignMsgInvoker HTTP/1.1" 200 TIME: 20608 150 21.123.42.232 - - [05/Nov/2017:23:59:59 +0800] "POST /IPPSRemote/remoting/RemoteIps35062Invoker HTTP/1.1" 200 TIME: 252234 139 21.123.42.232 - - [05/Nov/2017:23:59:59 +0800] "POST /IPPSRemote/remoting/RemoteIps35062Invoker HTTP/1.1" 200 TIME: 230661 135 21.123.42.232 - - [06/Nov/2017:00:00:00 +0800] "POST /IPPSRemote/remoting/RemoteCfcaQryCert4softCertInvoker HTTP/1.1" 200 TIME: 33536 1388 21.123.42.233 - - [06/Nov/2017:00:00:00 +0800] "POST /IPPSRemote/remoting/RemoteIps35062Invoker HTTP/1.1" 200 TIME: 138429 139 21.123.42.232 - - [06/Nov/2017:00:00:00 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1VerifySignMsgInvoker HTTP/1.1" 200 TIME: 24368 150 21.123.42.233 - - [06/Nov/2017:00:00:00 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1SignMsgInvoker HTTP/1.1" 200 TIME: 7075 330 21.123.42.232 - - [06/Nov/2017:00:00:00 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1SignMsgInvoker HTTP/1.1" 200 TIME: 23221 330 21.123.42.232 - - [06/Nov/2017:00:00:00 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1SignMsgInvoker HTTP/1.1" 200 TIME: 19719 330 21.123.42.232 - - [06/Nov/2017:00:00:00 +0800] "POST /IPPSRemote/remoting/RemoteIps35062Invoker HTTP/1.1" 200 TIME: 85528 135 21.123.42.232 - - [06/Nov/2017:00:00:00 +0800] "POST /IPPSRemote/remoting/RemoteCfcaQryCert4softCertInvoker HTTP/1.1" 200 TIME: 22324 1388 21.123.42.232 - - [06/Nov/2017:00:00:00 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1VerifySignMsgInvoker HTTP/1.1" 200 TIME: 15338 150 21.123.42.232 - - [06/Nov/2017:00:00:00 +0800] "POST /IPPSRemote/remoting/RemoteCfcaQryCert4softCertInvoker HTTP/1.1" 200 TIME: 20461 1388 21.123.42.232 - - [06/Nov/2017:00:00:00 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1SignMsgInvoker HTTP/1.1" 200 TIME: 24474 330 21.123.42.235 - - [06/Nov/2017:00:00:00 +0800] "POST /IPPSRemote/remoting/RemoteCfcaQryCert4softCertInvoker HTTP/1.1" 200 TIME: 10589 1388 21.123.42.232 - - [06/Nov/2017:00:00:00 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1VerifySignMsgInvoker HTTP/1.1" 200 TIME: 24163 150 21.123.42.235 - - [06/Nov/2017:00:00:00 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1VerifySignMsgInvoker HTTP/1.1" 200 TIME: 4449 150 21.123.42.236 - - [06/Nov/2017:00:00:00 +0800] "POST /IPPSRemote/remoting/RemoteIps35062Invoker HTTP/1.1" 200 TIME: 345698 139 21.123.42.236 - - [06/Nov/2017:00:00:00 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1SignMsgInvoker HTTP/1.1" 200 TIME: 19886 330 21.123.42.233 - - [06/Nov/2017:04:02:04 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1VerifySignMsgInvoker HTTP/1.1" 200 TIME: 5096 150 21.123.42.232 - - [06/Nov/2017:04:02:04 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1VerifySignMsgInvoker HTTP/1.1" 200 TIME: 5755 150 21.123.42.235 - - [06/Nov/2017:04:02:04 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1SignMsgInvoker HTTP/1.1" 200 TIME: 7182 330 21.123.42.235 - - [06/Nov/2017:04:02:04 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1VerifySignMsgInvoker HTTP/1.1" 200 TIME: 4752 150 21.123.42.233 - - [06/Nov/2017:04:02:04 +0800] "POST /IPPSRemote/remoting/RemoteIps35062Invoker HTTP/1.1" 200 TIME: 101802 139 21.123.42.232 - - [06/Nov/2017:04:02:04 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1SignMsgInvoker HTTP/1.1" 200 TIME: 7666 330 21.123.42.236 - - [06/Nov/2017:04:02:04 +0800] "POST /IPPSRemote/remoting/RemoteCfcaQryCert4softCertInvoker HTTP/1.1" 200 TIME: 11735 1388 21.123.42.232 - - [06/Nov/2017:04:02:04 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1VerifySignMsgInvoker HTTP/1.1" 200 TIME: 5376 150 21.123.42.234 - - [06/Nov/2017:04:02:04 +0800] "POST /IPPSRemote/remoting/RemoteCfcaQryCert4softCertInvoker HTTP/1.1" 200 TIME: 19922 1388 21.123.42.236 - - [06/Nov/2017:04:02:04 +0800] "POST /IPPSRemote/remoting/RemoteCfcaQryCert4softCertInvoker HTTP/1.1" 200 TIME: 8779 1388 21.123.42.235 - - [06/Nov/2017:04:02:04 +0800] "POST /IPPSRemote/remoting/RemoteIps35062Invoker HTTP/1.1" 200 TIME: 99890 135 21.123.42.234 - - [06/Nov/2017:04:02:04 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1VerifySignMsgInvoker HTTP/1.1" 200 TIME: 5242 150 21.123.42.236 - - [06/Nov/2017:04:02:04 +0800] "POST /IPPSRemote/remoting/RemoteCfcaQryCert4softCertInvoker HTTP/1.1" 200 TIME: 16658 1388 21.123.42.233 - - [06/Nov/2017:04:02:04 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1VerifySignMsgInvoker HTTP/1.1" 200 TIME: 4977 150 21.123.42.234 - - [06/Nov/2017:12:12:32 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1SignMsgInvoker HTTP/1.1" 200 TIME: 5559 3 21.123.42.232 - - [06/Nov/2017:12:12:32 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1SignMsgInvoker HTTP/1.1" 200 TIME: 5176 3 21.123.42.234 - - [06/Nov/2017:12:12:32 +0800] "POST /IPPSRemote/remoting/RemoteCfcaQryCert4softCertInvoker HTTP/1.1" 200 TIME: 21.123.42.234 - - [06/Nov/2017:12:12:32 +0800] "POST /IPPSRemote/remoting/RemoteCfcaQryCert4softCertInvoker HTTP/1.1" 200 TIME: 21.123.42.234 - - [06/Nov/2017:12:12:32 +0800] "POST /IPPSRemote/remoting/RemoteCfcaQryCert4softCertInvoker HTTP/1.1" 200 TIME: 21.123.42.234 - - [06/Nov/2017:12:12:32 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1VerifySignMsgInvoker HTTP/1.1" 200 TIME: 21.123.42.234 - - [06/Nov/2017:12:12:32 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1VerifySignMsgInvoker HTTP/1.1" 200 TIME: 21.123.42.235 - - [06/Nov/2017:12:12:31 +0800] "POST /IPPSRemote/remoting/RemoteIps35062Invoker HTTP/1.1" 200 TIME: 136333 135 21.123.42.234 - - [06/Nov/2017:12:12:32 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1SignMsgInvoker HTTP/1.1" 200 TIME: 8429 3 21.123.42.233 - - [06/Nov/2017:12:12:32 +0800] "POST /IPPSRemote/remoting/RemoteCfcaQryCert4softCertInvoker HTTP/1.1" 200 TIME: 21.123.42.236 - - [06/Nov/2017:12:12:31 +0800] "POST /IPPSRemote/remoting/RemoteIps35062Invoker HTTP/1.1" 200 TIME: 147556 135 21.123.42.233 - - [06/Nov/2017:12:12:32 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1VerifySignMsgInvoker HTTP/1.1" 200 TIME: 21.123.42.235 - - [06/Nov/2017:12:12:32 +0800] "POST /IPPSRemote/remoting/RemoteIps35062Invoker HTTP/1.1" 200 TIME: 119417 135 21.123.42.233 - - [06/Nov/2017:12:12:31 +0800] "POST /IPPSRemote/remoting/RemoteIps35062Invoker HTTP/1.1" 200 TIME: 194970 135 21.123.42.235 - - [06/Nov/2017:12:12:32 +0800] "POST /IPPSRemote/remoting/RemoteIps35062Invoker HTTP/1.1" 200 TIME: 76006 135 21.123.42.236 - - [06/Nov/2017:12:12:32 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1SignMsgInvoker HTTP/1.1" 200 TIME: 5892 3 21.123.42.233 - - [06/Nov/2017:12:12:32 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1SignMsgInvoker HTTP/1.1" 200 TIME: 4904 3 21.123.42.235 - - [06/Nov/2017:12:12:31 +0800] "POST /IPPSRemote/remoting/RemoteIps35963Invoker HTTP/1.1" 200 TIME: 297515 369 21.123.42.236 - - [06/Nov/2017:12:12:31 +0800] "POST /IPPSRemote/remoting/RemoteIps35062Invoker HTTP/1.1" 200 TIME: 402771 135 21.123.42.233 - - [06/Nov/2017:12:12:32 +0800] "POST /IPPSRemote/remoting/RemoteCfcaQryCert4softCertInvoker HTTP/1.1" 200 TIME: 21.123.42.234 - - [06/Nov/2017:12:12:32 +0800] "POST /IPPSRemote/remoting/RemoteIps35062Invoker HTTP/1.1" 200 TIME: 135743 135 21.123.42.235 - - [06/Nov/2017:12:12:32 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1SignMsgInvoker HTTP/1.1" 200 TIME: 5853 3 21.123.42.235 - - [06/Nov/2017:12:12:32 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1SignMsgInvoker HTTP/1.1" 200 TIME: 14536 21.123.42.233 - - [06/Nov/2017:12:12:32 +0800] "POST /IPPSRemote/remoting/RemoteCFCAP1VerifySignMsgInvoker HTTP/1.1" 200 TIME: 21.123.42.233 - - [06/Nov/2017:12:12:32 +0800] "POST /IPPSRemote/remoting/RemoteCfcaQryCert4softCertInvoker HTTP/1.1" 200 TIME: 21.123.42.233 - - [06/Nov/2017:12:12:31 +0800] "POST /IPPSRemote/remoting/RemoteIps35062Invoker HTTP/1.1" 200 TIME: 186202 135
Python代码如下所示：
#!/usr/bin/env python # -*- coding:utf-8 -*- __author__ = '40kuai' li = [0 for i in range(1440)] with open('/root/access_log.2017-11-06-00_00_00') as file: for line in file: hour = line.split(':')[1] mins = line.split(':')[2] haha = int(hour)*60+int(mins) li[haha] +=1 for index,i in enumerate(li): try: a,b=divmod(index,60) print '%s:%s' % (a, b), i except ZeroDivisionError as e: print '00:00', i
又用Shell编了一个简单Shell脚本，脚本如下：
#!/bin/bash
for i in {00..23} ;do
for j in {00..59} ;do
echo $i:$j `awk '{print $4}' /root/access_log.2017-11-06-00_00_00|awk -F ':' '{print $2$3}'|grep $i$j|wc -l`>>/lines.txt
done
done
两个脚本的效率差别很大。。。哪位大神可以帮忙看看能不能优化一下Shell脚本，Shell主要问题为循环读文件进行过滤
