Quick Nginx Log Analyzer

This is another way to quickly analyze nginx logs. It will spit out the top 25 ip’s, domains, requests and some other data. You may need to change the array value to match the format of your nginx logs.

It uses perl so it is very fast and takes just seconds to analyze hundreds of thousands of lines in a log file. It can also be used for apache too or other column whitespaced logs.

Here is the script:

[codesyntax lang=”bash”]

#!/bin/bash

# enable for debug
#set -x

# gets all the usefull information from a log you specify

### Values - see sample request below
# 0 ip
# 1 domain
# 7 request
# 9 http status code
# 11 referrer
# 12 request

# sample request
# since this is an array, perl counts from 0
# 127.0.0.1 www.abc.com 127.0.0.1 - [25/Apr/2017:06:00:26 +0000] "GET /page.php?somevar=3 HTTP/1.1" 200 619 "http://www.somereferrer.com/url/" "-" "0.000"

if [ "$1" = "" ]; then
	echo "what is the full path to the log? i.e. /some/path/my.log"
	read logfile
else
	logfile=$1
fi

# date for logfile - this is used in the output log file name
NDATE=`date +%Y%m%d-%H%M%S`

# we need to replace / with _ in logfile name for OFILE
# this is also used in the output log files name
logfile2=`echo ${logfile} | tr '/' '_'`

CWD=`pwd`

# output log file
OFILE=${CWD}/${NDATE}-${logfile2}.txt
echo "###############################################################################" >> ${OFILE}
echo "#### TOP 25 REQUESTING IPS" >> ${OFILE}
echo "###############################################################################" >> ${OFILE}
perl -e '$ip{(split)[0]}++ while <>; print map "$_ : $ip{$_}\n", sort {$ip{$b} <=> $ip{$a}} keys %ip' ${logfile} | head -25 >> ${OFILE}
echo "###############################################################################" >> ${OFILE}
echo "#### TOP 25 VISITED DOMAINS" >> ${OFILE}
echo "###############################################################################" >> ${OFILE}
perl -e '$ip{(split)[1]}++ while <>; print map "$_ : $ip{$_}\n", sort {$ip{$b} <=> $ip{$a}} keys %ip' ${logfile} | head -25 >> ${OFILE}
echo "###############################################################################" >> ${OFILE}
echo "#### TOP 25 REQUESTED URLS " >> ${OFILE}
echo "###############################################################################" >> ${OFILE}
perl -e '$ip{(split)[7]}++ while <>; print map "$_ : $ip{$_}\n", sort {$ip{$b} <=> $ip{$a}} keys %ip' ${logfile} | head -25 >> ${OFILE}
echo "###############################################################################" >> ${OFILE}
echo "#### TOP 25 STATUS CODES" >> ${OFILE}
echo "###############################################################################" >> ${OFILE}
perl -e '$ip{(split)[9]}++ while <>; print map "$_ : $ip{$_}\n", sort {$ip{$b} <=> $ip{$a}} keys %ip' ${logfile} | head -25 >> ${OFILE}
echo "###############################################################################" >> ${OFILE}
echo "#### TOP 25 REFERRERS" >> ${OFILE}
echo "###############################################################################" >> ${OFILE}
perl -e '$ip{(split)[11]}++ while <>; print map "$_ : $ip{$_}\n", sort {$ip{$b} <=> $ip{$a}} keys %ip' ${logfile} | head -25 >> ${OFILE}
echo "###############################################################################" >> ${OFILE}
echo "#### TOP 25 DATA" >> ${OFILE}
echo "###############################################################################" >> ${OFILE}
perl -e '$ip{(split)[12]}++ while <>; print map "$_ : $ip{$_}\n", sort {$ip{$b} <=> $ip{$a}} keys %ip' ${logfile} | head -25 >> ${OFILE}
echo ""
echo "report file is available at ${OFILE}"

[/codesyntax]

And here is example output of the above:

(todo)

Leave a Reply Cancel reply