This is a silly script but you would be surprised how many times a day I have to do this and no matter how many times I type the command, I always get it wrong (or more than likely I forget to escape something). Its also interesting to note that the scripts I find silly are usually the ones that are the most popular on this site….so here it is.
Basically, if you copy and paste this script into a file and run it, it will give you the exact date and time in the sed command to run to search all lines in a log file from the previous hour to now and save it to another file.
#!/bin/bash
#===============================================================================
#
# FILE: determine-time.sh
#
# USAGE: ./determine-time.sh
#
# DESCRIPTION: determines current date and time and previous hour for searching
# files between date and time using sed (example does previous hour)
# REQUIREMENTS: sed
# AUTHOR: Ed Wiget
# VERSION: 1.0
# CREATED: 02/17/2016 04:10:53 PM EST
#
#===============================================================================
# current year
CY=`date +%Y`
# current month
CM=`date +%b`
# current day of month
CD=`date +%d`
# current hour
CH=`date +%H`
# current minute
CMH=`date +%M`
# current second
CS=`date +%S`
echo "The current date and time is: ${CY}-${CM}-${CD} ${CH}:${CMH}:${CS}"
echo ""
# now determine previous hour
PH=`expr ${CH} - 1 `
echo "the previous hour is: ${PH}"
echo ""
# for sed we need this format
# sed -n '/27\/Mar\/2013\:16\:00\:00/,/27\/Mar\/2013\:16\:16\:00/ p' merged.log > 12_00_00-12_16_00EST_16_00_00-16_16_00UTC.txt
echo "our search beginning time in sed is:"
echo "/${CD}\/${CM}\/${CY}\:${PH}\:${CMH}\:${CS}/"
echo ""
echo "our search ending time in sed is:"
echo "/${CD}\/${CM}\/${CY}\:${CH}\:${CMH}\:${CS}/"
echo ""
echo "The entire sed command to search logs from previous hour to now is:"
echo "sed -n '/${CD}\/${CM}\/${CY}\:${PH}\:${CMH}\:${CS}/,/${CD}\/${CM}\/${CY}\:${CH}\:${CMH}\:${CS}/ p' /path/to/log/filename > /path/to/output_filename.txt"
So when you run it, it basically looks like this:
$ scripts/determine-time.sh
The current date and time is: 2016-Feb-17 16:50:51
the previous hour is: 15
our search beginning time in sed is:
/17\/Feb\/2016\:15\:50\:51/
our search ending time in sed is:
/17\/Feb\/2016\:16\:50\:51/
The entire sed command to search logs from previous hour to now is:
sed -n '/17\/Feb\/2016\:15\:50\:51/,/17\/Feb\/2016\:16\:50\:51/ p' /path/to/log/filename > /path/to/output_filename.txtThis is a much better version of the above script …
#!/bin/bash
#===============================================================================
#
# FILE: sed_time2.sh
#
# USAGE: ./sed_time2.sh
#
# DESCRIPTION:
#
# OPTIONS: ---
# REQUIREMENTS: ---
# BUGS: ---
# NOTES: ---
# AUTHOR: Ed Wiget (), [email protected]
# COMPANY:
# VERSION: 1.0
# CREATED: 04/28/2017 11:25:39 AM EDT
# REVISION: ---
#===============================================================================
if [ "$1" = "" ]; then
echo "how long ago as starting time in minutes, i.e. 30 for 30 minutes"
read PTIME
else
PTIME=$1
fi
# current time
DNOW=`date +%d/%b/%Y:%H:%M:%S`
# time formatted escaped sed ready
ESCAPED_DNOW=`echo ${DNOW} | sed 's,/,\\\/,g' | sed 's,:,\\\:,g'`
# previous time
PDATE=`date +%d/%b/%Y:%H:%M:%S --date="${PTIME} minutes ago"`
# time formatted escaped sed ready
ESCAPED_PDATE=`echo ${PDATE} | sed 's,/,\\\/,g' | sed 's,:,\\\:,g'`
# now we can simply echo the sed line
echo "sed -n '/${ESCAPED_PDATE}/,/${ESCAPED_DNOW}/ p'"
One Response to “Using sed to search between two dates/time in a log file (apache, nginx, etc)”
Leave a Reply
This site uses Akismet to reduce spam. Learn how your comment data is processed.
[…] wrote about this before in Using Sed to search between dates and offered a ad-hoc solution but the other day I came up with a much better solution using a […]