May 052018
 

We have all lost a hard drive at one time or another on a laptop or desktop computer and it always seems like it happens right after several weeks of not performing backups.  Last year, I lost about 15 years of research on an external drive that failed.  I had this system that has worked as long as I can remember where I simply swapped an external drive every two years with a new one after copying the data.  What failed on me though was I became over-confident in this system and wiped out the older drives in order to make room for something else, meanwhile the current drive decided to barf after only about 6 months of usage … literally within a couple weeks of me wiping the previous drives clean.  I was pretty pissed to say the least.  So, lesson learned, I decided to implement a better backup plan.  I wanted a way that would work and be simple.  Instead of a file server and transferring data over a wire, I wanted an external drive I could plug-in and leave plugged in while working or at home or in some motel.  I wanted full backups and I wanted it to be incremental to save space.  This was how I accomplished these tasks …

Continue reading »

May 012017
 

I’ve wrote about this before in Using Sed to search between dates and offered a ad-hoc solution but the other day I came up with a much better solution using a little known option of ‘date’ command.  Using this new method, you just pass the time in minutes prior to current time.  I.e. if you want the last hour, you would simply type ‘./sed_time.sh 60’ and it will spit out the correctly formatted sed command like this:

$ ./sed_time.sh 60
sed -n '/01\/May\/2017\:07\:16\:15/,/01\/May\/2017\:08\:16\:15/ p'

Continue reading »

Feb 172016
 

This is a silly script but you would be surprised how many times a day I have to do this and no matter how many times I type the command, I always get it wrong (or more than likely I forget to escape something).  Its also interesting to note that the scripts I find silly are usually the ones that are the most popular on this site….so here it is.

Basically, if you copy and paste this script into a file and run it, it will give you the exact date and time in the sed command to run to search all lines in a log file from the previous hour to now and save it to another file.

Continue reading »

Mar 162013
 

So one thing you learn pretty quick once you move into the cloud, is that what you normally would do to stop bots, rogue traffic, hackers, etc doesn’t quit work…even at the packet level.  What I mean is this, you have a web server sitting behind an aws load balancer and its under attack.  You are running linux.  First thing you do is set up an iptables rule to drop connections from that ip address.  The problem is, iptables never sees that ip address.  Iptables can’t look into packets.  Instead, it sees the load balancer ip address.  The ip address of the user is hidden in the x-forward-for.  So first thing you need to do is enable x-forward-for logging in your web server.  I will use nginx as an example:

Continue reading »

Mar 092013
 

As I use nginx more and more, one of the things I miss is being able to see who is connecting to the server and the request they are making.  This is often helpful in determining attacks.  So, I basically wrote this script which does it.

#!/bin/bash
 
# By Ed Wiget
# This shows active GET and POSTS to port 80
# One of the things i hate about nginx is the lack of an apache style status page showing requests
# hence I wront this script....which does it at a network layer, using ngrep
 
# 20130308 - original script
 
## grep all HTTP GET or POST requests from network traffic on eth0 interface  ##
# sudo ngrep -l -q -d eth0 "^GET |^POST " tcp and port 80
 
if [ `which ngrep | head -1 | wc -l` == "0" ]; then
	echo "missing ngrep....please install"
	exit
fi
 
if [ $1 = "" ]; then
	echo "You must pass the interface as an option, i.e. eth0 or eth1, etc"
	read CIF
else
	CIF=$1
fi
 
thing=1
 
until [ $thing = "0" ]; do
	# uncomment top line if you dont need x-forward-for
        #sudo ngrep -l -q -d eth0 "^GET |^POST " tcp and port 80
        # use the next line if you do need x-forward-for
	#sudo ngrep -d eth1 -q 'X-Forwarded-For'  tcp and port 80
	ngrep -d ${CIF} -t '^(GET|POST) ' tcp and port 80
	echo
	sleep 1;
done

CHANGES

20130905 – added option to pass interface

Mar 062013
 

This was tested on centos 6.3.  It is running at approx 900 – 3,000+ log events per second from approx 30 hosts.

Current load is about 900 messages per second:  load average: 1.57, 1.35, 1.29 with 8GB memory.

With the above in mind, there was approx 165GB of log data after running for 4 days.

graylog.org web site

elasticsearch web site

mongo db

passenger phusion web site

logstash web site

I wrote a script in order to install a graylog2 central log server.  Its a one shot run and be done kinda thing…..

CHANGES20130309 – see notes in script

Continue reading »

Feb 052013
 

So I wrote this script because I often need to run tcpdump on a remote host and then view it in wireshark.  The old method was to run tcpdump on remote host, scp/rsync the file back to my local machine, open it in wireshark, view it.  This script saves a lot of time.  It assumes you are logging in as root and will need modified if you are running as a normal user (change root to your username and make sure you have sudo privileges for tcpdump)

#!/bin/bash
 
# By Ed Wiget
# This runs tcpdump on a remote hosts and pipes it back locally to wireshark to view in realtime
 
# 20130205 - original script
 
if [ $1 == "" ]; then
	echo "What is the remote host by fqdn, i.e. server1.domain.com"
	read RHOST
else
	RHOST=$1
fi
 
wireshark -k -i <( ssh -l root ${RHOST} /usr/sbin/tcpdump -i eth0 -w - )
 
# after you kill wireshark, the tcpdump still runs on remove host...we need to kill it
PIDOF=`ssh root@${RHOST} "ps aux | grep [t]cpdump" | awk -F" " '{print$2}'`
 
echo "killing pid ${PIDOF} on ${RHOST}...please wait...."
ssh root@${RHOST} "pkill tcpdump"
 
# now we make sure it is killed
PIDOF2=`ssh root@${RHOST} "ps aux | grep [t]cpdump" | awk -F" " '{print$2}'`
if [ ${PIDOF2} == "" ]; then
	echo "pid check returns ${PIDOF2}"
else
	echo "pid check returns ${PIDOF2}"
fi