ÈÐ Wïgê† Ðð† ñåmê

ïñ£ð§ê¢, ïñ£ðrmå†ïðñ §ê¢µr, Ðïgï†ål £ðrêñ§ï¢§, hå¢kïñg, §¥§†êm åÐmïñ阮rå†ïðñ, lïñµx ßlðg

Dealing with openx append prepend malware

, , , , , ,

Openx has been a pain in my ass for some time now (5 years).  Even if you have the latest most up to date software release, you will still get append and prepend infections.  I’m not sure if it comes from client browsers when they log in or some other reason.  What I can assure you is that the file system in which openx resides is as secure as it can be while leaving openx functional (all files are owned by a different user than the web server process and are only readable by the web server.  All directories, except two, are also owned by a different process than the web server and are read only….while two have to be writable by the web server process.  The lamp stack is also up to date.).  Anyways, even with these restrictions, clean code, clean db, limited plugins, and even checked the meta data of all image files for backdoors (I first learned about this technique in approx 2010 but here is an article from 2011 detailing this – PHP Code into JPEG Metadata: From hide to unhide ) we still get an occasional append/prepend infection.

How to stop it?  This is pretty easy, I simply wrote a script that checks for append/prepend problems, logs if clean, logs and alerts if infected, and also disinfects.  This only works, if the append and prepend is NOT being used in your ads.

So I wrote this script that checks the append and prepend columns in both tables.  It was written for a redhat/centos server although I believe the requirements should work with any *Nix.  Basically you need the curl, logger, mail, mutt, mysql, mysqldump, ss binaries.

Check Script

[codesyntax lang=”bash”]

#!/bin/bash

# By Ed Wiget

# This script checks for malware-like entries in ox_banners and ox_zones append and prepend columns
# requires:  logger, mail, mutt, ss, mysql, mysqldump binaries

# 20110314 - original script
# 20131004 - added openx error and access logs to include last 100 lines
#          - added debug log 
# 20131118 - published version on edwiget.name

# enable next line for debugging
#set -x

# where is the logger binary
LOGR=`which logger`

# make sure we have everything for this script
which logger
if [ "$?" = "1" ]; then
	echo "logger does not exist"
	exit 1
fi 

which mail
if [ "$?" = "1" ]; then
	echo "mail does not exist"
	exit 1
fi 

which mutt
if [ "$?" = "1" ]; then
	echo "mutt does not exist"
	exit 1
fi 

which ss
if [ "$?" = "1" ]; then
	echo "ss does not exist"
	exit 1
fi 

which mysql 
if [ "$?" = "1" ]; then
	echo "mysql does not exist"
	exit 1
fi 

which mysqldump
if [ "$?" = "1" ]; then
	echo "mysqldump does not exist"
	exit 1
fi 

which curl
if [ "$?" = "1" ]; then
	echo "curl does not exist"
	exit 1
fi 
###################################################################################
##### YOU PROB NEED TO MODIFY THESE BELOW
###################################################################################

# alert messages sent in sms
STRING="OPENX ALERT: check ox_banners append and prepend columns for malware"
STRING2="OPENX ALERT: check ox_zones append and prepend columns for malware"

# emails to receive alerts
[email protected],[email protected]

# cell phone number to be sent sms alerts - find the format of sms numbers here:
# http://martinfitzpatrick.name/article/using-email-to-sms-gateways-to-send-free-sms
[email protected]
[email protected]

# DB Credentials
# database user for openx 
DB_USER='dbuser'
# DB_USERS PASSWORD
DB_PASS='password'
# Database Name
DB_NAME='openx'

# Path to openx installation, i.e. /home/www/openx/htdocs
OX_PATH=/home/www/openx/htdocs

# Where are the apache web server access/error logs, i.e. /var/log/www
SVR_LOGS=/var/log/www

# Path for logs collected during cleanup
# NOTE: this is the base path.  Directories with the time will be created inside to hold the logs
# the format will be ${SEC_LOGS}/${LTIME} , i.e. /root/openx_logs/20111023-112315 for nov 23, 2011 at 11:23:15
SEC_LOGS=/root/openx_logs

# if you have the apache status page enabled, what address does the server listen on?
SVR=127.0.0.1

# also what is the name of the apache status page, default is server-status
ASP=server-status

# path to the openx debug log, i.e. /home/www/openx/htdocs/var/debug.log 
DEBUG_LOG=/home/www/openx/htdocs/var/debug.log 

###################################################################################
##### START CODE
###################################################################################

# make sure our sec logs directory exists
if [ ! -d ${SEC_LOGS} ]; then
	mkdir -p ${SEC_LOGS}
fi 

# we actuall start all the checks here
${LOGR} "OPENX: starting malware check"

###################################################################################
##### ox_banners check 
###################################################################################
if [ "`mysql -u${DB_USER} -p${DB_PASS} -Be "select * from ${DB_NAME}.ox_banners where append != '' OR prepend != '';" | wc -l`" -ne "0" ]; then

	# log we are not clean
	${LOGR} "${STRING}"

	# set the time to now
	LTIME=`date +%Y%m%d-%H%M%S`

	# set a new dir to hold all logs
	LOGDIR=${SEC_LOGS}/${LTIME}
	mkdir -p ${LOGDIR}

	# send an alert sms to below
	echo ${STRING} | mail ${SMS_ALERT} # comment full name, i.e. jane doe
	#echo ${STRING} | mail ${SMS_ALERT2} # comment optional user full name

	# perform a dump of the table
	mysqldump -u${DB_USER} -p${DB_PASS} ${DB_NAME} ox_banners | gzip > ${LOGDIR}/${DB_NAME}_ox_banners-${LTIME}.sql.gz
	${LOGR} "${DB_NAME} OX_BANNERS with ${LTIME} dumped to ${LOGDIR}/${DB_NAME}_ox_banners-${LTIME}.sql.gz"
	mysqldump -u${DB_USER} -p${DB_PASS} ${DB_NAME} ox_audit | gzip > ${LOGDIR}/${DB_NAME}_ox_audit-${LTIME}.sql.gz
        ${LOGR} "${DB_NAME} OX_AUDIT with ${LTIME} dumped to ${LOGDIR}/${DB_NAME}_ox_audit-${LTIME}.sql.gz"

	# log who is on the system
	ss -o state established '( dport = :http or sport = :http )' | awk -F" " '{print $4}' | awk -F: '{print $1}' | uniq | grep -v Address > ${LOGDIR}/whoison-${LTIME}.txt

        # openx files changed in last 24 hours
	find ${OX_PATH}/ -type f -ctime -1 | grep -v cache > ${LOGDIR}/files-${LTIME}.txt

	# dump the admin users for review
	mysql -u${DB_USER} -p${DB_PASS} -Be "use ${DB_NAME}; SELECT u.user_id, u.contact_name, u.email_address, u.username FROM ox_users AS u, ox_account_user_assoc AS aua WHERE u.user_id=aua.user_id AND aua.account_id = (SELECT value FROM ox_application_variable WHERE name='admin_account_id');" > ${LOGDIR}/openx_admin_users-${LTIME}.txt

	# use curl to grab the status page
        curl -o ${LOGDIR}/${SVR}_${ASP}-status-${LTIME} http://${SVR}/${ASP}

	# grab the access and error logs
        for file in `ls -lt ${SVR_LOGS}/ | head -2 | awk '{print$9}'`; do
                tail -100 ${file} >> ${LOGDIR}/access_error_last100-${LTIME}.txt
        done

	# clean up the db
	${LOGR} "${DB_NAME} OX_BANNERS being cleaned up now"
	mysql -u${DB_USER} -p${DB_PASS} -Be "UPDATE ${DB_NAME}.ox_banners SET append = null, prepend = null WHERE append != '' OR prepend != '';"

        # grab the openx debug log too
	bzip2 -9 -c ${DEBUG_LOG} > ${LOGDIR}/debug.log-${LTIME}.bz2

	# archive everything and prepare to send
	tar czfvp /tmp/${LTIME}.tar.gz ${LOGDIR}

        # send that dump to below
        mutt -s "${DB_NAME} logs for event at ${LTIME}" -a /tmp/${LTIME}.tar.gz ${EMAILS} < /dev/null

	# uncomment the next two lines if you want to remove the logs from the server
	#rm -rf ${LOGDIR}
	#rm -f /tmp/${LTIME}.tar.gz

else

	${LOGR} "${DB_NAME} ox_banners clean"

fi

###################################################################################
##### ox_zones check 
###################################################################################
if [ "`mysql -u${DB_USER} -p${DB_PASS} -Be "select * from ${DB_NAME}.ox_zones where append != '' OR prepend != '';" | wc -l`" -ne "0" ]; then

	# log we are not clean
	${LOGR} "${STRING2}"

	# set time to now
	LTIME=`date +%Y%m%d-%H%M%S`

	# set a new dir to hold all logs
	LOGDIR=${SEC_LOGS}/${LTIME}
	mkdir -p ${LOGDIR}

	# send an alert sms to below
	echo ${STRING} | mail ${SMS_ALERT} # comment full name, i.e. jane doe
	#echo ${STRING} | mail ${SMS_ALERT2} # comment optional user full name

	# perform a dump of the table
        mysqldump -u${DB_USER} -p${DB_PASS} ${DB_NAME} ox_zones | gzip > ${LOGDIR}/${DB_NAME}_ox_zones-${LTIME}.sql.gz
        ${LOGR} "${DB_NAME} OX_ZONES with ${LTIME} dumped to ${LOGDIR}/${DB_NAME}_ox_zones-${LTIME}.sql.gz"
        mysqldump -u${DB_USER} -p${DB_PASS} ${DB_NAME} ox_audit | gzip > ${LOGDIR}/${DB_NAME}_ox_audit-${LTIME}.sql.gz
        ${LOGR} "${DB_NAME} OX_AUDIT with ${LTIME} dumped to ${LOGDIR}/${DB_NAME}_ox_audit-${LTIME}.sql.gz"

	# log who is on the system
	ss -o state established '( dport = :http or sport = :http )' | awk -F" " '{print $4}' | awk -F: '{print $1}' | uniq | grep -v Address > ${LOGDIR}/whoison-${LTIME}.txt

        # files changed in last 24 hours
        find ${OX_PATH}/ -type f -ctime -1 | grep -v cache > ${LOGDIR}/files-${LTIME}.txt

	# dump the admin users for review
	mysql -u${DB_USER} -p${DB_PASS} -Be "use ${DB_NAME}; SELECT u.user_id, u.contact_name, u.email_address, u.username FROM ox_users AS u, ox_account_user_assoc AS aua WHERE u.user_id=aua.user_id AND aua.account_id = (SELECT value FROM ox_application_variable WHERE name='admin_account_id');" > ${LOGDIR}/openx_admin_users-${LTIME}.txt

        # use curl to grab the status page
        curl -o ${LOGDIR}/${SVR}_${ASP}-status-${LTIME} http://${SVR}/${ASP}

        # grab the access and error logs
        for file in `ls -lt ${SVR_LOGS}/ | head -2 | awk '{print$9}'`; do
                tail -100 ${file} >> ${LOGDIR}/access_error_last100-${LTIME}.txt
        done

        # clean up the db
        ${LOGR} "${DB_NAME} OX_ZONES being cleaned up now"
	mysql -u${DB_USER} -p${DB_PASS} -Be "UPDATE ${DB_NAME}.ox_zones SET append = null, prepend = null WHERE append != '' OR prepend != '';"

        # grab the openx debug log too
	bzip2 -9 -c ${DEBUG_LOG} > ${LOGDIR}/debug.log-${LTIME}.bz2

	# archive everything and prepare to send
	tar czfvp /tmp/${LTIME}.tar.gz ${LOGDIR}

        # send that dump to below
        mutt -s "${DB_NAME} logs for event at ${LTIME}" -a /tmp/${LTIME}.tar.gz ${EMAILS} < /dev/null

	# uncomment the next two lines if you want to remove the logs from the server
	#rm -rf ${LOGDIR}
	#rm -f /tmp/${LTIME}.tar.gz

else

	${LOGR} "${DB_NAME} ox_zones clean"

fi

${LOGR} "OPENX: malware check completed"

[/codesyntax]

Installation – openx-check.sh

You basically copy the script somewhere like /usr/local/bin/openx-check.sh

make it executable:

[codesyntax lang=”bash”]

chmod 700 /usr/local/bin/openx-check.sh

[/codesyntax]

Set up a crontab for it (to check every minute below):

[codesyntax lang=”bash”]

*/1 * * * * root su -l -c /usr/local/bin/openx-check.sh > /tmp/openx-check.log 2> /tmp/openx-check.log.err

[/codesyntax]

One more script, this basically cleans the delivery cache…..originally I thought of adding it to the script above but I decided to make it separate in case the script above missed an infection, this one might catch it.

Clean Delivery Cache

[codesyntax lang=”bash”]

#!/bin/bash

# By Ed Wiget
# cleans the openx/var/cache/deliverycache* of infected files

#set -x
LOGR=`which logger`
STRING="OPENX ALERT: cleaning delivery cache"

# Path to openx installation, i.e. /home/www/openx/htdocs
OX_PATH=/home/www/openx/htdocs

${LOGR} "OPENX: starting deliverycache check"
if [ "`grep prepend ${OX_PATH}/var/cache/*.php | grep -v NULL | wc -l`" -gt "0" ]; then
	# we are infected
	${LOGR} "${STRING}"	
	for file in `grep prepend ${OX_PATH}/var/cache/*.php | grep -v NULL | awk -F: '{print$1}'`;
		do rm -f ${file} 
	done

else
	${LOGR} "OPENX: delivery cache is clean"
fi

[/codesyntax]

Installation – clean-cache.sh

You basically copy the script somewhere like /usr/local/bin/clean-cache.sh

make it executable:

[codesyntax lang=”bash”]

chmod 700 /usr/local/bin/clean-cache.sh

[/codesyntax]

Set up a crontab for it (to check every minute below):

[codesyntax lang=”bash”]

*/1 * * * * root su -l -c /usr/local/bin/clean-cache.sh > /tmp/clean-cache.log 2> /tmp/clean-cache.log.err

[/codesyntax]

Finally, once you have an incident, you should change all of the openx users passwords….and of course I have a script for that too.

 Change Passwords

[codesyntax lang=”bash”]

#!/bin/bash

# By Ed Wiget
# this script grabs all the users from the ox_users table and changes their password.

# 20120910 - original script

###################################################################################
##### YOU PROB NEED TO MODIFY THESE BELOW
###################################################################################

# DB Credentials
# database user for openx 
DB_USER='dbuser'
# DB_USERS PASSWORD
DB_PASS='password'
# Database Name
DB_NAME='openx'

# Path for logs collected during password change 
SEC_LOGS=/root/openx_logs

###################################################################################
##### YOU PROB DON'T NEED TO MODIFY THESE BELOW
###################################################################################

# auto generate passwords
function genpass() {
password=''
# Sets the maximum size of the password the script will generate
                                        MAXSIZE=15

# Holds valid password characters. I choose UC/LC-alpha-numeric keys
                                        array1=(
                                        q w e r t y u i o p a s d f g h j k l z x c v b n m
                                        Q W E R T Y U I O P A S D F G H J K L Z X C V B N M
                                        1 2 3 4 5 6 7 8 9 \! \@ \# \$ \% \^ \&
                                        )

# Used in conjunction with modulus to keep random numbers in range of the array size
                                        MODNUM=${#array1[*]}

# Keeps track of the number characters in the password we have generated
                                        pwd_len=0

                                        while [ $pwd_len -lt $MAXSIZE ]
                                        do
                                          index=$(($RANDOM%$MODNUM))
                                          password="${password}${array1[$index]}"
                                          ((pwd_len++))
                                        done
                                        }

# this runs the function above to get the sites password
echo -e "Username\t\tContact\t\tEmail\t\tPassword"
echo ""

for users in `sudo mysql -u${DB_USER} -p${DB_PASS} -Be "use ${DB_NAME}; select user_id from ox_users;" | grep -v user_id`; do

	# generate our new password
	genpass

	# the first password is used for sitedb user
	SITEDB_USER=${password}

	username=`mysql -u${DB_USER} -p${DB_PASS} -Be "use ${DB_NAME}; select username from ox_users where user_id = ${users};" | grep -v username`
	contact_name=`mysql -u${DB_USER} -p${DB_PASS} -Be "use ${DB_NAME}; select contact_name from ox_users where user_id = ${users};" | grep -v contact_name`
	email_address=`mysql -u${DB_USER} -p${DB_PASS} -Be "use ${DB_NAME}; select email_address from ox_users where user_id = ${users};" | grep -v email_address`

	echo -e "${username}\t\t${contact_name}\t\t${email_address}\t\t${SITEDB_USER}"
	echo -e "${username}\t\t${contact_name}\t\t${email_address}\t\t${SITEDB_USER}" >> ${SEC_LOGS}/openx-new-passwords.txt

	mysql -u${DB_USER} -p${DB_PASS} -Be "use ${DB_NAME}; update ox_users set \`password\` = md5('${SITEDB_USER}') where user_id = ${users};"

done

[/codesyntax]

 

Installation – change-openx-passwords.sh

You basically copy the script somewhere like /usr/local/bin/change-openx-passwords.sh

make it executable:

[codesyntax lang=”bash”]

chmod 700 /usr/local/bin/change-openx-passwords.sh

[/codesyntax]

The above code is basically run whenever you want to change all the users passwords for openx. It will dump the username, contact name, email address, and new password to a file defined in the SEC_LOG in the script.

One other recommendation I can make….get a free account at cloudflare or incapsula or similar which has the potential to stop unknown vulnerabilities from being exploited in openx.

Leave a Reply