Mar 062013
 

This was tested on centos 6.3.  It is running at approx 900 – 3,000+ log events per second from approx 30 hosts.

Current load is about 900 messages per second:  load average: 1.57, 1.35, 1.29 with 8GB memory.

With the above in mind, there was approx 165GB of log data after running for 4 days.

graylog.org web site

elasticsearch web site

mongo db

passenger phusion web site

logstash web site

I wrote a script in order to install a graylog2 central log server.  Its a one shot run and be done kinda thing…..

CHANGES20130309 – see notes in script

#!/bin/bash
 
# By Ed Wiget for TD
 
# 20130226 - original script
# 20130305 - version bump graylog2-server 0.10.0 and graylog2-web-interface 0.10.2 to version 0.11.0
# 20130800 - updated to add build environment required for rvm (missing reported by unknown user via blog comment)
#          - optimized order of installation
#          - tested in clean install of virtualbox
 
#####################################################################
##  REQUIREMENTS
#####################################################################
 
# enable next line for debugging
# set -x
 
# this is just a short name of the script used in log names, etc
SCRIPTIDENTIFIER=install_graylog2
 
# bc required for floating point disk space check
BCR=`which bc > /dev/null`
if [ $? -ne "0" ]; then
        echo "bc does not exist.  Please install it"
        exit
fi
#####################################################################
#####################################################################
##  PROCESS - every script needs a detailed step-by-step process
#####################################################################
#####################################################################
 
#####################################################################
## systems should never have two of the same scripts running at
## same time
#####################################################################
# create the pidfile location
PIDFILE=/var/run/${SCRIPTIDENTIFIER}.pid
 
# check if pidfile exists and that process also exists and if so, exits
if [ -e ${PIDFILE} ]; then
    PID=`cat ${PIDFILE}`
if kill -0 ${PID} > /dev/null 2>&1; then
    echo "Already running"
    exit 1
else
# if the process doesn't exist but pidfile does, remove it -- process crashed?
    rm ${PIDFILE}
fi
fi
# create the pidfile if it doesn't exist
    echo $$ > ${PIDFILE}
## end pidfile check - cleanup at end
 
#####################################################################
############# USER  CONFIG VARIABLES BELOW LOGGING START ############
#####################################################################
 
#####################################################################
## set basedir and ctime because both required by logging
## set the current time and is used for consistency
#####################################################################
CTIME=`date +%Y%m%d-%H%M%S`
BASEDIR=/root
LOGDIR=/root/installs
DIAGS=/root/diags
BUILDIR=/home/build
mkdir -p ${LOGDIR} ${DIAGS} ${BUILDIR}
FREESPACE=2
SHORTNAME=`hostname | awk -F. '{print$1}'`
 
## where to send build log to
EMAILS=me@my_username
 
#####################################################################
## make sure we are running as root user
#####################################################################
if [ $(whoami) != "root" ]; then
        echo "You need to run this script as root."
        exit 1
fi
#####################################################################
## Check Disk Space
#####################################################################
OUTPUT=`df -h ${BASEDIR} |  grep -vE '^Filesystem|tmpfs|cdrom|rootfs|cachedir|/dev/shm|udev|rc-svcdir|/boot' | awk '{print $1 " "  " " $4 " "  " " $6 }'`
USEP=$(echo ${OUTPUT} | awk '{ print $2}' | cut -d'G' -f1 )
PARTITION=$(echo ${OUTPUT} | awk '{ print $1 }' )
 
if [ "`echo \"${FREESPACE} > ${USEP}\" | bc`" = "1" ]; then 
  echo -e "\nThere is not enough free disk space available on\n\n\"${PARTITION} (${USEP} GB)\" \n\non $(hostname) as on $(date)\n\nto perform this migration.  You need to free up\n\nsome disk space on ${BASEDIR} to
continue\n"
  break
else
  echo -e "\nThere is \"${USEP} GB\" free space, enough to continue\n\nall log files will be stored in ${LOGDIR}"
  sleep 5
  echo ""
fi
#####################################################################
## Set up logging the correct way
#####################################################################
BUILD_LOG=${LOGDIR}/`hostname | awk -F. '{print$1}'`-${CTIME}.${SCRIPTIDENTIFIER}.log
BUILD_PIPE=${BUILD_LOG}.pipe
if [ ! -e ${BUILD_PIPE} ]; then
    mkfifo ${BUILD_PIPE}
fi
if [ -e ${BUILD_LOG} ]; then
    rm ${BUILD_LOG}
fi
exec 3>&1 4>&2
tee ${BUILD_LOG} < ${BUILD_PIPE} >&3 &
TPID=$!
exec > ${BUILD_PIPE} 2>&1
trap "rm -f ${BUILD_PIPE} ; rm -f ${PIDFILE}" EXIT
#####################################################################
########################### ACCOUNTABILITY ##########################
#####################################################################
echo -e "\nPlease enter your name:\n"
read ADMIN_NAME
echo -e "\n\n\n\n"
echo "COMPANY NAME"
echo "${SCRIPTIDENTIFIER} Performed by: ${ADMIN_NAME}"
echo "On: ${CTIME}"
echo "Server: `hostname`"
echo "Logfile: ${BUILD_LOG}"
echo -e "\n\n\n\n"
#####################################################################
## START CODE
#####################################################################
 
# get rid of timedhosts.txt so yum will chose new fastest mirror
find /var/ -type f -name timedhosts.txt -exec rm -f "{}" \;
 
# repos we need
MONGODB_REPO="/etc/yum.repos.d/10gen-mongodb.repo"
CENTOS_REPO="/etc/yum.repos.d/CentOS-Base.repo"
EPEL_REPO="/etc/yum.repos.d/epel.repo"
PASSENGER_REPO="/etc/yum.repos.d/passenger.repo"
# where we are installing all software
APP_ROOT="/export"
# graylog2 server version to install
GLSV=0.11.0
# graylog2 web interface to install
GLWV=0.11.0
# elasticsearch version to install
ELSV=0.20.4
 
# set up our installation directory
if [ ! -d ${APP_ROOT} ]; then
  mkdir -p ${APP_ROOT}
fi
 
# set up our log directory
if [ ! -d ${APP_ROOT}/logs ]; then
  mkdir -p ${APP_ROOT}/logs
fi
 
if [ -f ${MONGODB_REPO} ]; then
  echo "${MONGODB_REPO} found"
else
cat << 'EOF' > ${MONGODB_REPO}
[10gen]
name=10gen Repository
baseurl=http://downloads-distro.mongodb.org/repo/redhat/os/$basearch
gpgcheck=0
enabled=1
EOF
fi
 
if [ -f ${CENTOS_REPO} ]; then
    echo "${CENTOS_REPO} found"
else
    echo "Creating ${CENTOS_REPO}"
cat << 'EOF' > ${CENTOS_REPO}
# CentOS-Base.repo for RHEL6
[base]
name=CentOS-6 - Base
mirrorlist=http://mirrorlist.centos.org/?release=6&arch=$basearch&repo=os
#baseurl=http://mirror.centos.org/centos/6/os/$basearch/
gpgcheck=1
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-6
 
#released updates
[update]
name=CentOS-6 - Updates
mirrorlist=http://mirrorlist.centos.org/?release=6&arch=$basearch&repo=updates
#baseurl=http://mirror.centos.org/centos/6/updates/$basearch/
gpgcheck=1
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-6
EOF
fi
 
if [ -f ${EPEL_REPO} ]; then
    echo "${EPEL_REPO} found"
else
    echo "Creating ${EPEL_REPO}"
cat << 'EOF' > ${EPEL_REPO}
[epel]
name=Extra Packages for Enterprise Linux 6 - $basearch
#baseurl=http://download.fedoraproject.org/pub/epel/6/$basearch
mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-6&arch=$basearch
failovermethod=priority
enabled=1
gpgcheck=0
EOF
fi
 
if [ -f ${PASSENGER_REPO} ]; then
    echo "${PASSENGER_REPO} found"
else
    echo "Creating ${PASSENGER_REPO}"
cat << 'EOF' > ${PASSENGER_REPO}
### Name: Phusion Passenger RPM Repository for Red Hat Enterprise 6
### URL: http://passenger.stealthymonkeys.com/
[passenger]
name = Red Hat Enterprise $releasever - Phusion Passenger
baseurl = http://passenger.stealthymonkeys.com/rhel/$releasever/$basearch
mirrorlist = http://passenger.stealthymonkeys.com/rhel/mirrors
#mirrorlist = file:///etc/yum.repos.d/mirrors-passenger
enabled = 1
gpgcheck = 0
EOF
fi
 
# update for new repos and make sure system is up to date
yum -y update
 
# we need a build environment for rvm
yum install -y gcc-c++ patch readline readline-devel zlib zlib-devel libyaml-devel libffi-devel openssl-devel openssl make bzip2 autoconf automake libtool bison iconv-devel libyaml libyaml-devel
 
# disable selinux config (requires reboot for permanent disable)
# I am too lazy to fix selinux for all the ports required for this setup
# selinux causes issues with mongo, passenger, http, elasticsearch...pretty much everything in this software stack
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/sysconfig/selinux
# set temp to permissive
setenforce 0
 
# we set up rvm
curl -L https://get.rvm.io | bash -s stable --ruby
source /usr/local/rvm/scripts/rvm
RVER=`ruby -v | awk -F" " '{print$2}'`
if [ "${RVER}" == "2.0.0p0" ]; then
  echo "ruby successfully set up"
  echo "making it the default version"
  rvm --default use 2.0.0
  # add root user to rvm group
  usermod -a -G rvm root
else
  echo "ruby not set up correctly"
  exit
fi
gem install bundle
gem update
 
# we have to make sure mongo is not installed from default packages....
if [ "`rpm -qa | grep mongodb | wc -l`" = "0" ]; then
  echo "no older version of mongo installed"
else 
  echo -e "we have found an old version of mongo installed.  ...removing"
  echo -e "\nif you are using this version of mongo, press ctl+c to exit"
  echo -e "\nyou will then need to comment out the parts of mongo in this script related to installation"
  echo -e "\nI am not sure this software stack will work with an older version of mongo"
  sleep 60
  yum -y remove mongodb mongodb-server libmongodb
fi
 
echo "Installing MongoDB..."
yum install -y mongo-10gen mongo-10gen-server
 
# i had troubles with the above failing....it appears to be a server overloaded issue, yum would time out before install, so now we have this check
if [ "`rpm -qa | egrep \"mongo-10gen|mongo-10gen-server\" | wc -l`" = "2" ]; then
  echo "mongodb rpm installation was successful"
else
  echo "mongodb rpm installation failed....bailing"
  exit
fi
 
# make sure services are stopped before we relocate
service mongod stop 
/etc/init.d/mongod stop
 
echo "moving mongo db to ${APP_ROOT}/mongo/"
rsync -av /var/lib/mongo/ ${APP_ROOT}/mongo/
 
echo "removing original /var/lib/mongo"
rm -rf /var/lib/mongo
 
echo "creating symlink to ${APP_ROOT}/mongo"
ln -s ${APP_ROOT}/mongo /var/lib/mongo
 
# moving /var/log/mongo to /export/logs/mongo
echo "moving mongo logs to ${APP_ROOT}/logs/mongo/"
rsync -av /var/log/mongo/ ${APP_ROOT}/logs/mongo/
 
echo "removing original /var/log/mongo"
rm -rf /var/log/mongo
 
echo "creating symlink to ${APP_ROOT}/logs/mongo"
ln -s ${APP_ROOT}/logs/mongo /var/log/mongo
 
echo "Installing init script of MongoDB..."
chkconfig --level 2345 mongod on
 
echo "Starting MongoDB..."
service mongod start
 
echo "Installing ElasticSearch..."
cd ${APP_ROOT}
# all the documentation I have seen for graylog2 installs says to use elasticsearch 0.20.4 
# however, I have an installation working just fine with 0.20.5 so if you have issues, suspect the version number
curl -L http://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-${ELSV}.tar.gz | tar -xz
ln -s elasticsearch-${ELSV}/ elasticsearch
 
echo "Installing ElasticSearch service wrapper..."
cd elasticsearch/bin
curl -L http://github.com/elasticsearch/elasticsearch-servicewrapper/tarball/master | tar -xz
mv elasticsearch-elasticsearch-servicewrapper-*/service .
rm -rf elasticsearch-elasticsearch-servicewrapper-*
 
cd ${APP_ROOT}
useradd elasticsearch -d ${APP_ROOT}/elasticsearch
chown -R elasticsearch:elasticsearch elasticsearch*
 
sed -i "s/# cluster.name: elasticsearch/cluster.name: graylog2/g" ${APP_ROOT}/elasticsearch/config/elasticsearch.yml
 
sed -i "s/set.default.ES_HOME=<Path to ElasticSearch Home>/set.default.ES_HOME=${APP_ROOT}/elasticsearch/g" ${APP_ROOT}/elasticsearch/bin/service/elasticsearch.conf
 
sed -i -e "s/#RUN_AS_USER=/RUN_AS_USER=elasticsearch/g" ${APP_ROOT}/elasticsearch/bin/service/elasticsearch
 
echo "Installing init script of ElasticSearch..."
${APP_ROOT}/elasticsearch/bin/service/elasticsearch install
 
echo "Starting ElasticSearch..."
service elasticsearch start
 
# testing 
echo "data from elasticsearch should return here...."
sleep 5
curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'
 
chkconfig --add elasticsearch
chkconfig elasticsearch on
 
echo "Installing graylog2-server..."
cd ${APP_ROOT}
curl -L http://download.graylog2.org/graylog2-server/graylog2-server-${GLSV}.tar.gz | tar -xz
ln -s graylog2-server-${GLSV} graylog2-server
 
cd graylog2-server
mv graylog2.conf.example graylog2.conf
mv elasticsearch.yml.example graylog2-elasticsearch.yml
cd /etc
ln -s /${APP_ROOT}/graylog2-server/graylog2.conf graylog2.conf
ln -s /${APP_ROOT}/graylog2-server/graylog2-elasticsearch.yml graylog2-elasticsearch.yml
 
sed -i 's/mongodb_useauth = true/mongodb_useauth = false/g' /etc/graylog2.conf
# the value below is the total number of messages to keep in each index * number of indexes for total messages
sed -i 's/elasticsearch_max_docs_per_index = 20000000/elasticsearch_max_docs_per_index = 10000000/g' /etc/graylog2.conf
# the value below is the number of indexes to keep.  If you get to this number, the oldest index will be deleted
sed -i 's/elasticsearch_max_number_of_indices = 20/elasticsearch_max_number_of_indices = 10/g' /etc/graylog2.conf
# total number of shards to keep, a good rule of thumb, 1 shard per elasticsearch node
sed -i 's/elasticsearch_shards = 4/elasticsearch_shards = 1/g' /etc/graylog2.conf
# the number of minutes of "recent" messages kept on the web page when you log in - for high volume log servers, 1 minute
sed -i 's/recent_index_ttl_minutes = 60/recent_index_ttl_minutes = 1/g' /etc/graylog2.conf
 
echo "Installing init script of graylog2-server..."
cat << 'EOF' > /etc/init.d/graylog2-server
#! /bin/sh
### BEGIN INIT INFO
# Provides:          gaylog2-server
# Required-Start:    $all
# Required-Stop:     $all
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Graylog2-server init script
# Description: Graylog2-server init script
### END INIT INFO
 
## Source function library.
. /etc/init.d/functions
 
NAME="graylog2-server"
GRAYLOG2_HOME=
GRAYLOG2_JAR=$GRAYLOG2_HOME/graylog2-server.jar
GRAYLOG2_CONF="/etc/graylog2.conf"
GRAYLOG2_OUT="/var/log/graylog2.log"
PID_DIR="/var/run/graylog2"
PID_FILE="${PID_DIR}/${NAME}.pid"
LOCK_FILE="/var/lock/subsys/${NAME}"
 
JAVA="/usr/bin/java"
JAVA_OPTS="-server -Xms512m -Xmx512m"
RUN_AS_USER=root
 
if [ ! -d "$PID_DIR" ]; then
  mkdir "$PID_DIR"
  chown $RUN_AS_USER "$PID_DIR"
fi
 
if [ ! -r "$GRAYLOG2_JAR" ]; then
   echo "Cannot find $GRAYLOG2_JAR"
   echo "${GRAYLOG2_JAR} is absent or does not have read permission"
   exit 1
fi
 
touch "$GRAYLOG2_OUT"
chown $RUN_AS_USER "$GRAYLOG2_OUT"
 
start() {
  echo "Starting $NAME: "
  COMMAND="$JAVA $JAVA_OPTS -jar $GRAYLOG2_JAR -f $GRAYLOG2_CONF -p $PID_FILE >> \"$GRAYLOG2_OUT\" 2>&1 &"
  # for debugging enable next line
  # COMMAND="$JAVA $JAVA_OPTS -jar $GRAYLOG2_JAR --debug -f $GRAYLOG2_CONF -p $PID_FILE >> \"$GRAYLOG2_OUT\" 2>&1 &"
  # for statistics enable the next line
  # COMMAND="$JAVA $JAVA_OPTS -jar $GRAYLOG2_JAR --statistics -f $GRAYLOG2_CONF -p $PID_FILE >> \"$GRAYLOG2_OUT\" 2>&1 &"
  daemon --check=${NAME} --user=${RUN_AS_USER} --pidfile=${PID_FILE} ${COMMAND}
  RETVAL=$?
  [ $RETVAL -eq 0 ] && touch $LOCK_FILE
  return $RETVAL
}
 
stop() {
  echo "Stoping $NAME: "
  killproc -p ${PID_FILE} -d 5 ${NAME}
  RETVAL=$?
  [ $RETVAL -eq 0 ] && rm -f $LOCK_FILE
  return $RETVAL  
}
 
restart() {
  stop
  sleep 1
  start
}
 
dump() {
  echo "Dumping $NAME: "
  PID=`cat $PID_FILE`
  kill -3 $PID
  RETVAL=$?
  [ $RETVAL -eq 0 ] && success $"Dumped $NAME." || failure $"Failed to dump $NAME."
  return $RETVAL
}
 
rh_status() {
  status -p ${PID_FILE} ${NAME}
}
 
rh_status_q() {
  rh_status >/dev/null 2>&1
}
 
case "$1" in
  start)
    rh_status_q && exit 0
    start
    ;;
  stop)
    rh_status_q || exit 0
    stop
    ;;
  restart)
    restart
    ;;
  status)
    rh_status
    ;;
  dump)
    rh_status_q || exit 0
    dump
    ;; 
  condrestart|try-restart)
    rh_status_q || exit 0
    restart
    ;;
  *)
    echo "Usage: $0 {start|stop|status|restart|condrestart|try-restart|dump}"
    exit 1
esac
 
exit $?
EOF
 
sed -i "s/GRAYLOG2_HOME=/GRAYLOG2_HOME=\\${APP_ROOT}\/graylog2-server/g" /etc/init.d/graylog2-server
chmod u+x /etc/init.d/graylog2-server
chkconfig graylog2-server on
 
echo "Starting graylog2-server..."
service graylog2-server start
 
echo "Installing graylog2-web-interface..."
#yum install -y ruby ruby-devel ruby-rdoc rubygems 
 
yum install -y httpd mod_passenger
 
cd ${APP_ROOT}
curl -L http://download.graylog2.org/graylog2-web-interface/graylog2-web-interface-${GLWV}.tar.gz | tar -xz
ln -s graylog2-web-interface-${GLWV} graylog2-web-interface
 
cd graylog2-web-interface
 
# fix the Gemfile - has wrong versions of json bison bison_ext
cat << 'EOF' > ${APP_ROOT}/graylog2-web-interface/Gemfile
source :rubygems
 
gem 'rack', '~> 1.4.1'
gem 'rake', '~> 0.9.2'
gem 'rails', '~> 3.2.12'
gem 'json', '~> 1.7.7'
gem 'chronic', '~> 0.6.7'
gem 'pony', '~> 1.1'  # unusual version number
gem 'graylog2-declarative_authorization', '~> 0.5.2', :require => 'declarative_authorization'
gem "mongo", "~> 1.8.2"
gem 'mongoid', '2.4.5'
gem "tire", "~> 0.5.1"
gem 'bson', "~> 1.8.2"
gem 'bson_ext', "~> 1.8.2", :platforms => :ruby
gem 'home_run', '~> 1.0.2', :platforms => :ruby
gem 'SystemTimer', '~> 1.2.3', :require => 'system_timer', :platforms => :ruby_18
gem 'rails_autolink', '~> 1.0.4'
gem 'kaminari', '~> 0.12.4'
gem 'jquery-rails', '~> 2.1'
gem 'therubyracer', '~> 0.10.2'
gem 'net-ldap', '~> 0.3.1'
 
group :development, :test do
  # might be useful to generate fake data in development
  gem 'machinist_mongo', '~> 1.2.0', :require => 'machinist/mongoid'
  gem 'faker', '~> 0.9.5'
end
 
group :development do
  # gem 'ruby-prof', '~> 0.10.5'  # works nice with NewRelic RPM Developer Mode
  gem 'passenger', '~> 3.0.17'
end
 
group :test do
  gem 'ci_reporter', '~> 1.6.4'
  gem 'shoulda', '~> 2.11.3'
  gem 'shoulda-activemodel', '0.0.2', :require => 'shoulda/active_model'  # fixed version - too hacky
  gem 'mocha', '~> 0.9.12'
  gem 'database_cleaner', '~> 0.6.0'
  gem 'timecop', '~> 0.3.5'
end
 
# Needed for the new asset pipeline
group :assets do
  gem 'sass-rails',   "~> 3.2.5"
  gem 'coffee-rails', "~> 3.2.2"
  gem 'uglifier',     ">= 1.0.3"
end
EOF
 
# mongo gem required for http user
bundle update mongo json bson_ext
bundle install --without=development
 
# set up a graylog user
useradd graylog -d ${APP_ROOT}/graylog2-web-interface -G rvm
chown -R graylog2:graylog2 ${APP_ROOT}/graylog2-web-interface
usermod -g rvm root
source /etc/profile.d/rvm.sh
 
# set up the first graylog2 user
cd ${APP_ROOT}/graylog2-web-interface
#su graylog
 
# requried for next step
yum -y install lynx
 
echo "Open another shell on this server and browse to http://IPADDRESS:3000 and setup your first user and login."
sleep 20
# Note there is bug with the current mongo driver that gives a notice about bson_ext not being loaded.  It will be fixed in a future version.
RAILS_ENV=production script/rails server &
 
echo "after password is created, kill the server in the shell"
sleep 20
 
# install passenger
gem install passenger file-tail
 
# a fix to the bug in passenger <=3.0.19
cat << 'EOF' > /usr/local/rvm/gems/ruby-2.0.0-p0/gems/passenger-3.0.19/build/gempackagetask.rb
#!/usr/bin/env ruby
# Passenger note: this file is copied from Rake 0.8.1. The task names
# have been changed.
 
# Define a package task library to aid in the definition of GEM
# packages.
 
require 'rubygems'
require 'rake'
require 'build/packagetask'
require 'rubygems/user_interaction'
 
if /^2\./ =~ RUBY_VERSION
  require 'rubygems/package'
else
  require 'rubygems/builder'
end
 
module Rake
 
  # Create a package based upon a Gem spec.  Gem packages, as well as
  # zip files and tar/gzipped packages can be produced by this task.
  #
  # In addition to the Rake targets generated by PackageTask, a
  # GemPackageTask will also generate the following tasks:
  #
  # [<b>"<em>package_dir</em>/<em>name</em>-<em>version</em>.gem"</b>]
  #   Create a Ruby GEM package with the given name and version.
  #
  # Example using a Ruby GEM spec:
  #
  #   require 'rubygems'
  #
  #   spec = Gem::Specification.new do |s|
  #     s.platform = Gem::Platform::RUBY
  #     s.summary = "Ruby based make-like utility."
  #     s.name = 'rake'
  #     s.version = PKG_VERSION
  #     s.requirements << 'none'
  #     s.require_path = 'lib'
  #     s.autorequire = 'rake'
  #     s.files = PKG_FILES
  #     s.description = <<EOF
  #   Rake is a Make-like program implemented in Ruby. Tasks
  #   and dependencies are specified in standard Ruby syntax.
  #   EOF
  #   end
  #
  #   Rake::GemPackageTask.new(spec) do |pkg|
  #     pkg.need_zip = true
  #     pkg.need_tar = true
  #   end
  #
  class GemPackageTask < PackageTask
    # Ruby GEM spec containing the metadata for this package.  The
    # name, version and package_files are automatically determined
    # from the GEM spec and don't need to be explicitly provided.
    attr_accessor :gem_spec
 
    # Create a GEM Package task library.  Automatically define the gem
    # if a block is given.  If no block is supplied, then +define+
    # needs to be called to define the task.
    def initialize(gem_spec)
      init(gem_spec)
      yield self if block_given?
      define if block_given?
    end
 
    # Initialization tasks without the "yield self" or define
    # operations.
    def init(gem)
      super(gem.name, gem.version)
      @gem_spec = gem
      @package_files += gem_spec.files if gem_spec.files
    end
 
    # Create the Rake tasks and actions specified by this
    # GemPackageTask.  (+define+ is automatically called if a block is
    # given to +new+).
    def define
      super
      task :package => ['package:gem']
      desc "Build the gem file #{gem_file}"
      task 'package:gem' => ["#{package_dir}/#{gem_file}"]
      file "#{package_dir}/#{gem_file}" => [package_dir] + @gem_spec.files do
        when_writing("Creating GEM") {
          if /^2\./ =~ RUBY_VERSION
            Gem::Package.build(gem_spec)
          else
            Gem::Builder.new(gem_spec).build
          end
          verbose(true) {
            mv gem_file, "#{package_dir}/#{gem_file}"
          }
        }
      end
    end
 
    def gem_file
      if @gem_spec.platform == Gem::Platform::RUBY
        "#{package_name}.gem"
      else
        "#{package_name}-#{@gem_spec.platform}.gem"
      end
    end
 
  end
end
EOF
 
# install the apache module
passenger-install-apache2-module
 
# and the passenger.conf file for apache
cat << EOF >> /etc/httpd/conf.d/passenger.conf
   LoadModule passenger_module /usr/local/rvm/gems/ruby-2.0.0-p0/gems/passenger-3.0.19/ext/apache2/mod_passenger.so
   PassengerRoot /usr/local/rvm/gems/ruby-2.0.0-p0/gems/passenger-3.0.19
   PassengerRuby /usr/local/rvm/wrappers/ruby-2.0.0-p0/ruby 
   PassengerPoolIdleTime 0
   RailsFrameworkSpawnerIdleTime 0
   RailsAppSpawnerIdleTime 0
   PassengerMaxPoolSize 7
<VirtualHost *:80>
    ServerName logs.tritondigital.net
    ServerAlias logs.tritondigital.net
    ServerAlias aws-east1d-pro-flex-log-01.tritondigital.net
    DocumentRoot ${APP_ROOT}/graylog2-web-interface/public
    RailsEnv 'production'
    SetEnv MONGOID_HOST 'localhost'
    SetEnv MONGOID_PORT '27017'
    SetEnv MONGOID_DATABASE 'graylog2'
    <Directory ${APP_ROOT}/graylog2-web-interface/public>
        Allow from all
        Options -MultiViews
    </Directory>
</VirtualHost>
EOF
 
chown -R apache:apache ${APP_ROOT}/graylog2-web-interface
chkconfig --level 2345 httpd on
service httpd restart
#####################################################################
## END CODE
#####################################################################
 
###########################################################################
## LOGGING CLEANUP
###########################################################################
 
# 1>&3 and 2>&4 restore the original file descriptors for stdout and
# stderr from file descriptors 3 and 4.
 
# 3>&~ 4>&~: now that they are not needed, close file descriptors 3 and 4
 
exec 1>&3 3>&- 2>&4 4>&-
 
# waits on the tee pid to die before continuing
 
wait ${TPID}
 
# remove the named pipe which is no longer needed
sleep 5
rm ${BUILD_PIPE}
 
##############################################################################
## MAIL REPORT
##############################################################################
echo "using mutt to send ${BUILD_LOG} to ${EMAILS}"
 
bzip2 -9 ${BUILD_LOG}
 
# ubuntu and arch require -- before #{EMAILS}, redhat does not
if [ -f /etc/redhat-release ]; then
    mutt -s "${SCRIPTIDENTIFIER} log for `hostname | awk -F. '{print$1}'`" -a ${BUILD_LOG}.bz2  ${EMAILS} < /dev/null
else
    mutt -s "${SCRIPTIDENTIFIER} log for `hostname | awk -F. '{print$1}'`" -a ${BUILD_LOG}.bz2 -- ${EMAILS} < /dev/null
fi
#####################################################################
## remove pidfile create at start 
rm ${PIDFILE}
#####################################################################

 

For sending the log events to the log server, you can use this logstash script I also made:

input {
  file {
    type => "syslog"
    path => [ "/var/log/messages", "/var/log/dmesg", "/var/log/cron", "/var/log/secure", "/var/log/syslog" ]
  }
  file {
    type => "nginx-access"
    path => ["/var/log/nginx/access.log", "/var/log/nginx/error.log"]
  }
}
output {
  gelf {
    chunksize => "1420"
    facility => "logmessages-gelf"
    host => "log_server_ip_or_FQDN"
    port => "12201"
  }
}

I also have rpms build for each of the logstash dependancies for centos 6.3 but unfortunately they contain proprietary config files.  I will try to get a set rebuild without the proprietary configs.

Here is also a script that will remove the entire installation, in the event you mess something up.  It removes the major components only:

#!/bin/bash
 
# By Ed Wiget
# Easily remove all changes to the system for install-graylog2.sh
APP_ROOT="/export"
 
# stop services
#service graylog2-web-interface stop # no longer needed switched from ruby nginx to apache
service graylog2-server stop
service mongod stop
service elasticsearch stop
service httpd stop
#service rabbitmq-server stop
#service logstash stop
 
# remove mongo rpms
yum remove -y mongo-10gen mongo-10gen-server
 
# delete directories and software packages
rm -rf ${APP_ROOT}/elasticsearch*
rm -rf ${APP_ROOT}/graylog2-server*
rm -rf ${APP_ROOT}/graylog2-web-interface*
rm -rf ${APP_ROOT}/mongo
 
# remove logs
rm -rf /var/log/mongo
 
# remove symlink
rm -rf /var/lib/mongo
 
# remove /etc config files
rm -f /etc/graylog*
 
# remove /etc/init.d/files
rm -f /etc/init.d/graylog*
rm -f /etc/init.d/elasticsearch
 
# remove passenger.conf from /etc/httpd/conf.d/passenger.conf
rm -f /etc/httpd/conf.d/passenger.conf
 
echo "done"

A script to start all the services and a script to stop all the services:

#!/bin/bash
 
service elasticsearch start
service mongod start
service graylog2-server start
service httpd start
#service rabbitmq-server start
 
echo "checking running services......"
echo -e "\n\n"
ps aux | egrep "[e]lasticsearch|[m]ongo|[g]raylog2-server|[h]ttp"
#!/bin/bash
 
# By Ed Wiget
# stops graylog 
 
service graylog2-server stop
service mongod stop
service elasticsearch stop
service httpd stop
#service rabbitmq-server stop
 
echo "checking running services......"
echo -e "\n\n"
ps aux | egrep "[e]lasticsearch|[m]ongo|[g]raylog2-server|[h]ttp"

  6 Responses to “Auto script: graylog2 + elasticsearch + mongo + passenger phusion for central logging”

  1. Hi,
    I came across your site from researching graylog2 and elasticsearch. This setup looks awesome. Just one question, why nginx over apache2? Also, how hard would it be to switch nginx to apache2 inthe script? Just wondering. Thanks again!

    • Usuaully I always use nginx in order to keep the software stack lightweight but everything in this script is apache. The reason I used apache here is because I did a phusion install the first time using nginx instead of apache and had stability problems.

  2. Hey so i’m running your script but encoutering errors:

    Upgrade Notes:

    * No new notes to display.

    # RVM: Shell scripts enabling management of multiple ruby environments.
    # RTFM: https://rvm.io/
    # HELP: http://webchat.freenode.net/?channels=rvm (#rvm on irc.freenode.net)
    # Cheatsheet: http://cheat.errtheblog.com/s/rvm
    # Screencast: http://screencasts.org/episodes/how-to-use-rvm

    # In case of any issues run ‘rvm requirements’ or read ‘rvm notes’

    Upgrade of RVM in /usr/local/rvm/ is complete.

    # root,
    #
    # Thank you for using RVM!
    # I sincerely hope that RVM helps to make your life easier and
    # more enjoyable!!!
    #
    # ~Wayne

    rvm 1.19.3 (stable) by Wayne E. Seguin , Michal Papis [https://rvm.io/]

    Searching for binary rubies, this might take some time.
    No binary rubies available for: centos/6.4/i386/ruby-2.0.0-p0.
    Continuing with compilation. Please read ‘rvm mount’ to get more information on binary rubies.
    Installing requirements for centos, might require sudo password.
    Missing required packages: git, patch.
    RVM autolibs is now configured with mode ‘2’ => ‘check and stop if missing’,
    please run `rvm autolibs enable` to let RVM do its job or run and read `rvm autolibs [help]`
    or visit https://rvm.io/rvm/autolibs for more information.
    ./install_graylog2: line 240: ruby: command not found
    ruby not set up correctly

    Let me know what you think.

    • Looks like to me you don’t have all the requirements for rvm…..from the error message you posted:

      Missing required packages: git, patch.
      RVM autolibs is now configured with mode ’2′ => ‘check and stop if missing’,
      please run `rvm autolibs enable` to let RVM do its job or run and read `rvm autolibs [help]`
      or visit https://rvm.io/rvm/autolibs for more information.
      ./install_graylog2: line 240: ruby: command not found
      ruby not set up correctly

  3. If I run the following command from the script I get an error at the following point stating it does not appear to be a gzip file. Any ideas?

    # all the documentation I have seen for graylog2 installs says to use elasticsearch 0.20.4
    # however, I have an installation working just fine with 0.20.5 so if you have issues, suspect the version #number
    curl -L http://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-${ELSV}.tar.gz | tar -xz

    • It could be that the version number for elasticsearch has changed……I would visit their website and see and adjust the script as necessary.