This was tested on centos 6.3. It is running at approx 900 – 3,000+ log events per second from approx 30 hosts.
Current load is about 900 messages per second: load average: 1.57, 1.35, 1.29 with 8GB memory.
With the above in mind, there was approx 165GB of log data after running for 4 days.
I wrote a script in order to install a graylog2 central log server. Its a one shot run and be done kinda thing…..
CHANGES: 20130309 – see notes in script
[codesyntax lang=”bash”]
#!/bin/bash
# By Ed Wiget for TD
# 20130226 - original script
# 20130305 - version bump graylog2-server 0.10.0 and graylog2-web-interface 0.10.2 to version 0.11.0
# 20130800 - updated to add build environment required for rvm (missing reported by unknown user via blog comment)
# - optimized order of installation
# - tested in clean install of virtualbox
#####################################################################
## REQUIREMENTS
#####################################################################
# enable next line for debugging
# set -x
# this is just a short name of the script used in log names, etc
SCRIPTIDENTIFIER=install_graylog2
# bc required for floating point disk space check
BCR=`which bc > /dev/null`
if [ $? -ne "0" ]; then
echo "bc does not exist. Please install it"
exit
fi
#####################################################################
#####################################################################
## PROCESS - every script needs a detailed step-by-step process
#####################################################################
#####################################################################
#####################################################################
## systems should never have two of the same scripts running at
## same time
#####################################################################
# create the pidfile location
PIDFILE=/var/run/${SCRIPTIDENTIFIER}.pid
# check if pidfile exists and that process also exists and if so, exits
if [ -e ${PIDFILE} ]; then
PID=`cat ${PIDFILE}`
if kill -0 ${PID} > /dev/null 2>&1; then
echo "Already running"
exit 1
else
# if the process doesn't exist but pidfile does, remove it -- process crashed?
rm ${PIDFILE}
fi
fi
# create the pidfile if it doesn't exist
echo $$ > ${PIDFILE}
## end pidfile check - cleanup at end
#####################################################################
############# USER CONFIG VARIABLES BELOW LOGGING START ############
#####################################################################
#####################################################################
## set basedir and ctime because both required by logging
## set the current time and is used for consistency
#####################################################################
CTIME=`date +%Y%m%d-%H%M%S`
BASEDIR=/root
LOGDIR=/root/installs
DIAGS=/root/diags
BUILDIR=/home/build
mkdir -p ${LOGDIR} ${DIAGS} ${BUILDIR}
FREESPACE=2
SHORTNAME=`hostname | awk -F. '{print$1}'`
## where to send build log to
EMAILS=me@my_username
#####################################################################
## make sure we are running as root user
#####################################################################
if [ $(whoami) != "root" ]; then
echo "You need to run this script as root."
exit 1
fi
#####################################################################
## Check Disk Space
#####################################################################
OUTPUT=`df -h ${BASEDIR} | grep -vE '^Filesystem|tmpfs|cdrom|rootfs|cachedir|/dev/shm|udev|rc-svcdir|/boot' | awk '{print $1 " " " " $4 " " " " $6 }'`
USEP=$(echo ${OUTPUT} | awk '{ print $2}' | cut -d'G' -f1 )
PARTITION=$(echo ${OUTPUT} | awk '{ print $1 }' )
if [ "`echo \"${FREESPACE} > ${USEP}\" | bc`" = "1" ]; then
echo -e "\nThere is not enough free disk space available on\n\n\"${PARTITION} (${USEP} GB)\" \n\non $(hostname) as on $(date)\n\nto perform this migration. You need to free up\n\nsome disk space on ${BASEDIR} to
continue\n"
break
else
echo -e "\nThere is \"${USEP} GB\" free space, enough to continue\n\nall log files will be stored in ${LOGDIR}"
sleep 5
echo ""
fi
#####################################################################
## Set up logging the correct way
#####################################################################
BUILD_LOG=${LOGDIR}/`hostname | awk -F. '{print$1}'`-${CTIME}.${SCRIPTIDENTIFIER}.log
BUILD_PIPE=${BUILD_LOG}.pipe
if [ ! -e ${BUILD_PIPE} ]; then
mkfifo ${BUILD_PIPE}
fi
if [ -e ${BUILD_LOG} ]; then
rm ${BUILD_LOG}
fi
exec 3>&1 4>&2
tee ${BUILD_LOG} < ${BUILD_PIPE} >&3 &
TPID=$!
exec > ${BUILD_PIPE} 2>&1
trap "rm -f ${BUILD_PIPE} ; rm -f ${PIDFILE}" EXIT
#####################################################################
########################### ACCOUNTABILITY ##########################
#####################################################################
echo -e "\nPlease enter your name:\n"
read ADMIN_NAME
echo -e "\n\n\n\n"
echo "COMPANY NAME"
echo "${SCRIPTIDENTIFIER} Performed by: ${ADMIN_NAME}"
echo "On: ${CTIME}"
echo "Server: `hostname`"
echo "Logfile: ${BUILD_LOG}"
echo -e "\n\n\n\n"
#####################################################################
## START CODE
#####################################################################
# get rid of timedhosts.txt so yum will chose new fastest mirror
find /var/ -type f -name timedhosts.txt -exec rm -f "{}" \;
# repos we need
MONGODB_REPO="/etc/yum.repos.d/10gen-mongodb.repo"
CENTOS_REPO="/etc/yum.repos.d/CentOS-Base.repo"
EPEL_REPO="/etc/yum.repos.d/epel.repo"
PASSENGER_REPO="/etc/yum.repos.d/passenger.repo"
# where we are installing all software
APP_ROOT="/export"
# graylog2 server version to install
GLSV=0.11.0
# graylog2 web interface to install
GLWV=0.11.0
# elasticsearch version to install
ELSV=0.20.4
# set up our installation directory
if [ ! -d ${APP_ROOT} ]; then
mkdir -p ${APP_ROOT}
fi
# set up our log directory
if [ ! -d ${APP_ROOT}/logs ]; then
mkdir -p ${APP_ROOT}/logs
fi
if [ -f ${MONGODB_REPO} ]; then
echo "${MONGODB_REPO} found"
else
cat << 'EOF' > ${MONGODB_REPO}
[10gen]
name=10gen Repository
baseurl=http://downloads-distro.mongodb.org/repo/redhat/os/$basearch
gpgcheck=0
enabled=1
EOF
fi
if [ -f ${CENTOS_REPO} ]; then
echo "${CENTOS_REPO} found"
else
echo "Creating ${CENTOS_REPO}"
cat << 'EOF' > ${CENTOS_REPO}
# CentOS-Base.repo for RHEL6
[base]
name=CentOS-6 - Base
mirrorlist=http://mirrorlist.centos.org/?release=6&arch=$basearch&repo=os
#baseurl=http://mirror.centos.org/centos/6/os/$basearch/
gpgcheck=1
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-6
#released updates
[update]
name=CentOS-6 - Updates
mirrorlist=http://mirrorlist.centos.org/?release=6&arch=$basearch&repo=updates
#baseurl=http://mirror.centos.org/centos/6/updates/$basearch/
gpgcheck=1
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-6
EOF
fi
if [ -f ${EPEL_REPO} ]; then
echo "${EPEL_REPO} found"
else
echo "Creating ${EPEL_REPO}"
cat << 'EOF' > ${EPEL_REPO}
[epel]
name=Extra Packages for Enterprise Linux 6 - $basearch
#baseurl=http://download.fedoraproject.org/pub/epel/6/$basearch
mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-6&arch=$basearch
failovermethod=priority
enabled=1
gpgcheck=0
EOF
fi
if [ -f ${PASSENGER_REPO} ]; then
echo "${PASSENGER_REPO} found"
else
echo "Creating ${PASSENGER_REPO}"
cat << 'EOF' > ${PASSENGER_REPO}
### Name: Phusion Passenger RPM Repository for Red Hat Enterprise 6
### URL: http://passenger.stealthymonkeys.com/
[passenger]
name = Red Hat Enterprise $releasever - Phusion Passenger
baseurl = http://passenger.stealthymonkeys.com/rhel/$releasever/$basearch
mirrorlist = http://passenger.stealthymonkeys.com/rhel/mirrors
#mirrorlist = file:///etc/yum.repos.d/mirrors-passenger
enabled = 1
gpgcheck = 0
EOF
fi
# update for new repos and make sure system is up to date
yum -y update
# we need a build environment for rvm
yum install -y gcc-c++ patch readline readline-devel zlib zlib-devel libyaml-devel libffi-devel openssl-devel openssl make bzip2 autoconf automake libtool bison iconv-devel libyaml libyaml-devel
# disable selinux config (requires reboot for permanent disable)
# I am too lazy to fix selinux for all the ports required for this setup
# selinux causes issues with mongo, passenger, http, elasticsearch...pretty much everything in this software stack
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/sysconfig/selinux
# set temp to permissive
setenforce 0
# we set up rvm
curl -L https://get.rvm.io | bash -s stable --ruby
source /usr/local/rvm/scripts/rvm
RVER=`ruby -v | awk -F" " '{print$2}'`
if [ "${RVER}" == "2.0.0p0" ]; then
echo "ruby successfully set up"
echo "making it the default version"
rvm --default use 2.0.0
# add root user to rvm group
usermod -a -G rvm root
else
echo "ruby not set up correctly"
exit
fi
gem install bundle
gem update
# we have to make sure mongo is not installed from default packages....
if [ "`rpm -qa | grep mongodb | wc -l`" = "0" ]; then
echo "no older version of mongo installed"
else
echo -e "we have found an old version of mongo installed. ...removing"
echo -e "\nif you are using this version of mongo, press ctl+c to exit"
echo -e "\nyou will then need to comment out the parts of mongo in this script related to installation"
echo -e "\nI am not sure this software stack will work with an older version of mongo"
sleep 60
yum -y remove mongodb mongodb-server libmongodb
fi
echo "Installing MongoDB..."
yum install -y mongo-10gen mongo-10gen-server
# i had troubles with the above failing....it appears to be a server overloaded issue, yum would time out before install, so now we have this check
if [ "`rpm -qa | egrep \"mongo-10gen|mongo-10gen-server\" | wc -l`" = "2" ]; then
echo "mongodb rpm installation was successful"
else
echo "mongodb rpm installation failed....bailing"
exit
fi
# make sure services are stopped before we relocate
service mongod stop
/etc/init.d/mongod stop
echo "moving mongo db to ${APP_ROOT}/mongo/"
rsync -av /var/lib/mongo/ ${APP_ROOT}/mongo/
echo "removing original /var/lib/mongo"
rm -rf /var/lib/mongo
echo "creating symlink to ${APP_ROOT}/mongo"
ln -s ${APP_ROOT}/mongo /var/lib/mongo
# moving /var/log/mongo to /export/logs/mongo
echo "moving mongo logs to ${APP_ROOT}/logs/mongo/"
rsync -av /var/log/mongo/ ${APP_ROOT}/logs/mongo/
echo "removing original /var/log/mongo"
rm -rf /var/log/mongo
echo "creating symlink to ${APP_ROOT}/logs/mongo"
ln -s ${APP_ROOT}/logs/mongo /var/log/mongo
echo "Installing init script of MongoDB..."
chkconfig --level 2345 mongod on
echo "Starting MongoDB..."
service mongod start
echo "Installing ElasticSearch..."
cd ${APP_ROOT}
# all the documentation I have seen for graylog2 installs says to use elasticsearch 0.20.4
# however, I have an installation working just fine with 0.20.5 so if you have issues, suspect the version number
curl -L http://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-${ELSV}.tar.gz | tar -xz
ln -s elasticsearch-${ELSV}/ elasticsearch
echo "Installing ElasticSearch service wrapper..."
cd elasticsearch/bin
curl -L http://github.com/elasticsearch/elasticsearch-servicewrapper/tarball/master | tar -xz
mv elasticsearch-elasticsearch-servicewrapper-*/service .
rm -rf elasticsearch-elasticsearch-servicewrapper-*
cd ${APP_ROOT}
useradd elasticsearch -d ${APP_ROOT}/elasticsearch
chown -R elasticsearch:elasticsearch elasticsearch*
sed -i "s/# cluster.name: elasticsearch/cluster.name: graylog2/g" ${APP_ROOT}/elasticsearch/config/elasticsearch.yml
sed -i "s/set.default.ES_HOME=<Path to ElasticSearch Home>/set.default.ES_HOME=${APP_ROOT}/elasticsearch/g" ${APP_ROOT}/elasticsearch/bin/service/elasticsearch.conf
sed -i -e "s/#RUN_AS_USER=/RUN_AS_USER=elasticsearch/g" ${APP_ROOT}/elasticsearch/bin/service/elasticsearch
echo "Installing init script of ElasticSearch..."
${APP_ROOT}/elasticsearch/bin/service/elasticsearch install
echo "Starting ElasticSearch..."
service elasticsearch start
# testing
echo "data from elasticsearch should return here...."
sleep 5
curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'
chkconfig --add elasticsearch
chkconfig elasticsearch on
echo "Installing graylog2-server..."
cd ${APP_ROOT}
curl -L http://download.graylog2.org/graylog2-server/graylog2-server-${GLSV}.tar.gz | tar -xz
ln -s graylog2-server-${GLSV} graylog2-server
cd graylog2-server
mv graylog2.conf.example graylog2.conf
mv elasticsearch.yml.example graylog2-elasticsearch.yml
cd /etc
ln -s /${APP_ROOT}/graylog2-server/graylog2.conf graylog2.conf
ln -s /${APP_ROOT}/graylog2-server/graylog2-elasticsearch.yml graylog2-elasticsearch.yml
sed -i 's/mongodb_useauth = true/mongodb_useauth = false/g' /etc/graylog2.conf
# the value below is the total number of messages to keep in each index * number of indexes for total messages
sed -i 's/elasticsearch_max_docs_per_index = 20000000/elasticsearch_max_docs_per_index = 10000000/g' /etc/graylog2.conf
# the value below is the number of indexes to keep. If you get to this number, the oldest index will be deleted
sed -i 's/elasticsearch_max_number_of_indices = 20/elasticsearch_max_number_of_indices = 10/g' /etc/graylog2.conf
# total number of shards to keep, a good rule of thumb, 1 shard per elasticsearch node
sed -i 's/elasticsearch_shards = 4/elasticsearch_shards = 1/g' /etc/graylog2.conf
# the number of minutes of "recent" messages kept on the web page when you log in - for high volume log servers, 1 minute
sed -i 's/recent_index_ttl_minutes = 60/recent_index_ttl_minutes = 1/g' /etc/graylog2.conf
echo "Installing init script of graylog2-server..."
cat << 'EOF' > /etc/init.d/graylog2-server
#! /bin/sh
### BEGIN INIT INFO
# Provides: gaylog2-server
# Required-Start: $all
# Required-Stop: $all
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Graylog2-server init script
# Description: Graylog2-server init script
### END INIT INFO
## Source function library.
. /etc/init.d/functions
NAME="graylog2-server"
GRAYLOG2_HOME=
GRAYLOG2_JAR=$GRAYLOG2_HOME/graylog2-server.jar
GRAYLOG2_CONF="/etc/graylog2.conf"
GRAYLOG2_OUT="/var/log/graylog2.log"
PID_DIR="/var/run/graylog2"
PID_FILE="${PID_DIR}/${NAME}.pid"
LOCK_FILE="/var/lock/subsys/${NAME}"
JAVA="/usr/bin/java"
JAVA_OPTS="-server -Xms512m -Xmx512m"
RUN_AS_USER=root
if [ ! -d "$PID_DIR" ]; then
mkdir "$PID_DIR"
chown $RUN_AS_USER "$PID_DIR"
fi
if [ ! -r "$GRAYLOG2_JAR" ]; then
echo "Cannot find $GRAYLOG2_JAR"
echo "${GRAYLOG2_JAR} is absent or does not have read permission"
exit 1
fi
touch "$GRAYLOG2_OUT"
chown $RUN_AS_USER "$GRAYLOG2_OUT"
start() {
echo "Starting $NAME: "
COMMAND="$JAVA $JAVA_OPTS -jar $GRAYLOG2_JAR -f $GRAYLOG2_CONF -p $PID_FILE >> \"$GRAYLOG2_OUT\" 2>&1 &"
# for debugging enable next line
# COMMAND="$JAVA $JAVA_OPTS -jar $GRAYLOG2_JAR --debug -f $GRAYLOG2_CONF -p $PID_FILE >> \"$GRAYLOG2_OUT\" 2>&1 &"
# for statistics enable the next line
# COMMAND="$JAVA $JAVA_OPTS -jar $GRAYLOG2_JAR --statistics -f $GRAYLOG2_CONF -p $PID_FILE >> \"$GRAYLOG2_OUT\" 2>&1 &"
daemon --check=${NAME} --user=${RUN_AS_USER} --pidfile=${PID_FILE} ${COMMAND}
RETVAL=$?
[ $RETVAL -eq 0 ] && touch $LOCK_FILE
return $RETVAL
}
stop() {
echo "Stoping $NAME: "
killproc -p ${PID_FILE} -d 5 ${NAME}
RETVAL=$?
[ $RETVAL -eq 0 ] && rm -f $LOCK_FILE
return $RETVAL
}
restart() {
stop
sleep 1
start
}
dump() {
echo "Dumping $NAME: "
PID=`cat $PID_FILE`
kill -3 $PID
RETVAL=$?
[ $RETVAL -eq 0 ] && success $"Dumped $NAME." || failure $"Failed to dump $NAME."
return $RETVAL
}
rh_status() {
status -p ${PID_FILE} ${NAME}
}
rh_status_q() {
rh_status >/dev/null 2>&1
}
case "$1" in
start)
rh_status_q && exit 0
start
;;
stop)
rh_status_q || exit 0
stop
;;
restart)
restart
;;
status)
rh_status
;;
dump)
rh_status_q || exit 0
dump
;;
condrestart|try-restart)
rh_status_q || exit 0
restart
;;
*)
echo "Usage: $0 {start|stop|status|restart|condrestart|try-restart|dump}"
exit 1
esac
exit $?
EOF
sed -i "s/GRAYLOG2_HOME=/GRAYLOG2_HOME=\\${APP_ROOT}\/graylog2-server/g" /etc/init.d/graylog2-server
chmod u+x /etc/init.d/graylog2-server
chkconfig graylog2-server on
echo "Starting graylog2-server..."
service graylog2-server start
echo "Installing graylog2-web-interface..."
#yum install -y ruby ruby-devel ruby-rdoc rubygems
yum install -y httpd mod_passenger
cd ${APP_ROOT}
curl -L http://download.graylog2.org/graylog2-web-interface/graylog2-web-interface-${GLWV}.tar.gz | tar -xz
ln -s graylog2-web-interface-${GLWV} graylog2-web-interface
cd graylog2-web-interface
# fix the Gemfile - has wrong versions of json bison bison_ext
cat << 'EOF' > ${APP_ROOT}/graylog2-web-interface/Gemfile
source :rubygems
gem 'rack', '~> 1.4.1'
gem 'rake', '~> 0.9.2'
gem 'rails', '~> 3.2.12'
gem 'json', '~> 1.7.7'
gem 'chronic', '~> 0.6.7'
gem 'pony', '~> 1.1' # unusual version number
gem 'graylog2-declarative_authorization', '~> 0.5.2', :require => 'declarative_authorization'
gem "mongo", "~> 1.8.2"
gem 'mongoid', '2.4.5'
gem "tire", "~> 0.5.1"
gem 'bson', "~> 1.8.2"
gem 'bson_ext', "~> 1.8.2", :platforms => :ruby
gem 'home_run', '~> 1.0.2', :platforms => :ruby
gem 'SystemTimer', '~> 1.2.3', :require => 'system_timer', :platforms => :ruby_18
gem 'rails_autolink', '~> 1.0.4'
gem 'kaminari', '~> 0.12.4'
gem 'jquery-rails', '~> 2.1'
gem 'therubyracer', '~> 0.10.2'
gem 'net-ldap', '~> 0.3.1'
group :development, :test do
# might be useful to generate fake data in development
gem 'machinist_mongo', '~> 1.2.0', :require => 'machinist/mongoid'
gem 'faker', '~> 0.9.5'
end
group :development do
# gem 'ruby-prof', '~> 0.10.5' # works nice with NewRelic RPM Developer Mode
gem 'passenger', '~> 3.0.17'
end
group :test do
gem 'ci_reporter', '~> 1.6.4'
gem 'shoulda', '~> 2.11.3'
gem 'shoulda-activemodel', '0.0.2', :require => 'shoulda/active_model' # fixed version - too hacky
gem 'mocha', '~> 0.9.12'
gem 'database_cleaner', '~> 0.6.0'
gem 'timecop', '~> 0.3.5'
end
# Needed for the new asset pipeline
group :assets do
gem 'sass-rails', "~> 3.2.5"
gem 'coffee-rails', "~> 3.2.2"
gem 'uglifier', ">= 1.0.3"
end
EOF
# mongo gem required for http user
bundle update mongo json bson_ext
bundle install --without=development
# set up a graylog user
useradd graylog -d ${APP_ROOT}/graylog2-web-interface -G rvm
chown -R graylog2:graylog2 ${APP_ROOT}/graylog2-web-interface
usermod -g rvm root
source /etc/profile.d/rvm.sh
# set up the first graylog2 user
cd ${APP_ROOT}/graylog2-web-interface
#su graylog
# requried for next step
yum -y install lynx
echo "Open another shell on this server and browse to http://IPADDRESS:3000 and setup your first user and login."
sleep 20
# Note there is bug with the current mongo driver that gives a notice about bson_ext not being loaded. It will be fixed in a future version.
RAILS_ENV=production script/rails server &
echo "after password is created, kill the server in the shell"
sleep 20
# install passenger
gem install passenger file-tail
# a fix to the bug in passenger <=3.0.19
cat << 'EOF' > /usr/local/rvm/gems/ruby-2.0.0-p0/gems/passenger-3.0.19/build/gempackagetask.rb
#!/usr/bin/env ruby
# Passenger note: this file is copied from Rake 0.8.1. The task names
# have been changed.
# Define a package task library to aid in the definition of GEM
# packages.
require 'rubygems'
require 'rake'
require 'build/packagetask'
require 'rubygems/user_interaction'
if /^2\./ =~ RUBY_VERSION
require 'rubygems/package'
else
require 'rubygems/builder'
end
module Rake
# Create a package based upon a Gem spec. Gem packages, as well as
# zip files and tar/gzipped packages can be produced by this task.
#
# In addition to the Rake targets generated by PackageTask, a
# GemPackageTask will also generate the following tasks:
#
# [<b>"<em>package_dir</em>/<em>name</em>-<em>version</em>.gem"</b>]
# Create a Ruby GEM package with the given name and version.
#
# Example using a Ruby GEM spec:
#
# require 'rubygems'
#
# spec = Gem::Specification.new do |s|
# s.platform = Gem::Platform::RUBY
# s.summary = "Ruby based make-like utility."
# s.name = 'rake'
# s.version = PKG_VERSION
# s.requirements << 'none'
# s.require_path = 'lib'
# s.autorequire = 'rake'
# s.files = PKG_FILES
# s.description = <<EOF
# Rake is a Make-like program implemented in Ruby. Tasks
# and dependencies are specified in standard Ruby syntax.
# EOF
# end
#
# Rake::GemPackageTask.new(spec) do |pkg|
# pkg.need_zip = true
# pkg.need_tar = true
# end
#
class GemPackageTask < PackageTask
# Ruby GEM spec containing the metadata for this package. The
# name, version and package_files are automatically determined
# from the GEM spec and don't need to be explicitly provided.
attr_accessor :gem_spec
# Create a GEM Package task library. Automatically define the gem
# if a block is given. If no block is supplied, then +define+
# needs to be called to define the task.
def initialize(gem_spec)
init(gem_spec)
yield self if block_given?
define if block_given?
end
# Initialization tasks without the "yield self" or define
# operations.
def init(gem)
super(gem.name, gem.version)
@gem_spec = gem
@package_files += gem_spec.files if gem_spec.files
end
# Create the Rake tasks and actions specified by this
# GemPackageTask. (+define+ is automatically called if a block is
# given to +new+).
def define
super
task :package => ['package:gem']
desc "Build the gem file #{gem_file}"
task 'package:gem' => ["#{package_dir}/#{gem_file}"]
file "#{package_dir}/#{gem_file}" => [package_dir] + @gem_spec.files do
when_writing("Creating GEM") {
if /^2\./ =~ RUBY_VERSION
Gem::Package.build(gem_spec)
else
Gem::Builder.new(gem_spec).build
end
verbose(true) {
mv gem_file, "#{package_dir}/#{gem_file}"
}
}
end
end
def gem_file
if @gem_spec.platform == Gem::Platform::RUBY
"#{package_name}.gem"
else
"#{package_name}-#{@gem_spec.platform}.gem"
end
end
end
end
EOF
# install the apache module
passenger-install-apache2-module
# and the passenger.conf file for apache
cat << EOF >> /etc/httpd/conf.d/passenger.conf
LoadModule passenger_module /usr/local/rvm/gems/ruby-2.0.0-p0/gems/passenger-3.0.19/ext/apache2/mod_passenger.so
PassengerRoot /usr/local/rvm/gems/ruby-2.0.0-p0/gems/passenger-3.0.19
PassengerRuby /usr/local/rvm/wrappers/ruby-2.0.0-p0/ruby
PassengerPoolIdleTime 0
RailsFrameworkSpawnerIdleTime 0
RailsAppSpawnerIdleTime 0
PassengerMaxPoolSize 7
<VirtualHost *:80>
ServerName logs.tritondigital.net
ServerAlias logs.tritondigital.net
ServerAlias aws-east1d-pro-flex-log-01.tritondigital.net
DocumentRoot ${APP_ROOT}/graylog2-web-interface/public
RailsEnv 'production'
SetEnv MONGOID_HOST 'localhost'
SetEnv MONGOID_PORT '27017'
SetEnv MONGOID_DATABASE 'graylog2'
<Directory ${APP_ROOT}/graylog2-web-interface/public>
Allow from all
Options -MultiViews
</Directory>
</VirtualHost>
EOF
chown -R apache:apache ${APP_ROOT}/graylog2-web-interface
chkconfig --level 2345 httpd on
service httpd restart
#####################################################################
## END CODE
#####################################################################
###########################################################################
## LOGGING CLEANUP
###########################################################################
# 1>&3 and 2>&4 restore the original file descriptors for stdout and
# stderr from file descriptors 3 and 4.
# 3>&~ 4>&~: now that they are not needed, close file descriptors 3 and 4
exec 1>&3 3>&- 2>&4 4>&-
# waits on the tee pid to die before continuing
wait ${TPID}
# remove the named pipe which is no longer needed
sleep 5
rm ${BUILD_PIPE}
##############################################################################
## MAIL REPORT
##############################################################################
echo "using mutt to send ${BUILD_LOG} to ${EMAILS}"
bzip2 -9 ${BUILD_LOG}
# ubuntu and arch require -- before #{EMAILS}, redhat does not
if [ -f /etc/redhat-release ]; then
mutt -s "${SCRIPTIDENTIFIER} log for `hostname | awk -F. '{print$1}'`" -a ${BUILD_LOG}.bz2 ${EMAILS} < /dev/null
else
mutt -s "${SCRIPTIDENTIFIER} log for `hostname | awk -F. '{print$1}'`" -a ${BUILD_LOG}.bz2 -- ${EMAILS} < /dev/null
fi
#####################################################################
## remove pidfile create at start
rm ${PIDFILE}
#####################################################################
[/codesyntax]
For sending the log events to the log server, you can use this logstash script I also made:
[codesyntax lang=”bash”]
input {
file {
type => "syslog"
path => [ "/var/log/messages", "/var/log/dmesg", "/var/log/cron", "/var/log/secure", "/var/log/syslog" ]
}
file {
type => "nginx-access"
path => ["/var/log/nginx/access.log", "/var/log/nginx/error.log"]
}
}
output {
gelf {
chunksize => "1420"
facility => "logmessages-gelf"
host => "log_server_ip_or_FQDN"
port => "12201"
}
}
[/codesyntax]
I also have rpms build for each of the logstash dependancies for centos 6.3 but unfortunately they contain proprietary config files. I will try to get a set rebuild without the proprietary configs.
Here is also a script that will remove the entire installation, in the event you mess something up. It removes the major components only:
[codesyntax lang=”bash”]
#!/bin/bash
# By Ed Wiget
# Easily remove all changes to the system for install-graylog2.sh
APP_ROOT="/export"
# stop services
#service graylog2-web-interface stop # no longer needed switched from ruby nginx to apache
service graylog2-server stop
service mongod stop
service elasticsearch stop
service httpd stop
#service rabbitmq-server stop
#service logstash stop
# remove mongo rpms
yum remove -y mongo-10gen mongo-10gen-server
# delete directories and software packages
rm -rf ${APP_ROOT}/elasticsearch*
rm -rf ${APP_ROOT}/graylog2-server*
rm -rf ${APP_ROOT}/graylog2-web-interface*
rm -rf ${APP_ROOT}/mongo
# remove logs
rm -rf /var/log/mongo
# remove symlink
rm -rf /var/lib/mongo
# remove /etc config files
rm -f /etc/graylog*
# remove /etc/init.d/files
rm -f /etc/init.d/graylog*
rm -f /etc/init.d/elasticsearch
# remove passenger.conf from /etc/httpd/conf.d/passenger.conf
rm -f /etc/httpd/conf.d/passenger.conf
echo "done"
[/codesyntax]
A script to start all the services and a script to stop all the services:
[codesyntax lang=”bash”]
#!/bin/bash service elasticsearch start service mongod start service graylog2-server start service httpd start #service rabbitmq-server start echo "checking running services......" echo -e "\n\n" ps aux | egrep "[e]lasticsearch|[m]ongo|[g]raylog2-server|[h]ttp"
[/codesyntax]
[codesyntax lang=”bash”]
#!/bin/bash # By Ed Wiget # stops graylog service graylog2-server stop service mongod stop service elasticsearch stop service httpd stop #service rabbitmq-server stop echo "checking running services......" echo -e "\n\n" ps aux | egrep "[e]lasticsearch|[m]ongo|[g]raylog2-server|[h]ttp"
[/codesyntax]
Leave a Reply
You must be logged in to post a comment.