The Perils of TRUST - An IS Auditors Nightmare

I have worked for many years as a systems/network administrator, I have learned to live and adopt with the changing patterns of IT, which we have grown to know well. I have made enemies doing the right thing. I made friends to those enemies whom has done the wrong thing. In the end, we are rational beings tending along the stages of life. There are some science that just cannot be corrected due to its mortal beginnings and one of this truth is "TRUST"

To TRUST is to be believe that a thing, undertaking, and/or idea will work according to an individuals plans and desires. It is our belief system which we choose to adopt shall provide us the necessary cognitive apprehension of what "IS" and what "WAS". As an auditor, we are thought the high gaining values of TRUTH and HONESTY. When we learn to TRUST on something which we believe will generate a fair TRUTHFUL and HONEST outcome, we tend to relax a bit and put down our guard. This should not be the case.

TRUST is the most priced trait of an individual who PRIDES himself/herself of his accomplishment and works. Auditors are not an exemption. We are continually targeted by MALICIOUS self-centered agents of the trade. These agents DISGUISE themselves as CO-WORKERS, CLIENTS, FRIENDS and CORRUPTED POLICIES designed to harbor all the LIES and DECEIT man can think off. Therefore, it is our sworn duty to JUSTLY identify these agents and remove them from the SYSTEM. The SYSTEM is what we serve and though the SYSTEM we grow. Treat it with respect and it will reward you with peace of mind. Treat it with a twisted Intent and you are doomed to have sleepless nights.

The reason, why I find this so compelling is that I worked as a systems engineer for a good deal of time and I have learned everything there is necessary to understand what the DARK desires of an admin are and what they can do to a fellow admin.

SCENARIO:

A systems administrators builds a new server performs the necessary hardening and then performs the necessary ..... [to be continued ...]

Error during kernel upgrade: gzip: stdout: No space left on device

There are times when you will be surprised that package managers does not automatically remove older archives of the software installed. This happened to in for the first time, when one of the systems I was managing suddenly returned an exit status 1.

Removing the offensive application to free up much needed space is sure to fail! especially if the application mentioned here is a kernel.

Consider this line:
------------------------------

Setting up libcups2 (1.4.6-5ubuntu1.3) ...
dpkg: dependency problems prevent configuration of linux-image-generic:
linux-image-generic depends on linux-image-2.6.38-10-generic; however:
Package linux-image-2.6.38-10-generic is not configured yet.
dpkg: error processing linux-image-generic (--configure):
dependency problems - leaving unconfigured
No apport report written because the error message indicates its a followup error from a previous failure.
dpkg: dependency problems prevent configuration of linux-generic:
linux-generic depends on linux-image-generic (= 2.6.38.10.25); however:
Package linux-image-generic is not configured yet.
dpkg: error processing linux-generic (--configure):
dependency problems - leaving unconfigured
No apport report written because the error message indicates its a followup error from a previous failure.
Setting up linux-headers-2.6.38-10 (2.6.38-10.46) ...
Setting up linux-headers-2.6.38-10-generic (2.6.38-10.46) ...
Setting up linux-headers-generic (2.6.38.10.25) ...
Processing triggers for initramfs-tools ...
update-initramfs: Generating /boot/initrd.img-2.6.38-8-generic

gzip: stdout: No space left on device
E: mkinitramfs failure cpio 141 gzip 1
update-initramfs: failed for /boot/initrd.img-2.6.38-8-generic
dpkg: error processing initramfs-tools (--configure):
subprocess installed post-installation script returned error exit status 1
No apport report written because MaxReports is reached already
Processing triggers for libc-bin ...
ldconfig deferred processing now taking place
Errors were encountered while processing:
linux-image-2.6.38-10-generic
linux-image-generic
linux-generic
initramfs-tools
E: Sub-process /usr/bin/dpkg returned an error code (1)


If you are guessing that /boot is in deep trouble. You are correct. Now, the tricky part is this. Issue a purge; autoremove won't do the trick because the drive no longer has enough space. What do you do next?

1. If it is a kernel (old one) that needs to be removed, look closely at your grub.conf or grub.cfg configuration and Identify the kernels you no longer need. Check the currently loaded kernel by issuing uname -r.

2. Take note of the files that needs to be moved, in our example its a debian based system so you will have to move these files: abi, config, initrd.img, System.map, vmcoreinfo, vmlinuz. Just remove/move those that you don't need.

3. Once done issue the command updatedb to update the slocate database of the filesystem

4. Now you can issue the upgrade command back and this will install the new kernel correctly.

Time Snatchers

For the not so unconventional and always looking forward admin, I have a few brewed time killers to get you, looking back at how it used to be when things aren't too serious.

Movies:

1. Pirates of the Silicon Valley -- the Apple and Microsoft love-affair
2. Revolution OS - Documentary, it highlights the advocacies of Open Source and the Free Software Foundation.
3. VODO.NET - if you are a true advocate of Creative Commons, then find the time to download there huge array of mini-series,documents and short films. This is how tv should be, the viewers chooses if-when-where-it-ends!

Books:

1. Probably not everybody's choice but worth the read: Harry Potter (Books 1-7) ain't too old to have time discussing things with the kids. It brings out the ideas in them and sponsors confidence, trust, good learning through reading. A much needed exercise is not to common these days especially for the youth of today.

2. The Art of Unix Programming: A defacto standard! Understand why Unix is still around today and what universal "chi" it has spawned through its 40 years of existence.

3. Cathedral and the Bazzar: Every known X "As" a service must have read this book. Truly worth while.


Sports/Leisure -- Family:

Going to Church, Malling, Dinning with the Family. It gives you a sense of purpose on why you have been working so hard and where it is all being poured to.

Why "curl" is way better than "wget"

I am an OLD SCHOOL admin tought from the old school class of using wget, I guess its time to move on. In this section I will highlight to you some very important key features on why curl is way too robust than wget.

Curl is better than wget for the following reasons:
1. Uses libcurl a cross platform library
2. curl sends more stuff to stdout and reads more from stdin
3. curl supports ftp, ftps, http, https, scp sftp, tftp, telnet, dict, ldap and ldaps while wget supports only http, https and ftp
4. curl has SSL support
5. libcurl supports more http authentication methods
6. Curl is bidirectional while wget offers http post support only
7. Curl has more development activities

Example:

curl -l -O http://nginx.org/download/nginx-1.0.5.tar.gz

The one advantage I see in using wget is it’s ability to download recursively.
In short curl is better and more powerful. I actually don't need to install it on most UNIX servers as curl is already available by default.


ATTRIBUTION: http://jayamorin.blogspot.com/2011/02/curl-for-wget.html

12 Reasons Why Every Linux System Administrator Should be Lazy

Lazy sysadmin is the best sysadmin –Anonymous

System administrators job is not visible to other IT groups or end-users. Mostly they look at administrators and wonder why sysadmins don’t seem to have any work.

If you see a sysadmin who is always running around, and trying to put down fire, and constantly dealing with production issues, you might think he is working very hard, and really doing his job. But in reality he is not really doing his job.

If you see a sysadmin (UNIX/Linux sysadmin, or DBA, or Network Administrators), who doesn’t seem to be doing much around the office that you can see, he always seem to be relaxed, and he don’t seem to have any visible work, you can be assured that he is doing his job.

The following are the 12 reasons why a lazy sysadmin is the best sysadmin.

Who is the boss? The main reason why lazy sysadmin is the best sysadmin is because of his attitude. They look at the machines little differently than how other IT departments looks at them. There is a difference between developers and sysadmins. Developers thinks they are here to serve the machines by developing code. There is nothing wrong in this approach, as developers have lot of fun developing the code. But, sysadmins think other way around. They think the machines are there to serve them. All they have to do is feed the machine and keep it happy, and let the machine do all the heavy duty job, while they can relax and just be lazy. The first step in being a lazy sysadmin is a slight change in attitutde, and letting the machine know that you are the boss.

Write scripts for repeated jobs. Being lazy means being smart. A smart sysadmin is a master in all scripting languages (bash, awk, sed, etc.,). Anytime he is forced to do some work, and if there is a remote possibility that the work might be needed in the future, he writes a script to complete the job. This way, in the future when he was requested to do the same job, he doesn’t have to think; he just have to execute the script, and get back to being lazy.

Backup everything. Being lazy means taking backup. A lazy sysadmin knows that he has to put little work in creating a backup process, and write backup scripts for all critical systems and applications. When the disk space is not an issue, he schedules the backup job for every application, even for those that are not critical. This way, when something goes wrong, he doesn’t have to break a sweat, and just have to restore from the backup, and get back to whatever lazy stuff he was doing before. This is also the rule#1 in the three sysadmin rules that you shouldn’t break.

Create a DR plan. Sysadmins doesn’t like to run around when things go wrong. When things are running smoothly, they take some time to create a DR plan. This way, when things go wrong, they can follow the DR plan and quickly get things back to normal, and get back to being lazy again.

Configure highly redundant systems. Lazy sysadmins don’t like to get calls in the middle of the night because of some silly hardware failure problem. So, they make sure all the components are highly redundant. This includes both hardware and software. They have dual network card configured, they have dual power, they have dual hard drives, they have dual of everything. This way, when one component fails, the system still keeps running, and the lazy sysadmin can work on fixing the broken component after he wakes-up in the morning.

Head room for unexpected growth. Lazy sysadmin never allows his system to run in full capacity. He always has enough head room for unexpected growth. He make sure the system has plenty of CPU, RAM and hard disk available. When the business unit decides to dump tons of data over night, he doesn’t have to think about how to handle that unexpected growth.

Be proactive. Being lazy doesn’t mean you just sit and do nothing all the times. Being lazy means being proactive. Lazy sysadmins hate being reactive. They are always anticipating issues and anticipating growth. When they have some free time in their hand, they always work on proactive projects that helps them to avoid unexpected future issues, and to handle future growth.

Loves keyboard shortcut. Lazy sysadmin knows all the keyboard shortcuts for all his favorite applications. If he spends significant time everyday on an application, the first thing he’ll do is to master the keyboard shortcut for that application. He likes to spends less them on the application to get his things done, and likes to get back to being lazy.

Command line master. Every lazy sysadmin is a command line master. This applies to Linux sysadmin, dba, network administrator, etc. If you see an administrator launching a GUI, when the same task can be done from the command line, then you know he is not a lazy sysadmin. There are two reasons why lazy sysadmin loves command line. For one, he can do things quickly at the command line. For another, it makes him feel that he is the boss and not the system. When you use the command line, you are in control, you know exactly what you want to do. When you use GUI, you are at the mercy of the GUI workflow, and you are not in control.

Learns from mistake. Lazy sysadmin never likes to make the same mistake twice. He hates to work on unexpected issues. But, when an unexpected issue happens, he works on fixing it, and thinks about why it happened, and he immediately puts necessary things in place so that the same issue doesn’t happen again. Working on the same problem twice is a sin according to lazy sysadmin. He likes to work on the problem only once, do things to prevent the same mistake from happening in the future, and get back to being lazy.

Learn new technology. There is nothing wrong in learning a new technology to get a better job, or just to keep up with technology growth. But, lazy sysadmin doesn’t learn new technology for this reason. Instead, he learns new technology because he likes to be in control of the systems all the times. He knows he is the boss, and not the machine. So, when a new technology comes, he takes time to study them. Now he has new tools that he can use to keep the system busy, while he continue to be lazy. He learns new technology just for selfish lazy reason.

Document everything. Not every lazy sysadmin does this. Only the best lazy sysadmins does this. You see, lazy sysadmin never likes to be disturbed when he is on the beach enjoying his vacation. So, what does he do? He documents everything, so that when he is not around, other junior sysadmins can do the routine job, and get things moving without disturbing his vacation. There is also another reason for the lazy sysadmin to document everything; because he forgets things. Since he is lazy, he tends to forget what he did a month ago. Since he never likes to think and research the same topic twice, he documents everything, and when he needs to do the same thing in the future, he goes back to his documentation to understand what he did earlier.

Probably you are now convinced that begin a lazy sysadmin is not that easy. It is lot of hard work. If you are not a sysadmin, you can now appreacie a lazy sysadmin when you see one. If you are sysadmin, and always running around, now you know what you need to do to be lazy.

ATTRIBUTION: TechRepublic from one of their recent news 2011 copyright.

CentOS 6.0 is here

Waiting for the arrival of branch release updates for the 6.1 version. Took, some time for CentOS community to put this up. So, what is in-store for us? Plenty of new features something to look forward if you plan to move your apps and projects to be hosted on this platform. The links provided below will give you a wealth of information what to expect.

http://www.centos.org/
http://isoredirect.centos.org/centos/6/isos/
http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html-single/6.0_Release_Notes/index.html

Everybody, Somebody, Anybody, and Nobody

This is a little story about four people named Everybody, Somebody, Anybody, and Nobody.

There was an important job to be done and Everybody was sure that Somebody would do it.

Anybody could have done it, but Nobody did it.

Somebody got angry about that because it was Everybody's job.

Everybody thought that Anybody could do it, but Nobody realized that Everybody wouldn't do it.

It ended up that Everybody blamed Somebody when Nobody did what Anybody could have done.


original link: http://www.pimp-my-rig.com/2009/05/everybody-somebody-anybody-and-nobody.html

How Sitting Can Actually Kill Systems Admins

Just came across this link that details the truth about sitting. Well most of the time Systems Admins sit down. Wired in on there computers and work. Think again .... I guess I am one of them. :D


Via: Medical Billing And Coding

Rescue Linux using Live CD via chroot

There are times when you mess up something on the system and you need to rescue it using safe boot etc.  However, this can be a real daunting task if you need a fast cure to most common problems.  Like for example if you have stuff where you need to remount and extend partitions etc.

Solution:  Boot using Live-CD!

1. Boot to your Live-CD environment
2. Create a virtual directory to mount the partition
3. Issue the command:  #mount -t fs.type -o "options" /path/partion /path/directory
4. Issue the command #chroot /path/directory

This drops you to the working directory of your current system and fix things.  Note, that you may have to mount all partitions to the same directory to get all the libraries needed to run the system correctly.

Create files with a leading dash "-" in Linux/Unix

If you happen to need a file created with a leading dash "-" all you have to do is add the double "--" and the "-"  

Example:  touch -- -thisfile  

The concept is pretty much the same as on how you would delete,move,edit files with a leading dash.

Example:  rm -- -thisfile
                 mv -- -thisfile
                 vi -- -thisfile
                

This is handy if for example you have sites that needs id verification from site rating providers etc. 

It's one of those days when you really need to get things done immediately but you have to browse the Internet for some answers.  Cheers!

Removing old keys in your known_hosts file

If you happen to have changed server IP address and while at it you get this message doing a remote connnection:

ssh ants@192.168.1.1
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
eb:e8:de:47:00:77:2c:14:28:20:db:18:4c:9f:69:02.
Please contact your system administrator.
Add correct host key in /home/ants/.ssh/known_hosts to get rid of this message.
Offending key in /home/ants/.ssh/known_hosts:13
RSA host key for 192.168.1.1 has changed and you have requested strict checking.
Host key verification failed.

You will have to remove the entry by editing your known_hosts file.  Remove the old entry, in this case its on line "13" save the file and try connecting again, accept the new one when prompted.

Done.

SMS triggered service executions on Unix/Linux Systems,

There isn't enough material out there on the net that teaches systems administrators a thing or two about how to integrate your SMS monitoring to a complete real time system that allows you to execute commands via SMS.

In this post we will do just that.  The requirement is that you have an SMS server running and functional; plenty of time to brush up on your proof of concept skills.  It may not be the complete solution to what you have in mind but for me it works fine.  The goal is simply to respond to issues you deal on a day-to-day basis, albeit real-time response whenever a problem arises.

Planning 

Map out the logical flow behind your project.  We all know that SMS messages are triggered events, it could be automatic events or human response events.  A working SMS server will have both; by the time you receive a message coming from your monitoring system suggest that an automatic event has been triggered which based on the given configuration you made; sent out the notification informing you that such an event has taken place.  Now, often if a computer is within reach our first reflex-response is to login work on the issue.  This is the reality for the first given days, weeks and months.  Apparently, that ceases when you encounter too-many false positives. 


Working out a Real-time solution

To address this one-way communication between you and to those systems that you manage you have to be able to manage it directly via your mobile phone.  A cheap solution to a very urgent event which could mean alot to the company you manage.

.... to be continued ....

OpenKM: ERROR [MainDeployer] Could not create deployment: file:/opt/jboss-4.2.3.GA/server/default/conf/jboss-service.xml

If you happen to have a problem with this:  ERROR [MainDeployer] Could not create deployment: file:/opt/jboss-4.2.3.GA/server/default/conf/jboss-service.xml

This means that their is a problem with your host file.  Make sure that you have configured your hostname in line with your local ip address.

Install OCS Inventory Agents minus the Headaches

One of the major hurdles with OCS Inventory is understanding how the wiki pages addresses agent installation methods.  In my opinion the best documentation they had so far is written in french! So what I am going to do is make life easier!  Posting a hashed version which will allow you to install agents with very little problem.  Enough of the talk and lets get down to business.

Before installing the agent it is pressumed that you have OCS Inventory Server running correctly.  The goal in this mini-cookbook is to guide you without having to second guess how the installation might proceed as with the OCS pages do.

[For those using Linux/MacOS]

1)  Download the package http://launchpad.net/ocsinventory-unix-agent/stable-1.1.2/ocsinventory-unix-agent-1.1.2.1/+download/Ocsinventory-Agent-1.1.2.1.tar.gz  if you want a source install, on the otherhand if you are using newer versions of Debian/CentOS you can easily do an apt-get install "package name" and yum install "package name" though you will have to make some edits to your repos list to update your channel streams

2)  Process will check for availability of "perl" modules for features to work correctly. Please satisfy those dependencies before continuing.

3) During installation you will have to point your agent to listen to e.g.  172.16.24.109 <-- server. It is expected that you will have a  problem connecting this is fine. Once the agent is installed please proceed to item 4.

4) Edit /etc/ocsinventory-agent/ocsinventory-agent.cfg

          e.g.
             basevardir=/var/lib/ocsinventory-agent
             tag=
             server=xxx.xxx.xxx.xxx  <-- change this value to point to the OCS server.

5) Initialize a check, issue the command:

              ocsinventory-agent --debug --info --scan-homedirs --server=xxx.xxx.xxx.xxx
    This will send an agent report to the server. If this is successful  please proceed to item 6.

6) Daemonize the agent

              ocsinventory-agent --deamon --debug --info --scan-homedirs --server=xxx.xxx.xxx.xxx

     This will put the agent to work in the background.

7) To automatically allow your system to send updates to the inventory server you will have to bootstrap the previous command "item 6" on startup.


[For those using Windows] 

1) Download the installer package:  http://launchpad.net/ocsinventory-windows-agent/trunk/win32-agent-release-4061/+download/OCSNG_WINDOWS_AGENT_4061.1.zip  extract and install the package.
Look for the file "OcsAgentSetup.exe" to run setup.

2) On the Server Address use this IP: 172.16.24.109; Server Port: 80; Enable log file [check it]; Immediately launch inventory [check it].

3) Check if the service is running. You can do this by opening the "service management console > services" double click on the icon and check if OCS INVENTORY is "started" and the Startup type is set to "Automatic" your done.

Process should not take more than 30 minutes of your time. Login to the web console of the OCS Inventory Server and verify that the agents communicated your machine and has been registered.

Configuring Xymon for SMS Server Tools 3

I have done SMS configurations for a variety of monitoring tools in the past.  Nagios is one of them, however in this post we wont be dealing with nagios or any of those stuff related to nagios.  What we will be configuring is xymon/hobbit monitor to work with sms server tools 3.  If you notice, documentation from these two technologies are almost too stiff-straightforward and you have to do more than just read the man pages to get things done. 

The fun side in this tutorial is this; you will be re-using old and existing equipment to get the job done.  In my case I used an HSDPA modem to act as my SMS modem.  After that you will need an SMS gateway software to glue the entire project. 

Lets dig down to the details:

1. Configure your xymon server for mail notification alerts.  If you get alerts after a good configuration then you are ready to move foward with the next steps.

2. Download and install SMS Server Tools 3 http://smstools3.kekekasvi.com/packages/smstools3-3.1.14.tar.gz

3. Configure your modem to be detected by your system, it doesn't matter what NIX variant you are using.  In my case I used CentOS 5.5 to install and configure my HSDPA modem Huawei Technologies Co., Ltd. E220 HSDPA Modem / E270 HSDPA/HSUPA Modem.  It usually comes with a virtual cdrom and storage that can be detected by your system.  The goal here is remove those you dont need for you to use it only as a communication device.

4. Once you have installed SMS Server Tools 3 you will need to edit this file smsd.conf which can be located at /etc/smsd.conf and edit this line:
device=/dev/ttyS0 to device=/dev/ttyUSB1, save the file.

5. If you can see these lines from your tail log, it means you are doing good. 

$sudo /usr/bin/tail -f /var/log/smsd.log
2011-02-24 21:16:13,6, GSM1: Checking device for incoming SMS
2011-02-24 21:16:13,6, GSM1: Checking if modem is ready
2011-02-24 21:16:13,7, GSM1: -> AT
2011-02-24 21:16:13,7, GSM1: Command is sent, waiting for the answer
2011-02-24 21:16:13,7, GSM1: <- OK
2011-02-24 21:16:13,6, GSM1: Pre-initializing modem
2011-02-24 21:16:14,7, GSM1: -> ATE0+CMEE=1;+CREG=2
2011-02-24 21:16:14,7, GSM1: Command is sent, waiting for the answer
2011-02-24 21:16:14,7, GSM1: <- OK
2011-02-24 21:16:14,7, GSM1: -> AT+CSQ
2011-02-24 21:16:14,7, GSM1: Command is sent, waiting for the answer
2011-02-24 21:16:14,7, GSM1: <- +CSQ: 15,99 OK
2011-02-24 21:16:14,6, GSM1: Signal Strength Indicator: (15,99) -83 dBm (Good), Bit Error Rate: not known or not detectable
2011-02-24 21:16:14,6, GSM1: Checking if Modem is registered to the network

6. Test the sendsms binary, send a text message:  /usr/local/bin/sendsms 091781000102,  you should receive an sms message from your server.

7. Now you need to define the checks you need this will normally be equaivalent to the one you set for you email notification alerts In my case heres how it looks like:

SCRIPT /usr/local/bin/textalert FORMAT=sms
the "textalert" command is a simple bash script (or any scripting language you prefer would suffice) that handles the actual "sendsms" binary that actually sends out the messages. 

Done.  In the next post I will discuss escalations and other modifications necessary to make xymon work as you expect it to be.

This has been reposted to: http://www.howtoforge.com/node/6096

Apache Upgrade Woes affecting Compress::Zlib perl libraries

In this post I will sink deep on a problem I had when I upgraded my CentOS 5.3 to the latest 5.5 release.  The problem surfaced when I started working on apache for some changes.  Here is the haunting log of that problem.

[Mon Jan 10 17:34:29 2011] [error] Can't load Perl module Compress::Zlib for server netconsvr001.sulit.net:0, exiting...
[Mon Jan 10 17:34:46 2011] [error] dualvar is only available with the XS version of Scalar::Util at /usr/lib/perl5/site_perl/5.8.8/Compress/Zlib.pm line 8\nBEGIN failed--compilation aborted at /usr/lib/perl5/site_perl/5.8.8/Compress/Zlib.pm line 8.\nCompilation failed in require at (eval 7) line 3.\n

For the untrained eye this is a major disaster. Issuing the command $sudo package-clean --problem; $rpm -Va --nofiles --nodigest doesnt solve the problem worst you are even thinking of reverting back (rollback) changes to its original configuration.  Which is bad since you loose the availability of moving your box to a patch and security fixed version.

To solve the issue I did a "Hack" to the actual library that is causing the problem.

$sudo vi /usr/lib/perl5/site_perl/5.8.8/Compress/Zlib.pm  and checked on the problematic line (8)

1. Remove the qw(dualvar) entry on that line. Which looks like this:  use Scalar::Util qw(dualvar);

2. Save the file

3. Re-start apache:  service httpd restart

Viola!  Problem Fixed.

Hardening Apache on CentOS 5 with mod_security

Apache can be configured to use "mod_security".  Installing it can be quite a daunting task if you are new to apache or to a certain extent has been working on apache but has used only firewall systems to secure it.  What better way to have it work with mod_security as an added defense.

By default especially on modern CentOS systems 5.  mod_security is not included in the repos, you will have to enable the EPEL (Extra Packages for Enterprise Linux) to install mod_security.

Steps:

1.  Install the EPEL repos base
 # rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/i386/epel-release-5-4.noarch.rpm
  
2. Install the package
# yum install mod_security


3. Open /etc/httpd/modsecurity.d/modsecurity_crs_10_config.conf file, enter:
# vi /etc/httpd/modsecurity.d/modsecurity_crs_10_config.conf

4. Make sure SecRuleEngine set to "On" to protect webserver for the attacks:

SecRuleEngine On

5. Turn on other required options and policies as per your requirements. Finally, restart httpd:
# service httpd restart

6. Make sure everything is working:
# tail -f /var/log/httpd/error_log

[Thu Mar 31 03:27:07 2011] [notice] Digest: done
[Thu Mar 31 03:27:08 2011] [notice] mod_python: Creating 4 session mutexes based on 256 max processes and 0 max threads.
[Thu Mar 31 03:27:08 2011] [notice] Apache/2.2.3 (CentOS) configured -- resuming normal operations
[Thu Mar 31 04:10:17 2011] [notice] caught SIGTERM, shutting down
[Thu Mar 31 04:10:18 2011] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu Mar 31 04:10:19 2011] [notice] ModSecurity for Apache/2.5.12 (http://www.modsecurity.org/) configured.
[Thu Mar 31 04:10:19 2011] [notice] Digest: generating secret for digest authentication ...
[Thu Mar 31 04:10:19 2011] [notice] Digest: done
[Thu Mar 31 04:10:20 2011] [notice] mod_python: Creating 4 session mutexes based on 256 max processes and 0 max threads.
[Thu Mar 31 04:10:20 2011] [notice] Apache/2.2.3 (CentOS) configured -- resuming normal operations

This tutorial is not limited to CentOS only distributions for Debian systems you can use apt-get to intall mod_security or check the site documentation for procedures.


SUDO - A friend with a Nasty Make-over

One of the strengths of NIX systems is to ability to use tools that allows non-escalated users to perform administrative task within a limited set of controlled domains on a given system (reminiscent with the Windows "RUN-AS" command).  This is where SUDO comes in!  If you are a seasoned engineer sudo can help you alot in giving access to user who needs to issue what commands and do what commands.

However, if done incorrectly you will have one hell of a nightmare looking for someone in the team who performed what on where.   Ideally, you should only give access to certain users who are members of your systems engineering team.  But there are cases when members of the development needs to have escalated access to perform task.

At times a systems engineer who is on vacation will have to rely on his next in line colleague to do things right.  But what if that person is not around to do those things?  The next protocol would be to allow members of the technical to have access to a same level of privileged as yours or any member of your team.

The problem arises when someone misuses this privilege to play with something.  Recently, I had experience the worst possible example of a mismanaged "SUDO" (which I authored) implementation!  It started on a regular working day in which I need to login to production system (Note:  PRODUCTION SYSTEM), I issued the command $sudo su - (to drop me to root, because we virtually erased from memory root password!)  and voila!  I go this instead:

.... Your are not in the sudo list this will be reported ....

Not only one system but on two (2) different servers!

What are the lessons learned here

1.  Implement a sudo list that has levels of escalations.  It will help you identify the culprit immediately.

2.  Identify known vulnerabilities of key programs such us vi etc.  If you know it by now a user who can issue escalated vi privileges can easily drop him to root!

3.  Always be prepared to suspend all users with sudo capability!  This will prompt your users that you are serious in implementing control to which users will have escalated privileges to what program and to what extent.

SUDO is good.  Only if it serves you right and with the right mind set.  But if you think its a solution to help you solve fire at will hungry tech guys then you are mistaken. Better have a PASSWORD VAULT!  I think you know how the rest of the story from here on.

What I learned from DDoS attacks recently

It's a real pain to have your system constantly under increased levels of attacks from all points of entry.  But that is the challenge!  To be able to address issues once their ugly head pops out.

In my recent dealings as a systems engineer to an e-commerce site.  I will have to re-learn everything their is to know about the limits of what you could do to the domain you have control and to those that you don't have.

In this post I am going to give you my insights on how things can go terribly wrong and the things that we failed to realise on the soonest possible time.

I.  Patching Session

Its common knowledge to all systems admins the true worth and importance of patch management.  Patches are released to address known vulnerability issues to a software package installed on the system, it could be a port exploit or a known bug that allows an attacker to take control of a feature or functionality that a package is expected to perform.  In my seat, patches where never released for the reason of lack in understanding and a clean management handle for admins to perform their job.

If a patch is known to break the application sitting on the system.  Then the patch is not worth it!  This is sure guarantee for the troubles that lies ahead.

II.  Ensuing Budget to cover the Holes

As part of the core management team.  My goal is to have my department levy the necessary budgetary requirements that will en-corporate a great deal of resource focusing on securing those information technology investments.  Hardware Architecture and Platform design goes hand in hand and its best that you do it right early on than regret it afterwards.  In my recent memory,  this is my biggest disappointment.   If the ratio of income generated online is equivalent to a magnitude of ratio to all offline income pouring in to the company basket.  Then you should consider acquiring the best known DDoS provider around to help you combat unwanted attacks where it leaves your entire technical team entirely hands tied at their ends.

III.  Human Divestment (Incident Response Teams)

A core team of engineers composed of your systems/network engineering team and members of your development core team, should be tapped to organise your quick action force readily in dispose whenever the problem arises.  Though I will caution even with the most experienced systems engineers, they need to undergo training to counter new treats coming from the net.  There are areas in data and information security which helps organise the chaotic deluge of skill-sets and help shape the team to perform duties that require the following:  1) Risk management, 2) Threat assessment and 3) Exploit forensics.    Though I must warn that It takes awhile to fine tune this team but worth every penny you invest on.

IV.  The Ethical Hack

Ethical hacking maybe a name that spawned from the nasty reality of hackers doing more damage than good.  For me the idea was simply to exploit/identify  vulnerabilities and be able to fix them before others can.  You could do so much with tools that aides in performing such activities.  However, it is likely that part 1 of this document may be the only thing missing in your whole security plan.  In my experience working with enterprise systems patch management has played a considerable role in a systems admins life, thus aiding in the process of hardening your box.

In my opinion, the whole TCP/IP stack which holds up the world wide web is the problem!  The standard on which the OSI layers had been built was several decades ago and IPV4 is likewise a problem.  Countless number of papers have suggested otherwise on a need to research on the next logical predecessors of TCP/IP.  This means a whole new generation of routers and switches to handle how data,voice,image traffic will move.  I believe that "Nicola Tesla" envisioned such with the use of wireless communication without the need to use cables.  However, this would prompt a new generation of exploiters to counter check if this new standard will be the defining technology.

For the time being we are limited to the tools that our current technology can accommodate.  The sad thing is that any ordinary guy with some technical knowledge can do so much damage because of the available tools on the net.  Therefore hacking to your system (ethical hacking) is a must in order for you to see if those security placements will hold.   Give it your best shot! 

To be continued ...

Hacking into Linux packages and fixing problems when the hammer slams!

In the years I have been working on NIX systems (Unix, Linux) there are a couple of instances when you need to have a purely "nirvanic" sense to fix problems when things go wrong.

A couple of days ago I had a ghost revisiting my team.  This has something to do with a package that has been installed using rpm.  The problem started when we upgraded the box memory and viola!  The magic happened when we tried to fire up apache!

It failed! Why? Because the modules needed by apache was missing.  We have very little time to re-act and fix the issue.  We cannot wait another 30 or so minutes to get things done.  Obviously, the hardware engineers done a wonderful job of messing something or It could be just one of those days when things just got bad and their is not way of explaining what is happening.

I surveyed the situation and immediately it came to me to HACK!  It's what we are best in doing so and a simple understanding on how things work would mean saving your company millions of precious income and/or saving your job!

Ever wonder why you have those, test and development environment servers  almost identical (production servers) if not  a clone?  Aside from the fact the code should behave as expected once tested and deployed to those environments? 

The reality is simple!  Be ready to cannibalise those  boxes when the time calls for it.  In my case, we really did not cannibalise anything here or at-least to the hardware level.  What we did is simply to fix the issue with apache.   If you are a seasoned systems engineer, you may have already figured out what I am talking here.