Tuesday, December 28, 2010

Hacking into Linux packages and fixing problems when the hammer slams!

In the years I have been working on NIX systems (Unix, Linux) there are a couple of instances when you need to have a purely "nirvanic" sense to fix problems when things go wrong.

A couple of days ago I had a ghost revisiting my team.  This has something to do with a package that has been installed using rpm.  The problem started when we upgraded the box memory and viola!  The magic happened when we tried to fire up apache!

It failed! Why? Because the modules needed by apache was missing.  We have very little time to re-act and fix the issue.  We cannot wait another 30 or so minutes to get things done.  Obviously, the hardware engineers done a wonderful job of messing something or It could be just one of those days when things just got bad and their is not way of explaining what is happening.

I surveyed the situation and immediately it came to me to HACK!  It's what we are best in doing so and a simple understanding on how things work would mean saving your company millions of precious income and/or saving your job!

Ever wonder why you have those, test and development environment servers  almost identical (production servers) if not  a clone?  Aside from the fact the code should behave as expected once tested and deployed to those environments? 

The reality is simple!  Be ready to cannibalise those  boxes when the time calls for it.  In my case, we really did not cannibalise anything here or at-least to the hardware level.  What we did is simply to fix the issue with apache.   If you are a seasoned systems engineer, you may have already figured out what I am talking here.