Out of support, budget, capacity. The myth of the Mygyver IT Hero. (Part 1).

If you have worked in IT you’ve run into variations of the following question.
Help, my MD3000i thats 10 years old and out of support/life is out of space and hanging on by a questionable back plane connection! How do I fix this/keep using it for 5 years?

Many IT staff (Particularly in the SMB realm) get excited when they are faced with this challenge. They feel this is part of Why IT exists. They quickly brandish their chewing gum; coat hanger; easy bake oven; rubber chicken; and dive into these problems so they can brag about it and live to see another day. They are vaulted as mullet wearing hero’s with 92 disk RAID 5 or QNAP based HA Clusters that let them have Enterprise like features on 1/10 the budget. They are convinced their goal is to run an IT shop with as little budget as possible, and mask poor communication and architecture skills with a never end series or heroic 28 hour battles.

I come not to praise this hero but to bury him. He is a risk to his business, our profession, and he needs to be stopped as he undermines the credibility of us all. There is doing more with less, and then there is the ridiculous that is our Mullet wearing bandit. Lets examine the cast of characters that leads to these messes.

Mr. “I don’t need support, just more GB!”

Out of support critical hardware is not something that just happens overnight. At some point in purchasing that shiney new VNXe someone who had a 100K budget made a choice between buying the extra 2 years of support or getting more capacity/more ram in the hosts. You’ll recognize this guy as he will often dirrect his entire budget on making one number really high. Expect to find Quad Socket hosts with 32GB of RAM, or possibly an all Flash SAN with a Single Fibre Channel Switch. Everything will be redundant except the one thing he does not understand. Expect a Terabyte of RAM, and a 4 disk RAID 5 in his SQL server.

Mr. Brand Name

This is the IT guy who’s convinced that solid architecture or support agreements are not needed as long as he’s got brand names. He will go out and pick up Solid Brand names (EMC/Cisco/VMware) but pick their Small business offering that have the same support or feature or capacity that he needs. Expect to find Cisco/Linksys RV series Routers and SG switches. VNXe or EMC/Lenovo storage deployed for an Oracle RAC cluster. Do not be shocked when your discover he is running Production Servers on VMware GSX/Workstation/Fusion. He thinks he’s a hero as he’s got all the right “toys” without spending the real money required to get the right ones for the job.

Mr Open Source

In no way view this as an attack on open source. (I’m typing this into WordPress, and this server runs Appache Linux) which in this case is the right tool for the job. This IT guy’s lone goal is to spend nothing on software. If you ask him what storage he’s running he’ll mumble something about OpenSolaris ZFS, with Xen, and SquirrelMail for a 10 man office. Now he will often not actually fully understand the technologies that he is deploying or have the skills to soundly deploy them making things more difficult. He stands out in that he will deploy Servers on a non-LTS Ubuntu Desktop Edition. Generally it will take days or weeks even for a RHCE to make sense of the network. Expect his keyboard to be switched to Dvorak. You can run a test to identify this guy from a rational open source skilled admin.

Mr. Chicken Little

Chicken Little blends in with normal functioning small IT admins except he has one big flaw. He’s afraid of the sky failing. Every time someone mentions moving a simple, logical thing to the cloud (Email, Spam Filtering, Website hosting) he shrieks like a chicken fighting for his life.

The point of this post is partly to rant, and partly to explain something I’ve found to be common sense for a while.
Any project needs to scope what the baseline RPO/RTO/Reliability/Availability as well as capacity and performance baselines before it should be signed off on. What can I get for the change in my pockets is a game to play in a dollar store, not in IT. Saying No, or translating ridiculous budget reductions into reductions of user functionality and not reliability are skills that every good IT pro should have. Sadly virtualization, overcommitment of resources, and the consumerization of IT have made this problem worse. Part 2 of this article will talk about strategies for assuring that budget is tied to further updates and roll outs, and how to overcome the budget cliffs, and problems of scaling infastruture to meet “just in time” and other new trends from the operations side.