Category: Work

On Being Busy

Published by NiteMayr on May 17, 2011

It’s been too long; I’ve been so busy. I guess I’ll recap what I’ve done in these two years plus.

Fixed over 100 incorrectly configured linux boxes so that they would actually send their admin the output of the logwatch command.
Reconfigured the same to use ClamAV correctly and with consistent settings instead of the hodge-podge that they were.
Built a custom monitor for a series of servers that allowed non-techs to determine if the servers in question were up or down. I’d come back to this
Built a scripted installer for a 21 server farm, taking a 10-20 minute process down to a single command line. I’d come back to this too.
Fixed the log backup system that had been in place for months. It’s still there now, but it needs to change.
Got really into replacing complex manual functions with Bash scripts.
Built the data import system for a whole client. SUPER complex and modular, didn’t use most of the code anywhere else save for the functions method.
Got to know cron really well.
Got to know ssh -t “command” really well
Got lost in the weeds of random apps for random functions, the environment was becoming to large and entrenched to be managed remotely via a central console.
Built an extensible console for managing the environment in part.
Build Cache flushing tools
Learned how to compile bash apps at the command line using shc http://www.datsi.fi.upm.es/~frosal/sources/shc.html
This gave birth to a number of cool tools, remote fail-over tools that interacted with Cisco devices for example.
The Web Console built earlier evolved and got better and better.
Built automated localized monitors that could restart hung applications before remote monitors could catch the outage.
Built automated localized monitors that could restart hung applications and NOT cause two systems to restart simultaneously.
Installed DD-WRT a few times, lots of fun.
Gave up some weekends
Gave up some sleep
Gave up Family Time
Gave up Long Weekends
Built a custom log handler for Apache logs, produced delightful daily csv from an environment, imported this into MySQL and created views to deal with that.
Tried to hit the gym
Got too busy for the gym.
Trained up a replacement.
Left things running okay.

Bash Script to read in a big text file and do something with it

Published by NiteMayr on July 9, 2010

At the office I often end up producing little scripts to do this and that and today I had to deal with a large file that was causing a custom app to bork. In short it needed to be read to the app in chunks; usually I have to do more prosaic stuff, but this is a neat little foundational app to get things done.

I needed this today to solve a very basic problem, maybe you can use it too

#!/bin/bash

declare -i RESET=0
declare -i TOTAL=0
declare -i LINES=$(cat $1 | wc -l)
echo -n > somefile

while read LINE ; do
	RESET=$RESET+1	

	if [ $RESET -lt 1000 ]; then
		echo $LINE >> somefile
	else
	   RESET=0
	   somecommand -file somefile  | mail -s "Output from somecommand" 2someguy@somewhere.koo
	   echo -n > somefile
	   TOTAL=$TOTAL+1000;
	fi

	declare -i REMAINS=$LINES-$TOTAL
	if [ $REMAINS -lt 0 ]; then
	   tail -$REMAINS $1 > somefile
	   somecommand -file somefile  | mail -s "Final Run of somecomand" someguy@somewhere.kooo
	fi

done < $1

For Rememberance Sake and SSH in the Future

Published by NiteMayr on November 19, 2008

Credit to: http://pkeck.myweb.uga.edu/ssh/

Getting Started

First, install OpenSSH on two UNIX machines, hurly and burly. This works best using DSA keys and SSH2 by default as far as I can tell. All the other HOWTOs I’ve seen seem to deal with RSA keys and SSH1, and the instructions not surprisingly fail to work with SSH2.
On each machine type ssh somemachine.example.com and make a connection with your regular password. This will create a .ssh dir in your home directory with the proper perms.
On your primary machine where you want your secret keys to live (let’s say hurly), type
ssh-keygen -t dsa

This will prompt you for a secret passphrase. If this is your primary identity key, make sure to use a good passphrase. If this works right you will get two files called id_dsa and id_dsa.pub in your .ssh dir. Note: it is possible to just press the enter key when prompted for a passphrase, which will make a key with no passphrase. This is a Bad Idea ™ for an identity key, so don’t do it! See below for uses of keys without passphrases.
scp ~/.ssh/id_dsa.pub burly:.ssh/authorized_keys2

Copy the id_dsa.pub file to the other host’s .ssh dir with the name authorized_keys2.
Now burly is ready to accept your ssh key. How to tell it which keys to use? The ssh-add command will do it. For a test, type
ssh-agent sh -c 'ssh-add < /dev/null && bash'

This will start the ssh-agent, add your default identity(prompting you for your passphrase), and spawn a bash shell. From this new shell you should be able to:
ssh burly

Some Advice for IT Types

Published by NiteMayr on June 24, 2008

“IT is at the heart of business these days and there are real opportunities now to have a career in IT which will ultimately lead to a position on the board.”

If this is the case, why are so many IT jobs filled with people who have no idea what they are doing? I spoke to my share of IT reps from firms all over the Fortune 1000 and Fortune 50 that had no clue what they were doing, nor did they have any idea where they were going with their mandates. Often they had no plan or action plan.

One example really sticks out for me; a hardware changeover plan that had no “buffer” the IT rep wanted to replace an important firewall with another one. He felt assured that he could just replace the current device with a new and wholly different one if the new devide was configured correctly.

This was a bad plan for two reasons:

1) There was no fallback beyond dropping the old hardware in place.

2) The router was the MAIN ingress to their websites and mail systems. There were no external fallbacks or alternate sites for users to visit during the downtime. If the transition went BAD (new hardware fails and old device breaks during transition) there was no fallback.

I know, you’re thinking: Kevin, what would you have done?
I would have published a new set of DNS records with a TTL of about 15 minutes. I would publish them a week before I made the transition and made sure my DNS server was not inside the new router. Once in place you would have 15 minutes of downtime while you performed the transiton to a new host for your website if something went wrong during the switch. That’s fairly easy to deal with.

I like the idea of planning for downtime like that; you could even change the TTL on the DNS records back to 24 hours when you are done.

Here are some tips for outage planning

Have a fallback plan for total failure:
If it is an internet enabled service that users need access to, publish DNS records that point to a “Server is down” page on the net (for web services) when the primary record(s) is/are down.

Keep offsite hard copies (by hard copies I mean stored on Hard disk or Tape)

Keep enough cash in the IT budget to buy server time on multiple hosts should short-term downtime become extended overtime.

Any server that is important enough to serve all your needs should have a clone on hand with all the same data, backed up every 6 to 12 hours (or less) so that if your primary server(s) go down a clone can go online in seconds.
Announce the outage in as many ways possible. Email is never enough for big outages. Warn users in cloud writing if you think they will read it.
When the outage is going to take a machine out of service forever, contact any old admins and/or users and determine if they have stored anything important on the box. You never know.
Treat every outage as a potential crisis and be ready for complaints regardless of success or shortness of time.
Confirm that all parts and plans are in order before the outage in underway, if at all possible create a schedule and checklist for the outage that creates a series of milestones and ETAs that can be delivered to end users and managers.

After all, you are the heart of the business when you are in IT, right?

Non-IT Grads don't want IT Jobs

Published by NiteMayr on June 24, 2008

Just read this passage and wonder at it:

Non-IT graduates think a job in IT would be “boring,” despite its good career prospects, according to the Career Development Organisation (CDO).

http://www.computerweekly.com/Articles/2008/06/24/231173/it-is-boring-say-graduates.htm

Read it again, I’ll wait.

Okay, got it? It opens with “Non-IT Graduates” as if to say someone who went through school to get their MBA or Masters in Psychology would be interested or even qualified to fill an IT position. I think the article is grasping for the why not IT in the first place kind of feeling, but instead comes to a screeching halt right up front with that first line. I read it as “people who were never interested in IT think that IT jobs are boring” and you know what, they should not get into IT if they feel that way.

I’m fairly certain that there are a number of people in IT these days who got into it for the money; and through sheer personality have excelled. Good for them. It’s kept down a few really smart people in the ranks because they don’t have the social skills to impress the uppers, but maybe those types will be weeded out and the more focused geeks will rise to prominence.

Time will tell I guess.

So Sweaty…

Published by NiteMayr on June 5, 2008

Helped move PCs Yo

Left iPod in my pocket

Now iPod is broke

How nice, a haiku to my broken iPod.

It’s hot and sticky outside, I am trying to resist the temptation to complain about it, even though I had to work outside in it and get so sweaty that I think I may have developed trench foot from sock sweat. There are Germans from World War I who had drier socks than I have now.

The plant was worse; humid, sticky and smelly to boot.

We moved a ton of computers then cling-wrapped them using a big industrial roll of cling wrap that we had to carry around and so on by hand.

I better be losing weight doing all this stuff, I can certainly feel the burn in my muscles at the end of the day and I haven’t given up on salads instead of fries at lunch and so on. In fact, I’ve been eating fruit in favor of snacks during the day too.

The Stacks of Beer

Published by NiteMayr on May 28, 2008

The Stacks of Beer, originally uploaded by NiteMayr.

Yeah, this is what I see on my way into work every day, the stacks, not the clouds. It’s nice and sunny outside right now, also very green.

I was worried that I would be giving up all the green when I came down here to the City; I’m no nature lover, but I do like the trees and grass the come with nature. London has proven to be very green, excessively so. But like Kubla Khan, I find wisdom in excess and think that the tree lined streets are awesome and I go out of my way to drive down the back streets to stay among said streets.

Smokers and Jokers

Published by NiteMayr on April 23, 2008

Whirlpool suspends 39 workers, says they lied about smoking – Yahoo! News

A Whirlpool Corp. factory in Evansville, Ind., has suspended 39 workers who signed insurance paperwork claiming they don’t use tobacco and then were seen smoking or chewing tobacco on company property. Now, some could be fired for lying, company spokeswoman Debby Castrale said.

Whenever something like this happens I wonder where it’ll all end. I actually applaud the company for their intestinal fortitude, I’d bet that the magic number for regulations on matters like this is 40 though. As in, if 40 people are let go there needs to be prior notice. Interesting, no?

Now I could go on an on about how heavy smokers make for a poor work environment (mostly due to the smell) but alot of people have bad BO or use heavy perfumes. This looks like (on it’s face) a sneaky way to dismiss employees who could well be a drain on the company health plan. That being said, I’m seriously obese and could be looked upon in the same way.

The important question here is where these people given some form of warning or is this a surprise enforcement move?

Lewis Maltby, president of the National Workrights Institute, which
advocates for employee privacy, sees no problem with employers trying
to curb smoking. But he worries that the trend of cracking down on
employees’ unhealthy behavior is extending beyond tobacco use.

“We shouldn’t have to give employers complete control over our
private life so they can save a few dollars on medical care,” he said.

This I agree upon, I have been compelled to sign agreements that affected my private life in the past and have declined to do so, as I could not be effective in my job at the time if I was forced to comply with the spirit and letter of these agreements. So I commiserate with the persons affected, up until they lie on their health insurance forms. In the States, you pay for health care, someone lying on those forms and hurting the premiums of their coworkers is not the way things should be.

This is one of those wedge issues where Universal Health Care removes the impetus on the company to enforce health mandates. Don’t you think?

Excitement and Loss

Published by NiteMayr on April 22, 2008

So I didn’t get the job with the brewery. Not because of any personal failing (save accepting it weeks ago) but because it was no longer available. The good news is that I am heading out to Manitoulin Island next week in order to meet with their IT Manager and visit the site. The exciting part here is that Jen gets to come along and we’re spending two nights out there!

We’ll see how that goes. I took some pictures this weekend, some nudes, some family. The family pictures aren’t so good. I’m hoping to take better pictures this weekend of the confirmation of my newphew.

How to test an IP address against an RBL/DNSBL list

Published by NiteMayr on May 2, 2007

Hey There; this post is a technical post about how to test to see if your IP address is on an RBL list.

Locate an RBL provider or if you know you are on their list determine their zone name: for example the zone names for SPAMHAUS are sbl.spamhaus.org, xbl.spamhaus.org and pbl.spamhaus.org
Once you have those names, you need your IP address. For this example we will use a fake address:
212.122.234.565

(For those of you not in the know, this IP address can’t exist in the IP v4 numbering system)
Now reverse the IP address!
565.234.122.212
Now we need a command prompt; from windows XP, 2000 or Vista go to START -> Run and type cmd and enter. In Linux or another *nix system open a terminal. On Mac OSX, uh, open the terminal.
At the command prompt type nslookup 565.234.122.212.sbl.spamhaus.org and hit ENTER
You should see something like this:

Server: dnsserver.yourdomain.com
Address: xx.xx.xx.xx

*** dnsserver.yourdomain.com can’t find 565.234.122.212.sbl.spamhaus.org: Non-
existent domain
If you DON’T See “Non-Existent domain” and instead see something like:
Non-authoritative answer:
Name: 565.234.122.212.sbl.spamhaus.org
Address: 127.0.0.1

It’s a good bet you are on the RBL (in this case SBL) list.
You can repeat this test for each DNSBL zone (ie nslookup 565.234.122.212.xbl.spamhaus.org)
If all the tests come up as “non-existent domain” you’re clean!