Not sure about these hearts?

Yep i like David Bowie, I like Pink Floyd. I do not like little pink hearts. How do i turn these off?

Sunday. Yet more tales of IT woe.

Not being one to take things easy at the weekend – and taking the opportunity of having the network closed down, it is an ideal day to catch on up Windows Updates and playing about with all those little ‘tweaks’ you want to try during the week but never get around to it.

Well, i cant say i had learnt my lesson from Friday or indeed Saturday. I don’t think i wrote about this yet  but once i had everything up and running again yesterday morning i went through and applied about 40 updates to the server. I know how important it is to keep the servers patched – but i also know what can happen without properly testing updates before you apply them. LOL.

Testing – i don’t know if it is because we don’t have the time. Or if we don’t have the equipment. Likely as not it is because we cant be bothered – but the only updates that are tested are the ones applied to our own systems first. With no testing.

So perhaps again i am demonstrating why these problems are all self inflicted, and with a little time and effort you can have a network which doesn’t suffer from such problems. However i would also at this point stress that the at no time has our customer(s) experienced any downtime due to my seemingly uninterested approach to system administration. I am always conscious that a fellow IT Pro may stumble upon and read my blog – and laugh to themselves that if i only had done X or Y then i could have avoided all of Z – perhaps this is true. No it’s definitely true. But we have all been there and at least I’m being honest about it.

Anyway i digress.

Update number 40 of 40. A SQL Server 2000 SP4 (KB948110).

Which it seems has some issues. It seems to have issues with services restarting and if working remotely will completely close the server off to remote connections for roughly one hour. Among the services to be stopped are the exchange Information Store and Seemingly the World Wide Web Publishing service. Arrgh what am i to do now? All was working well, i just apply some updates and crash, there goes my connection.

Well for any other IT Pro it may be game over until Monday. Angry users phoning up complaining of no email or network – and the IT Pro forced to concede a reboot is required (have you tried turning it off and on again?)

But no. We you see use HP Servers. HP Servers with ILO. What does ILO stand for – Integrated Lights Out. And it couldn’t be more accurate. As long as the server has power to it, and i mean it could be switched off sitting 3 miles under the earth – inside a small room with a sign on the door saying ‘beware of the leopard’ we can get to it. (Of course it needs an internet connection as well but let’s not split hairs)

So to recap, as long as the server has mains power, and an active internet connection we can connect to it – we can then use wonderful tools like remote power management, and even the remote console, which runs through Java and telnet so we can get complete system access – from post. In fact i have used this tool a few times to change BIOS settings remotely – now that IS NOT for the faint of heart.

So i lost my connection mid update – which is never good because you end up in some kind limbo where you know the server is carrying on without a care in the world, your session is still running as well, your just not there, and cant get back in.

Connecting via ILO i log in to find many services stopped. A quick check on taskmgr and i can see the update processes are no longer running, and decided to reboot. The server comes up fine and Lights Out has saved the day.

This morning was a different story. I decide another server is in need of my gentle touch and log in and commence the updates. I glance over after 45 minutes to see my connection, disconnected and the last update progress indicator was SQL Server 2000 SP4…. Not again… oh well ill just use….

Shivers run down my spine as i have a sneaking suspicion someone unpatched the lights out interface some time ago and we never got around to plugging it back in. Well you wouldn’t would you?

So, time for some ‘blue sky thinking’ lets see.

I attempt a lights out connection, which fails. Hmm ok maybe the cable isn’t unplugged maybe the router isn’t doing the port forward correctly – ill ping the interface from the router. Nope, this routers firmware is not on the revision which includes the ping tool. Great – do i fancy a remote firmware upgrade? what’s the worse that could happen? As if i haven’t suffered enough.

Now this feature is nice, i really like this about the Draytek Vigors, you can probably do this with other routers but i haven’t ever seen it, and cant shout about it.

An ftp session into the router, upload the new firmware image, and reboot. Simple as that. And it works – keeping in place the existing settings as well. Quick tip on that if you are going to do this – use the .all version of the firmware not the .rst – the rst file WILL erase your settings.

Ok logged back in and trying the ping – nope request timed out.

How about a vpn into the router – and try to bypass ISA on the SBS box (really clutching at straws now)

Reconfigure the router to allow the vpn to terminate there, log in – ping/rdp/ no luck. Even tried the ISA Server MMC which is installed on my trusty laptop – i know this will fail even before i try but I’m desperate now…

Right, that’s it. Better phone the boss. Straight to voicemail and i don’t bother to leave one i don’t know what it is but voicemail greetings irritate me. Actually you may have noticed a lot of things irritate me.

Ok well ill put the router back to rights and have a think about it and come back to it later. I really have no idea how I’m going to get out of this but I’m sure ill think of something.

Router reconfigured, i hit the VPN dial button.. what’s this… verifying username and password? – connected?   could i have not configured the router properly. Most likely. Ipconfig /all – no. I’m in. I’m back IN. Amazing.

RDP back on to the server , where my original session is sitting there waiting – to finish updating you must restart.

Oh absolutely brilliant. Glad i didn’t leave that humbling voicemail after all.

So, it would appear this update likes to cut you off – seemingly for the fun of it.

Well reading the KB article, it seems to make more sense, a restart of the MSSQL$MSFW could definitely explain why i was cut off.

So, provision needs to be made i think and more checks in place that our lights out connections are actually connected.

Whilst this could be seen as embarrassing for me as an engineer, (i would agree) i think it will also server as a reminder/warning to any who may read this blog. Check your servers. Check them NOW!

We can never assume. We can never rely. We are the IT Admin and we are.. something.

So after all that excitement i got up. It was Sunday after all. I watched Scrubs on the telly – which is all E4 ever seem to show, had a lovely Sausage & Egg sandwich for breakfast and relaxed with Baby Stella before i headed out to football.

I play for a 5aside team locally – we are not the best, but we have a laugh which is all we are interested in. Today we had a full squad of 8 (5 on the field with 3 rolling subs) playing against a team who had just 5, no subs.

It is always fairly even between the two teams – but they seem to have the edge fitness wise, as our 3 man advantage didn’t seem to be helping much. With a few minutes to go we were 9-7 up, but we failed to take advantage and eventually drew 9-9. We were all gutted, not least of all poor Dan, the goalkeeper. I still stride on looking for my first goal, but i did chip in with an assist today which was nice, calmly riding two harsh challenges to pass into the path of the gaffer who netted from the edge of the D.

I could write football reviews – i think id do a good job. I can do myself a lot of favours as well because if the truth be told i stumbled over the ball, just about kept my balance before the ball hit my ankle and flew towards his feet – but i like my first version better. Oh well – next week is another chance, another attempt to improve my fitness and another chance to get my goal.

My attention now turns to this evening’s FIFA contest 0 i missed last night as me and Kat were at an engagement party but the next duel has been scheduled for 10pm this evening, the mighty RedParkRyan’s Man UTD team taking on the fearsome BennySlayer’s Arsenal side. With some tactical changes afoot, i feel confident i can build on my 5-1 victory.

it’s shocking how seriously we take this game, i wish i could record the headset chat as Ryan’s attitude after i score would do seriously well on YouTube i think.

Anyway that really is enough blogging for one day – i am going to see if i can get rid of these pink hearts.