Some Facebook groups that you may find helpful:
- js refugees (invitation only)
From Andrew’s Blog: How to recover most or all of your journalspace posts/images using Google cache
Hi there, Slashdotters. My blog has some more information which you may find interesting.
Journalspace is no more.
DriveSavers called today to inform me that the data was unrecoverable.
Here is what happened: the server which held the journalspace data had two large drives in a RAID configuration. As data is written (such as saving an item to the database), it’s automatically copied to both drives, as a backup mechanism.
The value of such a setup is that if one drive fails, the server keeps running, using the remaining drive. Since the remaining drive has a copy of the data on the other drive, the data is intact. The administrator simply replaces the drive that’s gone bad, and the server is back to operating with two redundant drives.
But that’s not what happened here. There was no hardware failure. Both drives are operating fine; DriveSavers had no problem in making images of the drives. The data was simply gone. Overwritten.
The data server had only one purpose: maintaining the journalspace database. There were no other web sites or processes running on the server, and it would be impossible for a software bug in journalspace to overwrite the drives, sector by sector.
The list of potential causes for this disaster is a short one. It includes a catastrophic failure by the operating system (OS X Server, in case you’re interested), or a deliberate effort. A disgruntled member of the Lagomorphics team sabotaged some key servers several months ago after he was caught stealing from the company; as awful as the thought is, we can’t rule out the possibility of additional sabotage.
But, clearly, we failed to take the steps to prevent this from happening. And for that we are very sorry.
So, after nearly six years, journalspace is no more.
If you haven’t yet, visit Dorrie’s Fun Forum; it’s operated by a long-time journalspace member. If you’re continuing your blog elsewhere, you can post the URL there so people can keep up with you.
We’re considering releasing the journalspace source code to the open source community. We may also sell the journalspace domain and trademarks. Follow us on twitter at twitter.com/jsupgrades for news.
As a somewhat newly-minted overnight admin for a firm that deals almost exclusivley in important information, this story is a reminder about how important it is to keep up to date backups (something that I don’t do on my personal machines as much as I should)
I don’t imagine for a second that my customers would be happy to find out that I was not keeping a non-volatile backup of my important data like these people were, but I don’t know if their customers were even paying for the service, so this may be a non-issue in terms of cash. I didn’t look any deeper to find out.
As for my personal sites, I keep backups off site and on other mediums, but not as often as one might want or expect. In Fact, I’m doing one now just in case.
Some Advice for IT TypesPublished by NiteMayr on June 24, 2008
If this is the case, why are so many IT jobs filled with people who have no idea what they are doing? I spoke to my share of IT reps from firms all over the Fortune 1000 and Fortune 50 that had no clue what they were doing, nor did they have any idea where they were going with their mandates. Often they had no plan or action plan.
One example really sticks out for me; a hardware changeover plan that had no “buffer” the IT rep wanted to replace an important firewall with another one. He felt assured that he could just replace the current device with a new and wholly different one if the new devide was configured correctly.
This was a bad plan for two reasons:
1) There was no fallback beyond dropping the old hardware in place.
2) The router was the MAIN ingress to their websites and mail systems. There were no external fallbacks or alternate sites for users to visit during the downtime. If the transition went BAD (new hardware fails and old device breaks during transition) there was no fallback.
I know, you’re thinking: Kevin, what would you have done?
I would have published a new set of DNS records with a TTL of about 15 minutes. I would publish them a week before I made the transition and made sure my DNS server was not inside the new router. Once in place you would have 15 minutes of downtime while you performed the transiton to a new host for your website if something went wrong during the switch. That’s fairly easy to deal with.
I like the idea of planning for downtime like that; you could even change the TTL on the DNS records back to 24 hours when you are done.
Here are some tips for outage planning
If it is an internet enabled service that users need access to, publish DNS records that point to a “Server is down” page on the net (for web services) when the primary record(s) is/are down.
Keep offsite hard copies (by hard copies I mean stored on Hard disk or Tape)
Keep enough cash in the IT budget to buy server time on multiple hosts should short-term downtime become extended overtime.
Any server that is important enough to serve all your needs should have a clone on hand with all the same data, backed up every 6 to 12 hours (or less) so that if your primary server(s) go down a clone can go online in seconds.
After all, you are the heart of the business when you are in IT, right?