It is a fact of life that at some point, things will go wrong. When they do go wrong, you probably wish for two things. Firstly, that any plan you may have to recover from the outage actually works when you need it (and if you test your disaster recovery plan often, you should be pretty confident on this). If you don't have a disaster recovery or business continuity plan or have never tested your plan, this is where you'll start sweating and possibly start considering moving to a foreign country. Your second wish will probably be that you can fix the problem so fast that just about no-one will notice. Unfortunately, if you don't recover quick enough, people will notice, and so beyond just loosing revenue and productivity during an outage, your reputation can also suffer. The stories in this section are about real life computer related disasters. Consider them case studies and use these bad experiences to improve your planning and recovery. If you know of an outage or a problem with a website or some other technology related system, please send the details to: downtime@itdr.info .
|
|
Government
|
|
Friday, 26 January 2007 |
|
EnergyAustralia experienced a SAN hardware problem in December 2006 that required restoring of files from backup. When trying to do the restore though, they found that about a quarter of the total files (about 1.2 million files) were missing. |
|
Read more...
|
|
|
Corporate
|
|
Saturday, 13 January 2007 |
|
According to computing.co.uk , a computer glitch meant thousands of UK customers could not use their VISA cards on New Years Eve. Other articles indicated that customers were also unable to draw money from their VISA credit and debit cards, unless they were using their own banks ATM machines. |
|
Read more...
|
|
|
Web
|
|
Saturday, 30 December 2006 |
|
It appears that the Apple iTunes store couldn't handle the load of users trying to download music and video on 25 - 26 December 2006 and slowed to a practically unusable level for many users.
According to reports, songs that normally took 1 - 2 minutes to download were taking up to 20 minutes, while some users were getting error messages indicating that their request could not be completed and some could not even login to the store at all.
|
|
Read more...
|
|
|
Web
|
|
Thursday, 28 December 2006 |
|
According to a blog post at Xbox 360 Fanboy , the Microsoft Xbox live service was down for a few hours on 27 December, 2006. At this stage, there are no details on what went wrong, but it seems that anyone wanting to play online games for those few hours was out of luck. According to the Major Nelson blog (written by a member of the Microsoft Xbox team) anyone trying to get into a game (matchmaking) were unable to do so and he had this confirmed by the Xbox Live ops team. |
|
|
Government
|
|
Sunday, 17 December 2006 |
|
The US Military Health System (MHS) has lost close to 5,000 records relating to patient encounters, due to software and hardware problems. An incorrect database setting resulted in 3,586 records being lost and a hardware issue caused a further 1,400 records to be lost. |
|
Read more...
|
|
|
Corporate
|
|
Monday, 11 December 2006 |
|
The Royal Bank of Canada (RBC) had a system outage in late May / early June 2004, after performing a software upgrade. As a result, millions of customers were inconvenienced when their transactions did not reflect in their accounts for up to a week. Now the Quebec Superior Court has ruled that a class action lawsuit relating to the outage may proceed. |
|
Read more...
|
|
|
Web
|
|
Sunday, 10 December 2006 |
The Open Directory Project (ODP), which is the 'largest, most comprehensive human-edited directory of the Web' is broken and has been for the past 7 weeks. Apparently they experienced a 'technical problem' around 20 October 2006 and have not been able to recover fully from it yet (9 December 2006).
|
|
Read more...
|
|
|
Web
|
|
Monday, 23 October 2006 |
|
On 21st October 2006, the eBay Live Auction service, which enables users to bid on items being auctioned on the floor of the some of the worlds leading auction houses, crashed for a number of hours. As a result, numerous real life auctions lost out on the potentially highest bidders who were participating via the eBay service. Of course, those losses are not eBay's problem as the eBay terms and conditions state that eBay offers no warranty or guarantee regarding the service. |
|
Read more...
|
|
|
Web
|
|
Tuesday, 08 August 2006 |
|
On 27 June 2006 the team running CouchSufing.com were doing some database administration when they noticed a filesystem error. To cut a long story short, they got to the point where they needed to recover from backup only to find that their backups were not recoverable. |
|
Read more...
|
|
|
Government
|
|
Monday, 07 August 2006 |
|
On Sunday, 30 July 2006, a problem at a regional data centre caused the loss of the patient administration system for 80 National Health Service (NHS) trusts in the UK. |
|
Read more...
|
|
|
DR in Real Life 
