Anyone know what's up with Stratics?

skylark · 9th May 2008, 07:30 AM

Quote:

Originally Posted by Nok

Hi Shannara,

You mean backups aren't capable of failing too? How do you know Stratics' backup didn't fail?

Do you have detailed knowledge of the backup systems that Stratics uses for their network, what specific contingencies they have in place, and how they are or are not configured? Are the Stratics servers co-located? Are the production servers and backups all in the same datacenter? Or are the production servers in one datacenter, and the backup servers in another datacenter... possibly in another city? What is the quantity of data being recovered? Are the backups on fixed disk, optical, or tape? Is the recovery being done in-house or contracted? If contracted, what is the turn-around time for whatever amount of data Stratics has? Has the recovery been completed?

Rumor has it, that do to the situation... Stratics might push up a previously planned network upgrade and a possible forum script switch (from UBB.Threads to something I'm not at liberty to say) to now. So as to take advantage of an unscheduled & extended downtime to make the changes now. If THAT is the case... has the recovered data (if it's finished) been returned to Stratics (if it's not being done on-site)? Are the sites going to use the existing infrastructure or be converted to a new one? If they are going to use a new forum, has the forum data been converted or is in the process of being converted? Has the new forums been installed, setup, and configured to resemble the old forums for user familiarity? Is the newly converted forum data in place? Has it been tested yet to reduce potential issues for when it goes live?

I am sincerely looking forward to your answers, I believe your insight will be newsworthy.

I will preface this reply to a post not directed at me, by saying I personally don't question the complexity of data recovery, repairing corrupt databases, nor the fact that backups can and do fail, and archived media, tapes in particular, are notorious for problems. Further, I certainly don't have any knowledge of Stratics network infrastructure or disaster recovery protocols. However, for me, it somewhat strains the imagination that there is a catastrophic failure of the live data systems at the same time as a loss of the off site backups. Perhaps that is not what Orvago meant to convey, and the loss of the offsite data was something that simply came to light when it very badly needed to be retrieved, but when I read his post, it came across as a simultaneous event. It further strains the imagination, that with such a complex nightmare recovery process underway, they would also be undergoing a simultaneous massive data conversion project, and platform migration as well, as you suggest. But, I guess anything is possible.

As a systems admin, whatever the situation, I have full sympathy for the people dealing with the mess (happy I am not one of them), but at the same time, a whole lot of questions as well as speculations start popping up in my head. I'll try to keep them to myself, though since speculations is all that they are. :-P

-Skylark

Orvago · 9th May 2008, 07:59 AM

(Was quoting another poster in which their posts were removed so I felt it only civil to remove the quotes from here too)

Sorry, its not simple as just restoring a backup. Might be that easy if you had the backups, but as I said in another post, the post you quoted, our backups were lost. We're not recovering from backups.. our tech teams are recovering lost data, piece by piece.

I won't even begin to pretend that I know half of what our tech teams have to go through to get our data back, all I know is that it is not as simple as uploading a backup then inflating it into a new forum.

kennykilleduo · 9th May 2008, 08:12 AM

Thanks for all the updates , now get back to work , going threw withdrawl , eyes are bleeding , its the blackrock I tell yea ( ohh wait , you cursed yourselves, lol) .................

Start over again , have everyone resign up , UO has changed over the years , wouldnt that be easier??

**DiP** · 9th May 2008, 08:27 AM

But couldn't the forums be put back up sans old posts?

Kayhynn · 9th May 2008, 08:41 AM

Sorry it took so long for me to get over here folks....

The forums can't go up sans posts unless we want to loose all user information as well.

Right now, the forums have only been down a week. We're in the process of recovering all the data to reinstall it.

As Orvago said, it's not as easy at it sounds and there are people working round the clock with more technical expertise than us to fix what got broke.

Kiminality · 9th May 2008, 08:56 AM

I gather I've missed more than half of the attacks, and snipes at Stratics and Stratics Staff. But with what I have read (the vanilla version, if you will), I'm getting pretty tired of being called an idiot, based on speculation or poor interpretation, for (and I quote from the announcement) "events beyond our immediate control".

On a related note, I'd personally like to thank Adam, and the other staff, for trying to keep it nice. If not for them, I'd probably be a little more than 'pretty tired' of it.

Orvago · 9th May 2008, 09:10 AM

Quote:

Originally Posted by Kiminality

But with what I have read (the vanilla version, if you will), I'm getting pretty tired of being called an idiot, based on speculation or poor interpretation, for (and I quote from the announcement) "events beyond our immediate control".

Who called you an idiot!? Shame on them!

It is true, it was all out of our immediate control. I do promise you all though, we will be able to reveal all the details once we get the forums back up. Until then, we have to (for other reasons) just say "out of our immediate control"

I hate saying it but it is very true.

silent · 9th May 2008, 09:15 AM

Well having worked in IT for 20 years now I feel your pain and wish you luck with the recovery. I hope at the end of this ordeal some important lessons are learned and this can be avoided in the future.

skylark · 9th May 2008, 09:20 AM

Quote:

Originally Posted by silent

I hope at the end of this ordeal some important lessons are learned and this can be avoided in the future.

QFT.

-Skylark

Mathena · 9th May 2008, 09:22 AM

I am not about to act like I know the set-up of Stratics or any of it's datacenters. Stratics had a catastrophic data loss. This is a given. But, this I know from personal experience. If they try and restore the databases from backups.....that are hmm let's say corrupted. It is possible they are dealing with a data recovery process that is more than a typical nightmare.

My company back in '98, had a database loss....it was corrupted by a DBA that was less than spectacular (that's another story). The recovery process began.. and 16 hours later we determined that we had corrupted back-ups back 9 days. Which means we recovered 9 days prior to the real-down time. That 9 days worth of data had to be re-created manually. We wrote programs to scrape data from hard copy reports...basically anything that would help. We allowed no one near that system until we had completed this process. Total downtime: 8 days

So, I know depending on the loss and all pieces of the puzzle. It is possible working around the clock that they are still putting everything back together and verifying everything. Let's cut them some slack.

9th May 2008, 07:59 AM	#132
Orvago "The blaze kitty begins taming you... you seem to accept blaze kitty as master" Join Date: Mar 2008 Shard: Landroval (LotRO) Posts: 12 Gold: 100 Thanks: 9 Thanked 24 Times in 4 Posts	Re: Anyone know what's up with Stratics? (Was quoting another poster in which their posts were removed so I felt it only civil to remove the quotes from here too) Sorry, its not simple as just restoring a backup. Might be that easy if you had the backups, but as I said in another post, the post you quoted, our backups were lost. We're not recovering from backups.. our tech teams are recovering lost data, piece by piece. I won't even begin to pretend that I know half of what our tech teams have to go through to get our data back, all I know is that it is not as simple as uploading a backup then inflating it into a new forum. orvago@stratics.com - uoforumadmin@stratics.com - lotronews@stratics.com Last edited by Orvago; 9th May 2008 at 10:49 AM. Reason: Removed quoted posts.

9th May 2008, 08:12 AM	#133
kennykilleduo Join Date: Sep 2007 Shard: Lake Superior Posts: 45 Gold: 858 Thanks: 6 Thanked 1 Time in 1 Post	Re: Anyone know what's up with Stratics? Thanks for all the updates , now get back to work , going threw withdrawl , eyes are bleeding , its the blackrock I tell yea ( ohh wait , you cursed yourselves, lol) ................. Start over again , have everyone resign up , UO has changed over the years , wouldnt that be easier?? A better UO...Vote here..... Last edited by kennykilleduo; 9th May 2008 at 08:14 AM.

9th May 2008, 08:27 AM	#134
DiP Ronin<br>UOForums Bouncer Join Date: Jan 2006 Shard: Black Water Raiders, Azeroth Posts: 9,261 Gold: 35,201 My Mood: Thanks: 115 Thanked 79 Times in 67 Posts	Re: Anyone know what's up with Stratics? But couldn't the forums be put back up sans old posts? Thanks again Maddux! Adri: women don't discuss men while in the bathroom, we sacrifice small animals and smoke tampons rolls eyes

9th May 2008, 08:41 AM	#135
Kayhynn Goddess. WoW Characters Join Date: May 2008 Shard: Origin Posts: 73 Gold: 7,644 My Mood: Thanks: 4 Thanked 10 Times in 8 Posts	Re: Anyone know what's up with Stratics? Sorry it took so long for me to get over here folks.... The forums can't go up sans posts unless we want to loose all user information as well. Right now, the forums have only been down a week. We're in the process of recovering all the data to reinstall it. As Orvago said, it's not as easy at it sounds and there are people working round the clock with more technical expertise than us to fix what got broke.

9th May 2008, 08:56 AM	#136
Kiminality Join Date: Jun 2007 Shard: Europa Posts: 109 Gold: 51 Thanks: 1 Thanked 1 Time in 1 Post	Re: Anyone know what's up with Stratics? I gather I've missed more than half of the attacks, and snipes at Stratics and Stratics Staff. But with what I have read (the vanilla version, if you will), I'm getting pretty tired of being called an idiot, based on speculation or poor interpretation, for (and I quote from the announcement) "events beyond our immediate control". On a related note, I'd personally like to thank Adam, and the other staff, for trying to keep it nice. If not for them, I'd probably be a little more than 'pretty tired' of it. Are you connected to you?

9th May 2008, 09:15 AM	#138
silent Join Date: May 2007 Shard: Chessy Posts: 56 Gold: 52 Thanks: 3 Thanked 10 Times in 6 Posts	Re: Anyone know what's up with Stratics? Well having worked in IT for 20 years now I feel your pain and wish you luck with the recovery. I hope at the end of this ordeal some important lessons are learned and this can be avoided in the future.

9th May 2008, 09:22 AM	#140
Mathena Join Date: Feb 2008 Shard: Catskills Posts: 259 Gold: 652 My Mood: Thanks: 41 Thanked 35 Times in 29 Posts	Re: Anyone know what's up with Stratics? I am not about to act like I know the set-up of Stratics or any of it's datacenters. Stratics had a catastrophic data loss. This is a given. But, this I know from personal experience. If they try and restore the databases from backups.....that are hmm let's say corrupted. It is possible they are dealing with a data recovery process that is more than a typical nightmare. My company back in '98, had a database loss....it was corrupted by a DBA that was less than spectacular (that's another story). The recovery process began.. and 16 hours later we determined that we had corrupted back-ups back 9 days. Which means we recovered 9 days prior to the real-down time. That 9 days worth of data had to be re-created manually. We wrote programs to scrape data from hard copy reports...basically anything that would help. We allowed no one near that system until we had completed this process. Total downtime: 8 days So, I know depending on the loss and all pieces of the puzzle. It is possible working around the clock that they are still putting everything back together and verifying everything. Let's cut them some slack.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Possible Stratics Advertisement	Landicine	The Divided Lands - OOC	5	4th March 2008 05:18 PM
stratics	woody	Ultima Online General Discussion	5	30th October 2006 01:59 PM
UO Stratics down again???	Dealin'Burgers69	Ultima Online General Discussion	4	27th April 2006 11:27 AM