Page 1 of 2

Recent downtime and data loss

PostPosted: Sun Jun 12, 2011 8:38 pm
by Fliggerty
We have suffered a very unfortunate loss of data due to some severe hardware failure.  I was absolutely confident that the two layers of backups that we utilize here would be sufficient; unfortunately they were not.  I learned one very important lesson: never trust your host's backups.  Those were apparently the first thing they deleted (or rather, neglected to save.)  In fact, I have entirely disabled them going forward, and will handle it entirely myself with a new master/slave redundant configuration.

So you will notice that about a week's worth of posts are gone.  I apologize profusely, and hope that we can all just pick ourselves up, dust off our knees, and keep on keeping on.

Please let me know if you have any questions.  And certainly please report any problems you see!  It will probably be several days until I can get everything running optimally again.

Thank you all for your patience and support!

--Fligg

Re: Recent downtime and data loss

PostPosted: Sun Jun 12, 2011 8:52 pm
by latendresse76
THOSE BASTARDS!

Glad to see it's back...

Re: Recent downtime and data loss

PostPosted: Sun Jun 12, 2011 9:37 pm
by Wolf
:dancing-banana:  :dancing-banana:  :dancing-banana:  
:dancing-banana: Yay!!! :dancing-banana:                       -    Cant have enough nanars.
 :dancing-banana:  :dancing-banana:  :dancing-banana:

As for the week of posts gone; im sure we'll manage.
And on the bright side:  Its always entertaining to yell at people when they don't do their job right.

Re: Recent downtime and data loss

PostPosted: Sun Jun 12, 2011 10:54 pm
by Leonardo
latendresse76 wrote:THOSE BASTARDS!

Glad to see it's back...

YEAH!!! Yes yeah of those bastards who don't know a damn thing about the service to their customers.  :beathorse:  :hitlol:

YEAH!!! Yes yeah to glad to see it's back... I was almost disappearing from the face of earth if GHF wouldn't be back in the long run, but I'm so HAPPY that GHF survive this disaster.  :dancing-banana: :mrgreen: :dancing-banana: :sheo: :dancing-banana: :mrgreen: :dancing-banana:  :cool:

Re: Recent downtime and data loss

PostPosted: Mon Jun 13, 2011 1:10 am
by £exa
I lost my WIP topic and some postings... well, I will repost it.

But I'm happy to see GHF back. Now let's sacrifice a guar and pray for stability.

Re: Recent downtime and data loss

PostPosted: Mon Jun 13, 2011 1:54 am
by Miserina
Huh, so that's where my last couple posts went.  Odd.

Re: Recent downtime and data loss

PostPosted: Mon Jun 13, 2011 3:01 am
by somesz
Finally! I was scared when I saw GHF inactive. What a relief to see it's online again. ;-) I lost some screenshots from Census Office but that's no problem. I'll reupload them once the site is at full throttle.:)

Re: Recent downtime and data loss

PostPosted: Mon Jun 13, 2011 5:31 am
by Maiqdabomb
I posted a topic but it disappeared. I no longer have a topic. Oh well.

Some stupid son of a b**** needs to get fired for this.

Re: Recent downtime and data loss

PostPosted: Mon Jun 13, 2011 5:39 am
by Melchior Dahrk
I was wondering what was up. Seemed to be fluctuating between being online and offline. I'll have to check and see if I lost anything important, but I doubt it... maybe those masks I posted in my WIP thread...

Oh, and I'll repost my signature too I guess :D

Re: Recent downtime and data loss

PostPosted: Mon Jun 13, 2011 5:44 am
by Leonardo
There is one thing I notice earlier today and that was the file size of uploading has decreased from 10-20 Mb (not exactly sure how much :unsure:) to max 2 Mb in file size and top of that my recent upload of the new version of Wrye Bash, WBSA 292 download thread was gone. I know now why I must re-upload it one more time, but that's not a big deal though. Of course I'll wait until you have time to take a closer look of this.  :smile:

Is the new file size only temporarily or is it permanent?  :chinscratch:

Re: Recent downtime and data loss

PostPosted: Mon Jun 13, 2011 6:00 am
by £exa
The file size reduction took place earlier and is no consequence of the rebuild.

Re: Recent downtime and data loss

PostPosted: Mon Jun 13, 2011 6:18 am
by Leonardo
£exa wrote:The file size reduction took place earlier and is no consequence of the rebuild.

No it wasn't reduced, because I uploaded the same file just before GHF went down, but today I couldn't re-upload it.  :D

Re: Recent downtime and data loss

PostPosted: Mon Jun 13, 2011 6:41 am
by Kiteflyer61
Woo Hoo!  :dancing-banana:  :dancing-banana:  :dancing-banana:

We're back! I didn't know what to do with all the extra time and nowhere to waste it.  :sheo:

Hopefully we won't even notice the bump in the road this caused. :) I'd hate to be the guy who caused it though. Unemployment sucks.  :violent:





KF

Re: Recent downtime and data loss

PostPosted: Mon Jun 13, 2011 7:03 am
by £exa
Leonardo wrote:
£exa wrote:The file size reduction took place earlier and is no consequence of the rebuild.

No it wasn't reduced, because I uploaded the same file just before GHF went down, but today I couldn't re-upload it.  :-D
It was. Otherwise the new Picks'n'Probes Fix (> 3MB) would have been upped here and not @ PES - that's a fact. It could be an issue of different user group settings though since I'm just a noob user here...  :sheo:

...ah, and to celebrate the revival I'll take the chance and get myself a new avatar. After two years it's about time :)

Re: Recent downtime and data loss

PostPosted: Mon Jun 13, 2011 7:07 am
by Jac
PHPBB's default file upload limit is 2MB. The board probably reset when Fligg put it back up. It's good to see the old girl back in action though.

Re: Recent downtime and data loss

PostPosted: Mon Jun 13, 2011 7:41 am
by Fliggerty
Yeah, a lot of the configurations on the server are set back to default.  The upload size limit is a PHP setting.  I'm working on tweaking things a bit today to get us back to speed.

Also note that I will be taking the board offline for an hour or so later.  I need to take MySQL offline for a bit while I set up an asynchronous (ie immediate) database replication system for backups.

Re: Recent downtime and data loss

PostPosted: Mon Jun 13, 2011 7:46 am
by Jac
Good idea. That should mitigate any future issues with site down time.

Re: Recent downtime and data loss

PostPosted: Mon Jun 13, 2011 8:28 am
by Leonardo
Me too Jac. :)

Re: Recent downtime and data loss

PostPosted: Mon Jun 13, 2011 9:43 am
by Fliggerty
MMH is back online!  Image

Re: Recent downtime and data loss

PostPosted: Mon Jun 13, 2011 9:49 am
by Maiqdabomb
Hooray! :nanadance:

Re: Recent downtime and data loss

PostPosted: Mon Jun 13, 2011 9:50 am
by Kiteflyer61
Cool!  :twothumbs:

Looks like the only thing missing is a few screenshots.  :sheo:  :cheer:  :hoorah:




KF

Re: Recent downtime and data loss

PostPosted: Mon Jun 13, 2011 10:03 am
by Leonardo
That was fast Fligg I mean really fast.  :grin:

Well done and we Stewards can get back to work.  :whip:

Re: Recent downtime and data loss

PostPosted: Mon Jun 13, 2011 11:12 am
by Melchior Dahrk
Woot! Nice work Fliggerty. I'm glad I hadn't put any extensive time into the site recently, all that I saw effected that I had done was maybe 3 mods reverted to their old directories and then the associated screenshots removed.

I'll fix that pronto :D

edit: Taken care of 8-)

Re: Recent downtime and data loss

PostPosted: Mon Jun 13, 2011 2:54 pm
by Lokichan
I thought it was my internet acting up...I was making a detailed post about some stuff and I hit submit and it was like the world turned off. Reset internet and everything but the site didn't work...I was a sad panda.
But yay its working now and I can re-post whatever.

Thank you for being so savvy Fligg and taking care of the site. If it had been someone else at the controls we mighta lost a lot more than a few days worth of posts. :-)

Re: Recent downtime and data loss

PostPosted: Mon Jun 13, 2011 2:55 pm
by Fliggerty
Yeah...I know how you felt.  I was quite sad too.  There was a plethora of emotions really....

It has been a learning experience of course.  I have learned of some really neat backup methods today that I never thought of before.  So overall, we'll be better than we were originally.