Moderator: Community Team
Their techie explained to me that this hardware-level auto-shutdown is being done and is part of the problem. The database engine doesn't get a chance to close properly when the power gets cut suddenly and that messes up the data (but only data stored in MySQL InnoDB tables).bedub1 wrote:Do the servers have a temp sensor? Set them to automatically shut themselves down if the CPU exceeds 70 degrees C and if the ambiant air/motherboard temp exceeds 60 or so degrees C. Then the next time this happens, CC will be down, but you won't experience any corruption in the database and it *should* be quick and easy to bring everything back up.
The software I've seen, will send a command to the OS to shut down. The OS then sends a command to the database engine that "hey...the computer is turning off.....you need to shut your shit down". I know there are scripts that can be written to manually close the database etc. When my power gets lost, my server runs till 50% power left...then issues a command to itself, and all my other workstations, to power off. Vista/XP just shuts the system down properly....just like you hit start- shut down - but sometimes the script might need to be modified to send commands to the database prior to sending commands to the OS. I know it's possible...and an ounce of prevention can keep away serious disaster....lackattack wrote:Their techie explained to me that this hardware-level auto-shutdown is being done and is part of the problem. The database engine doesn't get a chance to close properly when the power gets cut suddenly and that messes up the data (but only data stored in MySQL InnoDB tables).bedub1 wrote:Do the servers have a temp sensor? Set them to automatically shut themselves down if the CPU exceeds 70 degrees C and if the ambiant air/motherboard temp exceeds 60 or so degrees C. Then the next time this happens, CC will be down, but you won't experience any corruption in the database and it *should* be quick and easy to bring everything back up.
I'm looking into ways to prevent sudden power loss from corrupting this data.
Sorry to double post.Twill wrote:@ Lack
damn 4:30am?
I was monitoring it til 1:30am when I gave up...this is why I leave these things to you
@ Everyone
We will be looking at other data centers, but as I'm sure you're aware, moving's not an easy nor a inexpensive prospect.
We will also, of course, be following up with the Data Center to figure out this whole "it's gettin' hot in hear, so shut off all your sites" thing and if they intend to actually fix it.
Considering Rackspace's generally very good history, this is surprising to say the least.
As always, sorry for the disruptions.
Twill
It would be nice if you would get as sick of "always" saying that as we are of "always" hearing it, and take immediate practical steps to prevent any future disruptions, BEFORE they occur. (Among professionals, prevention of shabby service issues before the fact is called a "win-win" situation).Twill wrote:As always, sorry for the disruptions.
Twill
KLOBBER wrote:It would be nice if you would get as sick of "always" saying that as we are of "always" hearing it, and take immediate practical steps to prevent any future disruptions, BEFORE they occur. It is highly unprofessional to be constantly waiting in a state of impotence for the next disruption, fixing it only after the fact, and finding yourself in the embarrassing position of "always" having to apologize.Twill wrote:As always, sorry for the disruptions.
Twill
This person makes a good point. Your opening page even states that I can take my turn "with my morning cup of coffee". But if the site is down in the morning, there is no more coffee until tomorrow.Markomuncho wrote: how come you only add 10 hours???
I can only log on once a day, you ecpect me to get up in the night too see if you fixed it yet.
Not a happy chappy
I'm not sure what planet you and the other complainers live on, but on Earth, computers are occasionally unreliable. Utter, 24/7/365 reliability, like you might see on espn.com or cnn.com or fark (which itself goes down every so often), costs great heaping gobs of money and still doesn't guarantee anything.Caleb the Cruel wrote:I'm no computer techie, but I know that in 2008 this is unacceptable. There must be easy ways to prevent such a disaster, such as switching to a more reliable server like others have stated. We pay to play, not to say "Ah man, CC is down again!". I've paid, let me play when I want.
Perfection is not possible, I know that. However improvement is possible and should be expected! Downtimes are happening more frequently rather than less frequently which is simply not right. And no, I do not feel I have gotten what I paid for. I paid to play as many games as I want, when I want. Not just when the server decides to have a good day.Incandenza wrote:I'm not sure what planet you and the other complainers live on, but on Earth, computers are occasionally unreliable. Utter, 24/7/365 reliability, like you might see on espn.com or cnn.com or fark (which itself goes down every so often), costs great heaping gobs of money and still doesn't guarantee anything.Caleb the Cruel wrote:I'm no computer techie, but I know that in 2008 this is unacceptable. There must be easy ways to prevent such a disaster, such as switching to a more reliable server like others have stated. We pay to play, not to say "Ah man, CC is down again!". I've paid, let me play when I want.
Expecting any website with a small handful of paid employees and a limited revenue stream to perform flawlessly, always, is lunacy. Besides, in the last 12 months, CC has been down for, what, 24 hours total? Maybe 36? You guys are getting up in arms about that? Seriously? You don't feel like you're getting your 25 bucks worth?
Note to others - Rackspace supported YouTube - until Google bought them out...Twill wrote: Considering Rackspace's generally very good history, this is surprising to say the least.
Okay, I've got an even better than superb dice on Round 3 so I'm well into the lead now ...looks like everything is definitely under my control.MOBAJOBG wrote:Well, I've to disagree with that statement because I definitely remember that I've taken over S.America successfully but it is now showing me that I've ran out of time without even bothered to deploy my 8 armies.lackattack wrote:Conquer Club was down for 10 hours
The cause was similar to the mishap on June 15:
Our database server got too hot and shut down, corrupting the data. When they got the server back online the database engine wouldn't even run. Our last backup was 13 hours old, and it would have sucked to roll back the games and user accounts so far back in time. Fortunately the hosting company's database expert was able to fully recover all the data. As I requested he called me at 4:17 am to tell me it's ready so that I can add 10 hours to all games before putting the website back online.Hosting Company wrote:At approximately 5:00 P.M. CDT, our DFW data center experienced a loss of utility power that required we fail over to generator power. During the transition, temperatures increased unexpectedly causing disruption to some customer equipment. We began to bring affected customers back online as temperatures stabilized at approximately 7:30pm CDT.
So there you have it.
My apologies for the downtime, at least your games shouldn't be affected this time (except for speed games, please submit a support ticket if your speed game was ruined).
I'm going back to sleep now...
http://www.conquerclub.com/game.php?game=2806331
2008-07-09 22:21:44 - Incrementing game to round 2
2008-07-09 22:22:35 - MOBAJOBG receives 3 armies for holding Africa
2008-07-09 22:22:35 - MOBAJOBG receives 5 armies for 16 territories
2008-07-09 23:22:35 - MOBAJOBG ran out of time
I'm disappointed as I did experience superb dice which gave me the S.America continent.
I'm clearly at the mercy of my opponent since both Oceania & S.America except for 1 territory each are within red's grasp.
we all aplaude Lack's hard work. If I had a credit card, I'd buy membership just as a way of saying "good job Lack" 
KLOBBER wrote:A better plan would be having no server interruptions, wouldn't it?
awesome, ty lacklackattack wrote:Conquer Club was down for 10 hours
The cause was similar to the mishap on June 15:
Our database server got too hot and shut down, corrupting the data. When they got the server back online the database engine wouldn't even run. Our last backup was 13 hours old, and it would have sucked to roll back the games and user accounts so far back in time. Fortunately the hosting company's database expert was able to fully recover all the data. As I requested he called me at 4:17 am to tell me it's ready so that I can add 10 hours to all games before putting the website back online.Hosting Company wrote:At approximately 5:00 P.M. CDT, our DFW data center experienced a loss of utility power that required we fail over to generator power. During the transition, temperatures increased unexpectedly causing disruption to some customer equipment. We began to bring affected customers back online as temperatures stabilized at approximately 7:30pm CDT.
So there you have it.
My apologies for the downtime, at least your games shouldn't be affected this time (except for speed games, please submit a support ticket if your speed game was ruined).
I'm going back to sleep now...
