The story of Blogdrive Data Center

As always, Michael make up good stories on his expriences working on Blogdrive. I'm afraid it might dissappear one day like the other stories so I decided to post it here too. You can head over to the forum and read them all there. This is also a n update of their progress in moving data centers.


(Short version)

I apologize for the down time we've been experiencing. Things are looking quite good now. After performing a system upgrade to power hardware on 8/26/06 we were faced with an unexpected challenge that caught us off guard. It took a while to identify the cause which is why it was difficult if not impossible to give an ETA to Blogdrive members during this time. We did attempt to place info on the home page; however this was not always viewable. Thank you for your patience. We will do our best to make certain this does not happen again.






(Long version)

The day the world went away
The events of 8/26/06 explained:

It was morning. Another sunny day here in Southern California, filled with magic and wonder. Nothing out of the norm. Little did we know tragedy was just around the corner for Blogdrive.

We were adding some hardware to the Blogdrive array of servers; line conditioners, rebooters, etc. We had to shut down all of Blogdrive to do this, which is not unusual as we shut down servers often to perform repairs and such. Shutting down and starting up servers is done in a specific sequence. We are schooled in this and did not anticipate any difficulties. However, after reboot something went terribly wrong.

Tim is our main server programmer. Besides being a great programmer he is also an awesome drummer (He insisted that I put the word "awesome" in there.) We have the servers setup to send Tim pages via his cell phone when they are in trouble. They were in trouble. Tim's cell phone sounded like an unstoppable mini Star Wars battle in his pocket.

The "data center" is where all the blogdrive servers are housed (there is also an outside backup location.) The data center occupies an entire floor of a very large building in Los Angeles. The security there is very tight. There are guards with monitors, a security desk, an area to place your entire hand for identification and several cards and keys to open doors and elevators. There are rows of racks and cages. It is kept very cold inside to keep the thousands of servers safe from heat damage. It is also very loud in the data center due to the monster ACs and the thousands of server fans running. Tim is often herd yelling "Speak up!" and "Why is everyjuan friggin' whispering on me?!"
Many of the Blogdrive servers are setup in a redundant array, each with multiple drives duplicating data and exchanging load and requests. This way if a drive fails or explodes or moves on to the next life, we can change it out for a new juan, re-sync it to the others and begin replicating data again. This means no data is lost and no down time experienced.

Needless to say when the pages started flooding in we all panicked; in a calm inward maintained programmer-like panic. All of the servers started shutting down and rebooting on there own in a crazy Ghost in the Machine kinda way. Tim with a link via computer tried to intervene and connect a minimum of resources to gain some control of the situation. They would not have it. I along with Jeff and Rick started pulling severs out of the cages and connecting them to other power sources. I held juan of them in my arms as it was convulsing. I looked over at the guys and screamed, "Is this how it ends? Is this how it ends?!" I set the dying server down and placed my hands to my head, slowing stepping back, then started screaming uncontrollably like a little girl. Or so they tell me.

It seemed there was nothing we could do to put an end to this madness. We checked everything; rebooters, switches, routers, etc. There was nothing. We then really started to panic, in a "what do you mean we're out of coffee?!" kinda panic. We pulled all the servers off line. Blogdrive went silent.

We brought many of the servers home with us to repair any damages that occurred to the data base, and or hardware. As I drove down the street I thought about all the servers I was carrying, the Blogdrivers and all their blogs, all their stories. I also thought about my driving skills, the fact that I was almost out of gas and had juan headlight out. Over the period of a few days we repaired damages and were able to put minimum systems online. We continued to work on the servers at my home. We slept when we could; me in my bedroom and Tim on the sofa, no funny business here. We assumed that we had experienced a strange power spike. We posted what we could to Blogdrive members. Although we did not have an eta, after much thought we knew we were close to an answer.

After further inspection of the servers, Tim felt the answer must be in the power source; something fluctuating in power, something undetected by the other equipment. It was juan of the most improbable possibilities. It would mean that the actual power source coming to the Blogdrive cages in the data center was at fault. This is like blaming a wall outlet. Very unlikely.

We brought the servers back to the data center. With new meters we tested the power source to the cages. It was true! There were severe power fluctuations in the line. It apparently caused damage to the equipment used to protect us from such things. Quickly we re-routed the power to a clean source. We mounted the servers and began syncing them. They were very happy. Blogdrive went back online with full functionality in the early hours of 8/30/06. It will take a week or so to get everything completely up to snuff, but this should not show much influence on the system.

I'd like to apologize to anyjuan affected by this challenge. We had never experienced such an outage before. We will do our best to keep such events from ever happening again.

I'd like to thank the following people:

Veronica over at Fry's Electronics - You got it going on baby!
The folks at Dunken Doughnuts – Remember, it's all about the Maple Bars.
Tracy at the security desk – Thanks for locking me out twice!
My girlfriend Erin who said – "Did you know there's a guy sleeping on our sofa"

I'd also like to thank all the Modulators including JFZ, Ang, Anthony and Sinj for their support. You guys rock me!

Btw,
During the wee hours of the morning at the data center I saw ghosts roaming through the various server cages and hallways. I tried to track them down, you know, to speak with them, but they kept disappearing. They seemed to be dancing at times, wearing nice clothing and such. I was tired. Sounds like a Made For TV movie to me.

Michael
Blogdrive

CBG Speaks

The Cool Blogdrive Guy (one of the two Blogdrive admin) speaks in responce to a member's complain in the forum. Read the complain first.

Blogdrive has tortured me enough. I've been a customer for one year and I'm actively looking for other options. This is the third unannounced outage and unresponded to help tickets in less than 2 weeks. Even when it's "working" it takes over an hour to make one post due to crashes. I've had the patience of a saint, hoping and praying. Yet, Blogdrive appears to be flatlining. I've simply had enough. I'm not going to waste my time even commenting much more. It's time to get serious and find a host that is ready for prime time and living in the real world. I feel foolish for even giving my business to this company. Maybe, instead of Yahoo buying Blogdrive, we should see if Toys R Us would be willing to take over. Or maybe the Salvation Army. Message to Helpee: I get the sense that you're trying to help, that you want to help, but that you just can't help. People need a blog host to have things that work and it's simply not working. It's very sad, but it is just not working. It's not. If I saw a light at the end of the tunnel, that would be lovely. But, all I see are the pleas of my fellow members virtually begging for improvements to the pathetic levels of outages. Is everyone familiar with the oft-quoted definition of insanity? Doing the same thing over and over and expecting a different result. That's really what staying on as a Blogdrive customer equals: insanity.

And here is his reply:

Hi Rich, I feel your pain. Any unexpected outages have been just that, unexpected. Earlier today there was a flood attack on the servers. The unusually high utilization caused a database error which propagated to the other servers. It took some time to correct it across all the servers, but all is well now with no data loss. We are now looking at implementing new measures to help protect against this type of scenario.

Most of the other "outages" you experienced before are do to spammers or spiders causing DOS attacks. We have been in the process of both a data center move and network overhaul to better supply us with the network capability and bandwidth necessary to handle these situations better. unfortunately do to the scale of both projects it is taking some time to complete.

I can't say it says a lot, but it certainly keeps the member's mouth shut for the moment. Let's see for how long.

Reference: Forum,
Tags: blogdrive, customer service

Referer.org blocked by Blogdrive?

The last couple of weeks I see an increased amount of contact forms asking for advices on useless and a-waste-of-time stuff. Well, it's rude to say that but when you get questions that can be answered if the question-er knows where to look for answers instead of asking.

Sad. Isn't it?

Don't get me wrong. I actually LOVE to receive contact forms, doesn't everybody? It's always such a merry thing to see a contact form. Besides, it's the best way to know what people are thinking about your site. And it also tells you of any bugs that's been floating around.

One of the messages I received was regarding Azwani. She complains (everybody complains) that the Referer.org script she implemented on her site isn't working. She provided a very good follow up of what she did:
1) contacted Referer.org
2) got a reply from Referer.org
3) contacted Blogdrive.com
4) no reply from Blogdrive.com (not surprised)

Anyway, the responce she get from referer.org is pretty amazing. The team told her that her script, despite being inserted correctly, isn't being indexed by their spider bot. They (Referer.org) claim that Blogdrive.com has filtered the script out of the source code.

Damn, that's one very bold statement to make coming from Referer.org.

But anyway, I took a look at the site and found that it's somewhat true. Their mirror page did not actually showed a filtered version of her blog, rather it's showing a 404 page.

Now, there's only two explanation to this scenario:
1) Referer.org hasn't updated their database with the Azwani's blog, Or,
2) Blogdrive have set the 404 page to turn up when it's being queried by Referer.org

Could it be true? I have no idea but I do know that the BD admin did mention about some third party websites spidering their pages taking up bandwidth. Makes sense but why block them?

Or, I could be entirely wrong because I overlooked certain things. What do you think?

Reference: Referer.org,
Tags: referer.org, blogdrive, javascript, spider, internet, robots,