Brian Mark - Guitaraholic

Servers. Gotta love 'em. I know I do. It's been a while since I posted up some hardware to keep my geeky friends drooling, so I thought I'd show a few pictures of our hardware setup now. The first picture is our server room, which I haven't featured before. Top to bottom: Tape backup, the server that had our inventory / accounting system on it before (now just file storage for the most part), our old dev box, our internet gateway box (XML interface between our site and our system as well as a firewall / NAT system), a Google Mini for ProperPet, the KVM switch for these servers, 3 dev servers for the sites (1 dev, 2 staging to mirror our setup downtown), the UPS for those 3 (which barely shows up with some green lights), the keyboard and monitor for it, our ACD server, and the big Dell box that powers our system now with its battery right below. That's a decent amount of hardware for one cabinet, but it's not nearly as dense as our environment downtown. This is o...

Now that we're going to put our current back-end into a new site, it turned out to be the perfect time to buy another mini. We'll have two v2 minis running for ToolBarn and a v1 mini on the new site. That'll make load balancing much less of a headache. Also interesting is that they no longer include the print books with the machine, instead giving a CD with the instructions. Seems much smarter to me. Oh, and they now have a 30 day trial. Excellent. I look forward to experimenting with the price range search on the v2, so overall this should be a great upgrade. I'm pretty happy with the $2k we spent this week.

How is it that woot got to be so popular that showing a brown paper bag on their site immediately crashes their servers? Ok, so they don't crash, but they don't respond because of the DOS caused by everyone trying to purchase all at once. Of course, if it were hosted over dialup, that wouldn't be so impressive. But they're not, and I don't get it. How did woot get to be so popular while upsetting so many people when they can't buy something? Ok, so I'm mostly speaking out of frustration, but there are some real issues going on. For example, during the Woot-off last week there were plenty of times I couldn't get it to load. At one point, I saw a BOC (woot fans know what that is), clicked "I want one", then it went back to the previous item saying "Sold out". 30 seconds later the BOC was again on the site, I clicked "I want one", got it added, but the payment timed out. By the time it would actually accept my payment, they we...

Talking about eBay: The ultrapopular auction/sales Web site continues its exponential growth and finds itself adding 10 terabytes of new storage every week. That's a lot of data. Holy cow! We've been running an ecommerce site for several years now, and we've got quite a bit of data, but nowhere near 10 terabytes. And that's just what they add per week? Wow. Of course, I don't have an inside scoop at eBay. This was a quote from an eWeek article . But when I stop to think about 10 terabytes of data, then imagine 10 more the following week (yes, that's more than 1 terabyte per day), that makes me quickly realize we need a new word, because 3 years will top 1,000 terabyes of storage added. That's just incredible. One other thing that seemed very interesting in that article and worth mentioning, was on the final page. When it comes to handling all that traffic, Strong said the world's time zones provide a kind of natural "load-balancer." "Wh...

Since we got our new database servers up and running, it had been about a month. I had both of the servers in sync, but because it was an emergency type of a changeover I skipped setting up replication at that time. Earlier this past week, I went ahead and started setting that up again. It didn't seem like a big of a deal at the time, but it ended up taking 2 and a half days to get the second server caught up. I guess that means we run a few queries each day, although only one slave SQL thread can run at a time which limits the amount of CPU that can be used by either of these severs for replication to about 25%. Since we have dual dual-core machines, it only uses one of the 4 available threads, making 25% the maximum. That was sort of disappointing from a performance point of view, but understandable to keep things perfectly aligned between the machines. Once it was caught up, I went ahead and set up replication the other direction as well. We now have 2 servers which can handle a...

For the past several months now, we've been using Kemp LM-1500 load balancers to split the load among multiple web servers. These are pretty simple Linux based load balancers which don't have hard drives - everything is on CF cards. Well, to make our servers even more balanced, we're going to move to load balancing our database servers and our Mini(s). Yes... load balancing replicated database servers isn't unusual. That's actually a lot of the reason that the load balancers have a second network interface. But mini-balancing? Since Google has been generous enough to offer us some cash off of a second mini, we're going to go ahead and look at buying a second one to put behind the load balancers as well. Load balancing a couple of minis will do several nice things for us. 1) If a mini fails or needs rebooted, we will still have a functional site search. 2) We can use one primarily for searching ToolBarn and one primarily for searching ToolPartsDirect, making fo...

As we're getting our new hosting platform in place, we've noticed a few issues popping up. The first larger issue was PayFlowPro from Verisign and their lack of support for 64 bit operating systems. That was easily worked around, but the next issue has me perplexed. The load balancers we're using work as NAT devices (Network Address Translation), so we have a private network behind them with all of our servers attached and the public network on the other side of them. All traffic to/from our servers goes through the load balancers, making for a firewall. To allow the web servers to talk to sites they want to visit (or request services and information from), NAT is used to allow for the public address of the load balancers to be used in the request to the servers on the other end. This works well for most things I've tried. OS updates install just fine (rather quickly over the large pipes our data center has, I might add). Payment related data works just fine (or at leas...

An essential part of any balanced breakfast... ok, so these ain't no Lucky Charms. We ordered load balancers from Kemp Technologies (no, not hemp... I said Kemp) yesterday and they've already shipped. Load balancers are an essential part of any highly available website infrastructure, allowing for multiple servers to handle the requests based upon rules set up ahead of time. Example #1: Server down The first example of when someone needs a load balancer is when a server crashes. For some websites, this means they're gone. For a load balanced website, this means that the traffic starts hitting the other server(s) and the users have no idea that something happened. Example #2: Server slow Sometimes, a processes goes in an unplanned direction and uses a ton of resources. This presents a more difficult problem - the server responds as if it is up, but it just doesn't perform well. That means that the load balancer needs to watch what's going on (response times) and not ...

We ordered our database servers from IBM. These have less RAM than the Dell machines we ordered as web servers, but they have much more processor. I didn't even think about the Itaniums in the Dell's being 64 bit when I placed the order, so they were more CPU than I thought, but that's nothing compared to what I've got going on here. Base Model The base model is an Opteron powered 1U server with 1GB of RAM, 1 Gb card, and a SCSI adapter on-board. Memory Upgrade We upgraded the RAM on each of these to be a mere 4GB each. Processor Upgrade We went from the base model Opteron to dual Opteron 275's (dual-core) in each machine. Hard Drive Upgrade Databases are only as reliable as their drives, so we went with mirrored storage. Another NIC I also added in a second NIC for total redundancy (one for each switch). Warranty Upgrade IBM seems to have more confidence in their servers than Dell, because for a little less than what Dell's 3 year warranty was we're gettin...

We're trying to upgrade to a load balanced, fully redundant web serving platform now. What we currently have was a $3000 server about 30 months ago, so it's time to look at upgrades. I don't usually like Dell, but their prices are something else. As we're getting ready to upgrade our servers, I happened across Dell's website and checked what they had for a deal at the moment. We ended up getting a few at stupid prices - stupid meaning very low for what we're getting. Base Model The base model on these machines is a 1U rackmount 2.8GHz/2MB Cache Xeon with 1GB RAM, 2 x GB ethernet ports and an 80GB HDD. Nothing too impressive. Free Upgrade To start making it seem a little nicer, they offered a FREE second processor. Customization We added in 7GB more RAM, another drive (mirroring is a good thing) and a few other odds and ends including the 3 year service plan. Since it's Dell, we'll probably need that. ;-) Price Reduction Since it was over $2700, Dell took...