Brian Mark - Guitaraholic

Since people seem to enjoy seeing our hardware (server pron), I thought I'd show our current setup again. I'll start at the bottom of our rack. This is a pic with a few things. First, it's got the white system, which is our memcache box. Just your basic setup (dual dual-core opterons, mirrored 250gig SATA drives, and 32gigs of RAM) for temporary storage of blocks of output. Above it is our Breach WebDefend appliance. I'm not sure the specs (dual quads and 4 gigs of RAM I think), but since it's sold as an appliance it doesn't really matter. Last in this pic are 3 APC power management boxes. Since quite a few of our devices only have one power supply, this allows us to plug them all in to both power sources in case of an outage. Continuing on up the line, we have one of our oldest servers. We call it www1, although being a www server isn't something it does any more. It basically runs cron jobs on its dual xeon processors with 8gigs of RAM and mirrored 80gig S...

A couple of weeks after I read about Facebook getting $$$ to buy 50,000 more servers , I found it amusing to see this image on Twitter: Come on already, Twitter. Everyone can see your resources aren't sufficient. Get with the program and buy some servers already.

Servers. Gotta love 'em. I know I do. It's been a while since I posted up some hardware to keep my geeky friends drooling, so I thought I'd show a few pictures of our hardware setup now. The first picture is our server room, which I haven't featured before. Top to bottom: Tape backup, the server that had our inventory / accounting system on it before (now just file storage for the most part), our old dev box, our internet gateway box (XML interface between our site and our system as well as a firewall / NAT system), a Google Mini for ProperPet, the KVM switch for these servers, 3 dev servers for the sites (1 dev, 2 staging to mirror our setup downtown), the UPS for those 3 (which barely shows up with some green lights), the keyboard and monitor for it, our ACD server, and the big Dell box that powers our system now with its battery right below. That's a decent amount of hardware for one cabinet, but it's not nearly as dense as our environment downtown. This is o...

My dad got me a shiny new Porsche for Christmas. Actually, it was a 500GB L.A. Porsche designed hard drive from Lacie that he bought through Amazon. Faster for data access, but not so fast at getting me to work. Still, this is pretty sleek and shiny, and that's what matters when picking out your Porsche, right? I guess I'll have to wait for the real Porsche for a while yet. In the meantime, I'll just enjoy my 500 gigs of external storage. Maybe next year... hint, hint.

Getting ready for the holiday season means beefing up our infrastructure around here. So, I got to order a new box for Memcached . It was even a simple server, just two dual-core processors with dual 250 gig drives from Sun running Enterprise Linux. What's fun about it, though, is that it has 32 gigs of RAM from the start (once I drop the chips into the box, anyway) and is upgradeable to 64 gigs. Now that's fun. I figure that going from the 4 gigs on one server we were running last week to 4 gigs on 3 servers we were running this week to 2 gigs on 3 servers plus 30 gigs on a dedicated box we'll be running in a couple weeks means we're going to be set for some serious growth. Now it's time to plan for a happy holiday season. At least our website shouldn't go black come black Monday.

I've been working on configuring a development web server for toolbarn.com over the past couple of days. Was it a quad processor, quad core system with terabytes of RAM and storage? Perhaps a Sun T-2000 with more threads than you can shake a stick at? Not a chance. I'm using old hardware to simulate our web servers under a heavier load. That, and it was sitting here not being used. I commissioned a Dell PowerEdge 2300 with 768MB of RAM, dual PIII 600's, and 3 8GB SCSI drives in Software Raid 5. Not exactly high-end stuff that we'd want to run anything serious on, but for developing a website, it'll be just fine. We also set up a PhpWiki for keeping track of development notes internally among the team in IT, and we'll be doing lots of documentation over the next little while. Oh, what fun. I remembered, after getting 403 permission denied errors on all folders, that Apache requires some special tuning of the selinux parameters. Note to self: find this post ne...

We've got a pair of replicated MySQL database servers. That's great - until the replication process corrupts both. That's the situation we're currently in. Much to my (dis)pleasure, I get to use the platinum support that we paid a bunch for. I called them and had an engineer on the phone within 15 - 20 minutes (they answered right away and had him call me back, which was awesome.) The little bit of hair I have left is getting pulled out by the minute, though. It's really not simple troubleshooting a crashed pair of servers, as we've already been working on this for 4-1/2 hours. I'm waiting for the engineer to contact me once again, so I thought I'd post something quick. This was, however, a great opportunity to upgrade to the latest version of MySQL enterprise. Oh, the joys of doing business online.

Today we came across some really cool PC cases while trying to find one to hold my 12"x13" motherboard. Not only do some of the new cases not look like PC's at all, they have some neat features. This got Matt and I talking about upgrading our home PC's again, and we came to an interesting conclusion. In the past, it was easy enough to buy a new motherboard and processor and call it an upgrade. Now, the accessory technologies have changed so much that both of the PC's we have at home would basically need complete replacements. This has us both frustrated to the point of not wanting to upgrade. Basically, we both have AGP video cards, so we'd want to change to SLI or PCI-Express as a minimum. We'd also want SATA Raid-Spec drives, and memory technology has changed enough to make our RAM need replacement if we get a new machine. But what really bugs us is that we would only be re-using the sound card, and those are sort of outdated as well. Oh, and we'd ...

An essential part of any balanced breakfast... ok, so these ain't no Lucky Charms. We ordered load balancers from Kemp Technologies (no, not hemp... I said Kemp) yesterday and they've already shipped. Load balancers are an essential part of any highly available website infrastructure, allowing for multiple servers to handle the requests based upon rules set up ahead of time. Example #1: Server down The first example of when someone needs a load balancer is when a server crashes. For some websites, this means they're gone. For a load balanced website, this means that the traffic starts hitting the other server(s) and the users have no idea that something happened. Example #2: Server slow Sometimes, a processes goes in an unplanned direction and uses a ton of resources. This presents a more difficult problem - the server responds as if it is up, but it just doesn't perform well. That means that the load balancer needs to watch what's going on (response times) and not ...

We ordered our database servers from IBM. These have less RAM than the Dell machines we ordered as web servers, but they have much more processor. I didn't even think about the Itaniums in the Dell's being 64 bit when I placed the order, so they were more CPU than I thought, but that's nothing compared to what I've got going on here. Base Model The base model is an Opteron powered 1U server with 1GB of RAM, 1 Gb card, and a SCSI adapter on-board. Memory Upgrade We upgraded the RAM on each of these to be a mere 4GB each. Processor Upgrade We went from the base model Opteron to dual Opteron 275's (dual-core) in each machine. Hard Drive Upgrade Databases are only as reliable as their drives, so we went with mirrored storage. Another NIC I also added in a second NIC for total redundancy (one for each switch). Warranty Upgrade IBM seems to have more confidence in their servers than Dell, because for a little less than what Dell's 3 year warranty was we're gettin...

We're trying to upgrade to a load balanced, fully redundant web serving platform now. What we currently have was a $3000 server about 30 months ago, so it's time to look at upgrades. I don't usually like Dell, but their prices are something else. As we're getting ready to upgrade our servers, I happened across Dell's website and checked what they had for a deal at the moment. We ended up getting a few at stupid prices - stupid meaning very low for what we're getting. Base Model The base model on these machines is a 1U rackmount 2.8GHz/2MB Cache Xeon with 1GB RAM, 2 x GB ethernet ports and an 80GB HDD. Nothing too impressive. Free Upgrade To start making it seem a little nicer, they offered a FREE second processor. Customization We added in 7GB more RAM, another drive (mirroring is a good thing) and a few other odds and ends including the 3 year service plan. Since it's Dell, we'll probably need that. ;-) Price Reduction Since it was over $2700, Dell took...

Yesterday, as we were meeting with our local Sun VAR, he mentioned that some data pipes had been run into their area. He wasn't talking the normal data pipes that we hear about every day. No, not just a T-1. Not a DS-3. Not an OC-3 at 155Mbps. No, he was talking INSANE bandwidth. Definition of Insane My usual definition of insane is doing the same action over and over expecting different results. But that doesn't apply here. Perhaps X-Treme would fit better. This is like the X-Games of bandwidth. He was talking about OC-192's, which are 10Gbps each. But he wasn't talking about 1. Nor was he talking about 2. He was talking about 50. Yes, that makes 500Gbps. What could possibly require that type of bandwidth? Better yet, how can I attach a server at the end of those cable runs? Just how big is this? I'll leave you with a visualization of just how big of a data pipe I feel this is. I doubt it actually occupies all that much space physically, but in our virtual world si...