Brian Mark - Guitaraholic

Since people seem to enjoy seeing our hardware (server pron), I thought I'd show our current setup again. I'll start at the bottom of our rack. This is a pic with a few things. First, it's got the white system, which is our memcache box. Just your basic setup (dual dual-core opterons, mirrored 250gig SATA drives, and 32gigs of RAM) for temporary storage of blocks of output. Above it is our Breach WebDefend appliance. I'm not sure the specs (dual quads and 4 gigs of RAM I think), but since it's sold as an appliance it doesn't really matter. Last in this pic are 3 APC power management boxes. Since quite a few of our devices only have one power supply, this allows us to plug them all in to both power sources in case of an outage. Continuing on up the line, we have one of our oldest servers. We call it www1, although being a www server isn't something it does any more. It basically runs cron jobs on its dual xeon processors with 8gigs of RAM and mirrored 80gig S...

Since we moved to a new data center from our old one, people have asked how much empty space we have. I mentioned we got more, but I never really said how much. The picture to the right should show you (sort of) how much space we got. I had to take 3 pictures and stitch them together (and I didn't spend much time doing it) since my phone couldn't get it all at once. Yes, we've finally got some room for expansion. I like expansion... that means more money I get to spend!!! Oh, and we still have 1 Google Mini, 1 Database server and 1 web server to move over. It's getting closer to a fully functional death star... er... web star... er... something.

A couple of weeks after I read about Facebook getting $$$ to buy 50,000 more servers , I found it amusing to see this image on Twitter: Come on already, Twitter. Everyone can see your resources aren't sufficient. Get with the program and buy some servers already.

Servers. Gotta love 'em. I know I do. It's been a while since I posted up some hardware to keep my geeky friends drooling, so I thought I'd show a few pictures of our hardware setup now. The first picture is our server room, which I haven't featured before. Top to bottom: Tape backup, the server that had our inventory / accounting system on it before (now just file storage for the most part), our old dev box, our internet gateway box (XML interface between our site and our system as well as a firewall / NAT system), a Google Mini for ProperPet, the KVM switch for these servers, 3 dev servers for the sites (1 dev, 2 staging to mirror our setup downtown), the UPS for those 3 (which barely shows up with some green lights), the keyboard and monitor for it, our ACD server, and the big Dell box that powers our system now with its battery right below. That's a decent amount of hardware for one cabinet, but it's not nearly as dense as our environment downtown. This is o...

Getting ready for the holiday season means beefing up our infrastructure around here. So, I got to order a new box for Memcached . It was even a simple server, just two dual-core processors with dual 250 gig drives from Sun running Enterprise Linux. What's fun about it, though, is that it has 32 gigs of RAM from the start (once I drop the chips into the box, anyway) and is upgradeable to 64 gigs. Now that's fun. I figure that going from the 4 gigs on one server we were running last week to 4 gigs on 3 servers we were running this week to 2 gigs on 3 servers plus 30 gigs on a dedicated box we'll be running in a couple weeks means we're going to be set for some serious growth. Now it's time to plan for a happy holiday season. At least our website shouldn't go black come black Monday.

Today, I was realizing I haven't done a post in a while, and I also got a new camera from the recent Woot-off. Combine those, and I got pictures of some geeky things. First off, above is my podcasting setup. I don't have everything mounted in my new rack from Rack Solutions yet, but you can see the stuff I regularly use right now. That cost a few pennies over the years. :) Secondly, those are the servers we use for ToolBarn. We have 1 more Google Mini that we'll be adding shortly to this mess, making 12U of our 13U being used and 1U for storing the external drives we swap out weekly for off-site backups as well as pass-through to one of the switches (we ran out of room on the back for those.) Finally, that mess is my desk at work. I regularly get working on too many things at once and can't really keep up on things. However, the dual screens are almost a necessity now, and I have a matching 22" at home along with a really nice 20" Samsung. I remember in the o...

If you use shared hosting, be aware that your server may not always be correctly configured to handle daylight savings time changes. The podcast I had intended to go live at 10am will actually be live at 11am central due to the server David is hosting on not being updated with the latest daylight savings time changes. How I wish I had hosted this thing on my own... but I'd rather not put out the cash when David had plenty of bandwidth available. It just seems silly to spend money when he already had something available.

With everyone being so hot to use Ajax for things now, it seems that people forget how much simpler things can be done if you just stop and use some old-school techniques. I know I like Ajax for some things, but if it isn't necessary, then why introduce the extra bit of complexity? I'm speaking here about the 204 status code. Quoting the W3 204 No Content The server has fulfilled the request but does not need to return an entity-body, and might want to return updated metainformation. The response MAY include new or updated metainformation in the form of entity-headers, which if present SHOULD be associated with the requested variant. If the client is a user agent, it SHOULD NOT change its document view from that which caused the request to be sent. This response is primarily intended to allow input for actions to take place without causing a change to the user agent's active document view, although any new or updated metainformation SHOULD ...

After a couple of "What happened" type moments in the past, we've finally decided to set up a versioning system. At this point, we're trying to set up subversion because Nick has seen good things about it. CVS seems a little out of date, and SVN seems to find strengths in CVS's weaknesses. Yet another adventure in a growing company begins today...

I reported not that long ago that we finally got a development server . Well, we've already shut that one off and moved to a new one. Apparently, that was much more of an antique than I thought it would be. We're now running a Hyperthreaded P4 with 2GB of RAM and hardware mirrored SCSI drives. I thought that the old machine would just act like our web servers under heavy load, but when it was taking 2 minutes or longer just to load a couple fields via Ajax calls, I decided that wasn't worth waiting for any more. Now, I'm going to be able to develop a little quicker... and I've got some sweet projects planned out. When I get them a little further along, I'll be sure to let anyone reading here know about them.

Every once in a while, a must-have product comes along that, for the price, just can't be passed up. I was excited when Simple Network Monitor offered to pay me to review their product, because quite simply, I have a need for it and getting paid to evaluate it seems almost too good to pass up. The download of the 30 day trial was pretty quick and painless, as was the simple installation process. On starting the program, it came up with "You are on day 1 of your 30 day evaluation" and the options of close, buy, and register. I chose close and the program started up. Clicking on the "Add a Server" icon brought up a very simple form with Hostname, Friendly Name, Location, Server Notes, Alert E-Mail Address(es), and checkboxes for Override E-Mail Addresses, Server Monitoring Enabled, and Create a Ping Monitor. Once I filled in the basic fields, it showed up in my list of servers. From there, I could add a monitor. I selected HTTP Monitor, set the interval and time...

I've been working on configuring a development web server for toolbarn.com over the past couple of days. Was it a quad processor, quad core system with terabytes of RAM and storage? Perhaps a Sun T-2000 with more threads than you can shake a stick at? Not a chance. I'm using old hardware to simulate our web servers under a heavier load. That, and it was sitting here not being used. I commissioned a Dell PowerEdge 2300 with 768MB of RAM, dual PIII 600's, and 3 8GB SCSI drives in Software Raid 5. Not exactly high-end stuff that we'd want to run anything serious on, but for developing a website, it'll be just fine. We also set up a PhpWiki for keeping track of development notes internally among the team in IT, and we'll be doing lots of documentation over the next little while. Oh, what fun. I remembered, after getting 403 permission denied errors on all folders, that Apache requires some special tuning of the selinux parameters. Note to self: find this post ne...

I've never really had to work on a hosting platform I didn't have complete control over. Ok, so back in 1999 we were hosting ToolBarn on Earthlink (yes, many many moons ago), but since then we've always had root access to our servers, and it wasn't actually too long after that when we started using our own hardware. ISP's are fine for getting started (we used dedicated server options at a few ISP's before moving to our own hardware), but when you need to make a change nothing beats root level access. As I'm doing some simple changes to a site hosted on Yahoo's hosting platform, it really feels foreign. It's almost as bad as trying to work on the first site I ever worked on... maybe worse than that. I'm so used to making changes via SSH or some other server side login that even the simple things seem to take me forever. Wow... how does the rest of the world get along this way? The more I have to do on this project, the happier I am that we have ou...

As we're getting our new hosting platform in place, we've noticed a few issues popping up. The first larger issue was PayFlowPro from Verisign and their lack of support for 64 bit operating systems. That was easily worked around, but the next issue has me perplexed. The load balancers we're using work as NAT devices (Network Address Translation), so we have a private network behind them with all of our servers attached and the public network on the other side of them. All traffic to/from our servers goes through the load balancers, making for a firewall. To allow the web servers to talk to sites they want to visit (or request services and information from), NAT is used to allow for the public address of the load balancers to be used in the request to the servers on the other end. This works well for most things I've tried. OS updates install just fine (rather quickly over the large pipes our data center has, I might add). Payment related data works just fine (or at leas...

So what's the first thing I had to do when shown a pile of servers to install? That's right, crack them open and see what makes 'em tick. I'll start out with the web servers. Web Servers These are 3 machines that were laid out pretty cleanly. You probably can't see it, but there are 4 fans in front of each CPU. Several fans on the power supply and another across the motherboard make this one noisy server. Of course, since heat is the natural enemy of computers, I don't really care that it has more fans than all of the PC's in my office combined. I think that's a good thing. As you can see from the picture above, the memory is easy to get at, and the wiring is pretty straight forward. There wasn't even a need for an add-on board in these servers. Of course, pretty much anything failing means the motherboard needs to be replaced in a Dell, but that's the way most any name brand server is these days. Mail Server The mail server that we'll be usi...

An essential part of any balanced breakfast... ok, so these ain't no Lucky Charms. We ordered load balancers from Kemp Technologies (no, not hemp... I said Kemp) yesterday and they've already shipped. Load balancers are an essential part of any highly available website infrastructure, allowing for multiple servers to handle the requests based upon rules set up ahead of time. Example #1: Server down The first example of when someone needs a load balancer is when a server crashes. For some websites, this means they're gone. For a load balanced website, this means that the traffic starts hitting the other server(s) and the users have no idea that something happened. Example #2: Server slow Sometimes, a processes goes in an unplanned direction and uses a ton of resources. This presents a more difficult problem - the server responds as if it is up, but it just doesn't perform well. That means that the load balancer needs to watch what's going on (response times) and not ...

We ordered our database servers from IBM. These have less RAM than the Dell machines we ordered as web servers, but they have much more processor. I didn't even think about the Itaniums in the Dell's being 64 bit when I placed the order, so they were more CPU than I thought, but that's nothing compared to what I've got going on here. Base Model The base model is an Opteron powered 1U server with 1GB of RAM, 1 Gb card, and a SCSI adapter on-board. Memory Upgrade We upgraded the RAM on each of these to be a mere 4GB each. Processor Upgrade We went from the base model Opteron to dual Opteron 275's (dual-core) in each machine. Hard Drive Upgrade Databases are only as reliable as their drives, so we went with mirrored storage. Another NIC I also added in a second NIC for total redundancy (one for each switch). Warranty Upgrade IBM seems to have more confidence in their servers than Dell, because for a little less than what Dell's 3 year warranty was we're gettin...

We're trying to upgrade to a load balanced, fully redundant web serving platform now. What we currently have was a $3000 server about 30 months ago, so it's time to look at upgrades. I don't usually like Dell, but their prices are something else. As we're getting ready to upgrade our servers, I happened across Dell's website and checked what they had for a deal at the moment. We ended up getting a few at stupid prices - stupid meaning very low for what we're getting. Base Model The base model on these machines is a 1U rackmount 2.8GHz/2MB Cache Xeon with 1GB RAM, 2 x GB ethernet ports and an 80GB HDD. Nothing too impressive. Free Upgrade To start making it seem a little nicer, they offered a FREE second processor. Customization We added in 7GB more RAM, another drive (mirroring is a good thing) and a few other odds and ends including the 3 year service plan. Since it's Dell, we'll probably need that. ;-) Price Reduction Since it was over $2700, Dell took...

Yesterday, as we were meeting with our local Sun VAR, he mentioned that some data pipes had been run into their area. He wasn't talking the normal data pipes that we hear about every day. No, not just a T-1. Not a DS-3. Not an OC-3 at 155Mbps. No, he was talking INSANE bandwidth. Definition of Insane My usual definition of insane is doing the same action over and over expecting different results. But that doesn't apply here. Perhaps X-Treme would fit better. This is like the X-Games of bandwidth. He was talking about OC-192's, which are 10Gbps each. But he wasn't talking about 1. Nor was he talking about 2. He was talking about 50. Yes, that makes 500Gbps. What could possibly require that type of bandwidth? Better yet, how can I attach a server at the end of those cable runs? Just how big is this? I'll leave you with a visualization of just how big of a data pipe I feel this is. I doubt it actually occupies all that much space physically, but in our virtual world si...