Brian Mark - Guitaraholic

Getting ready for the holiday season means beefing up our infrastructure around here. So, I got to order a new box for Memcached . It was even a simple server, just two dual-core processors with dual 250 gig drives from Sun running Enterprise Linux. What's fun about it, though, is that it has 32 gigs of RAM from the start (once I drop the chips into the box, anyway) and is upgradeable to 64 gigs. Now that's fun. I figure that going from the 4 gigs on one server we were running last week to 4 gigs on 3 servers we were running this week to 2 gigs on 3 servers plus 30 gigs on a dedicated box we'll be running in a couple weeks means we're going to be set for some serious growth. Now it's time to plan for a happy holiday season. At least our website shouldn't go black come black Monday.

We recently got fed up with UPS's server outages (of the 2 IP's we typically use for shipping quotes, 1 was down pretty regularly for a while). So, I decided to populate our shippingrates table with all of the data we could ever need. It all went along fine, until it got to about 10M records. At that point, our site started slowing and slowing, and what I found was the keys weren't able to be used for our standard query. It wasn't that big of a deal with 100k or even 200k records, but when I got up to 10M (it's over 30M now), that caused problems. I did an alter table and made our index include one more field, and it now sorts everything the way I wanted by default. It wasn't actually the select statement causing the problem, it was the ORDER BY portion. It was having to use a filesort, which for a huge table doesn't work real quickly. What other symptoms was this causing? For one, I was seeing the time spent waiting for IO growing. When looking at the outpu...

After a couple of "What happened" type moments in the past, we've finally decided to set up a versioning system. At this point, we're trying to set up subversion because Nick has seen good things about it. CVS seems a little out of date, and SVN seems to find strengths in CVS's weaknesses. Yet another adventure in a growing company begins today...

I reported not that long ago that we finally got a development server . Well, we've already shut that one off and moved to a new one. Apparently, that was much more of an antique than I thought it would be. We're now running a Hyperthreaded P4 with 2GB of RAM and hardware mirrored SCSI drives. I thought that the old machine would just act like our web servers under heavy load, but when it was taking 2 minutes or longer just to load a couple fields via Ajax calls, I decided that wasn't worth waiting for any more. Now, I'm going to be able to develop a little quicker... and I've got some sweet projects planned out. When I get them a little further along, I'll be sure to let anyone reading here know about them.

I've been working on configuring a development web server for toolbarn.com over the past couple of days. Was it a quad processor, quad core system with terabytes of RAM and storage? Perhaps a Sun T-2000 with more threads than you can shake a stick at? Not a chance. I'm using old hardware to simulate our web servers under a heavier load. That, and it was sitting here not being used. I commissioned a Dell PowerEdge 2300 with 768MB of RAM, dual PIII 600's, and 3 8GB SCSI drives in Software Raid 5. Not exactly high-end stuff that we'd want to run anything serious on, but for developing a website, it'll be just fine. We also set up a PhpWiki for keeping track of development notes internally among the team in IT, and we'll be doing lots of documentation over the next little while. Oh, what fun. I remembered, after getting 403 permission denied errors on all folders, that Apache requires some special tuning of the selinux parameters. Note to self: find this post ne...

I'm well into hour 13 of the site being down, but I'm finally making progress. MySQL support got me to rebuild all the tables to our last successful backup , and now I'm using bin-log files to rerun all the queries that had been run since then. With any luck, I'll have this almost ready to go around the time the call center and warehouse start showing up and needing it. Wow... this sure cut into sales. But it could have been worse, I guess. I'm not sure how, but I'm sure it could have been. I guess if none of the backups were successful, that'd qualify.

We've got a pair of replicated MySQL database servers. That's great - until the replication process corrupts both. That's the situation we're currently in. Much to my (dis)pleasure, I get to use the platinum support that we paid a bunch for. I called them and had an engineer on the phone within 15 - 20 minutes (they answered right away and had him call me back, which was awesome.) The little bit of hair I have left is getting pulled out by the minute, though. It's really not simple troubleshooting a crashed pair of servers, as we've already been working on this for 4-1/2 hours. I'm waiting for the engineer to contact me once again, so I thought I'd post something quick. This was, however, a great opportunity to upgrade to the latest version of MySQL enterprise. Oh, the joys of doing business online.

“They’re always after ‘me Open Source Secrets!” A favorite for more geeks than sheiks, delicious Linux Charms cereal features frosted oats and colored marshmallows. Tux the Penguin creates the marshmallow shapes – yellow dogs , red hats , and white boxes – with his Open Source magic. While kids love the taste, parents are happy that Linux Charms is fortified with 256 bit security, and is a good source of independent thought. It’s openly delicious! This magical Yellow Dog marshmallow, while not all that well known, has been found to be particularly tasty for those with iPod, cell phone and PS3 serving utensils. Probably the most famous of the Linux charms marshmallows is the Red Hat, which has taken the enterprise breakfast to the level of banquet in a short amount of time. White boxes have a flavor similar to that of Red Hats, but are getting to be much more plentiful due to their low cost and quality ingredients. We include White Boxes in each and every spoonful of Linux Charms ...

For many developers, running a simple web server on a windows box would simplify things a lot. I've been asked many times how to do that best. Well, my answer has been the same for a while now: use Indigo . Indigo Perl installs Apache as a service with PHP and Perl support. You can install this right to your Windows workstation and be off and running in a few minutes. No more uploading a file and saying "Oops" when there's a typo. Install it and view your site on your own PC before you FTP the files up to your web server. I've installed this on a few Windows servers as well for some Intranet apps. It sure beats manually configuring things to run Perl, since that's the language I program almost everything in. Yep, I'm old school, but that's mostly linked to my past as a Unix sysadmin. Wow... that makes me really old school.

Physical inventory. Two words I dread more than, well, anything. Ok, so "We need to talk" would scare me more. But in this virtual age, why can't I count all the bits and bytes while the people who deal with physical goods deal with the physical inventory? That'd be fair, right? Look, I can do it now. df -v Ok, so that's a bit of Linux humor, and not all that good either. But physical inventory with orders literally flying in the door is going to be interesting to say the least. But I guess it's better to have too many orders than not enough. I better get started counting. 1, 2, 3, 4...

I've never really had to work on a hosting platform I didn't have complete control over. Ok, so back in 1999 we were hosting ToolBarn on Earthlink (yes, many many moons ago), but since then we've always had root access to our servers, and it wasn't actually too long after that when we started using our own hardware. ISP's are fine for getting started (we used dedicated server options at a few ISP's before moving to our own hardware), but when you need to make a change nothing beats root level access. As I'm doing some simple changes to a site hosted on Yahoo's hosting platform, it really feels foreign. It's almost as bad as trying to work on the first site I ever worked on... maybe worse than that. I'm so used to making changes via SSH or some other server side login that even the simple things seem to take me forever. Wow... how does the rest of the world get along this way? The more I have to do on this project, the happier I am that we have ou...

Since we got our new database servers up and running, it had been about a month. I had both of the servers in sync, but because it was an emergency type of a changeover I skipped setting up replication at that time. Earlier this past week, I went ahead and started setting that up again. It didn't seem like a big of a deal at the time, but it ended up taking 2 and a half days to get the second server caught up. I guess that means we run a few queries each day, although only one slave SQL thread can run at a time which limits the amount of CPU that can be used by either of these severs for replication to about 25%. Since we have dual dual-core machines, it only uses one of the 4 available threads, making 25% the maximum. That was sort of disappointing from a performance point of view, but understandable to keep things perfectly aligned between the machines. Once it was caught up, I went ahead and set up replication the other direction as well. We now have 2 servers which can handle a...

For the past several months now, we've been using Kemp LM-1500 load balancers to split the load among multiple web servers. These are pretty simple Linux based load balancers which don't have hard drives - everything is on CF cards. Well, to make our servers even more balanced, we're going to move to load balancing our database servers and our Mini(s). Yes... load balancing replicated database servers isn't unusual. That's actually a lot of the reason that the load balancers have a second network interface. But mini-balancing? Since Google has been generous enough to offer us some cash off of a second mini, we're going to go ahead and look at buying a second one to put behind the load balancers as well. Load balancing a couple of minis will do several nice things for us. 1) If a mini fails or needs rebooted, we will still have a functional site search. 2) We can use one primarily for searching ToolBarn and one primarily for searching ToolPartsDirect, making fo...

As we're getting our new hosting platform in place, we've noticed a few issues popping up. The first larger issue was PayFlowPro from Verisign and their lack of support for 64 bit operating systems. That was easily worked around, but the next issue has me perplexed. The load balancers we're using work as NAT devices (Network Address Translation), so we have a private network behind them with all of our servers attached and the public network on the other side of them. All traffic to/from our servers goes through the load balancers, making for a firewall. To allow the web servers to talk to sites they want to visit (or request services and information from), NAT is used to allow for the public address of the load balancers to be used in the request to the servers on the other end. This works well for most things I've tried. OS updates install just fine (rather quickly over the large pipes our data center has, I might add). Payment related data works just fine (or at leas...

So what's the first thing I had to do when shown a pile of servers to install? That's right, crack them open and see what makes 'em tick. I'll start out with the web servers. Web Servers These are 3 machines that were laid out pretty cleanly. You probably can't see it, but there are 4 fans in front of each CPU. Several fans on the power supply and another across the motherboard make this one noisy server. Of course, since heat is the natural enemy of computers, I don't really care that it has more fans than all of the PC's in my office combined. I think that's a good thing. As you can see from the picture above, the memory is easy to get at, and the wiring is pretty straight forward. There wasn't even a need for an add-on board in these servers. Of course, pretty much anything failing means the motherboard needs to be replaced in a Dell, but that's the way most any name brand server is these days. Mail Server The mail server that we'll be usi...

An essential part of any balanced breakfast... ok, so these ain't no Lucky Charms. We ordered load balancers from Kemp Technologies (no, not hemp... I said Kemp) yesterday and they've already shipped. Load balancers are an essential part of any highly available website infrastructure, allowing for multiple servers to handle the requests based upon rules set up ahead of time. Example #1: Server down The first example of when someone needs a load balancer is when a server crashes. For some websites, this means they're gone. For a load balanced website, this means that the traffic starts hitting the other server(s) and the users have no idea that something happened. Example #2: Server slow Sometimes, a processes goes in an unplanned direction and uses a ton of resources. This presents a more difficult problem - the server responds as if it is up, but it just doesn't perform well. That means that the load balancer needs to watch what's going on (response times) and not ...

We ordered our database servers from IBM. These have less RAM than the Dell machines we ordered as web servers, but they have much more processor. I didn't even think about the Itaniums in the Dell's being 64 bit when I placed the order, so they were more CPU than I thought, but that's nothing compared to what I've got going on here. Base Model The base model is an Opteron powered 1U server with 1GB of RAM, 1 Gb card, and a SCSI adapter on-board. Memory Upgrade We upgraded the RAM on each of these to be a mere 4GB each. Processor Upgrade We went from the base model Opteron to dual Opteron 275's (dual-core) in each machine. Hard Drive Upgrade Databases are only as reliable as their drives, so we went with mirrored storage. Another NIC I also added in a second NIC for total redundancy (one for each switch). Warranty Upgrade IBM seems to have more confidence in their servers than Dell, because for a little less than what Dell's 3 year warranty was we're gettin...

We're trying to upgrade to a load balanced, fully redundant web serving platform now. What we currently have was a $3000 server about 30 months ago, so it's time to look at upgrades. I don't usually like Dell, but their prices are something else. As we're getting ready to upgrade our servers, I happened across Dell's website and checked what they had for a deal at the moment. We ended up getting a few at stupid prices - stupid meaning very low for what we're getting. Base Model The base model on these machines is a 1U rackmount 2.8GHz/2MB Cache Xeon with 1GB RAM, 2 x GB ethernet ports and an 80GB HDD. Nothing too impressive. Free Upgrade To start making it seem a little nicer, they offered a FREE second processor. Customization We added in 7GB more RAM, another drive (mirroring is a good thing) and a few other odds and ends including the 3 year service plan. Since it's Dell, we'll probably need that. ;-) Price Reduction Since it was over $2700, Dell took...

Ok, so not really. But, they have an easter egg in their search appliances that is sort of funny. If you click on the "About" link at the bottom of the main page, then click on the snowy or sandy looking image along the left side on that page, you end up with the image shown below. I believe they're celebrating it's completion and they're sending it off into the world, but who is this mystery googler in the picture?

I enjoy image manipulation. If I can do it via code, even better. For example, the following image will get text added to it because I'm hotlinking (using it from the ToolBarn.com web server from a different site). We're doing that mostly for the benefit of eBay users that decide to use our images. It's resulted in some visits and sales for us, and fewer sales for the eBay users. Of course, nothing is as cool as dynamically created images. When I decide to put a thumbnail of my blog on this page, I created a tool that does that. www.toolbarn.com/web-tools/ has a webpage thumbnail generator now. I've been playing with it for a while, trying to make it display as many pages as possible correctly, and it seems to be working fairly well now. I got a pop-up blocker working, it renders flash properly, and does a few other "Goodies" like timestamping the image and giving a guess as to the PR of the page in the image. Again, simple stuff, but cool none the less. I in...