Brian Mark - Guitaraholic

Today, we had an XML process change so invalid XML was being returned (or changed invalid XML, because it's always invalid, but we had a fix for the prior version). Instead of just affecting us while the bad data was being returned, it actually lasted a little while longer. Why? Because we use memcached to store the result, lessening the load on our inventory management system. While it isn't normally an issue (most people don't deal with this oddball system), if we were storing those results longer than we had been, this could have been a huge issue. It was rendering product pages non-functional, which isn't a good thing. As a result of this, we're going to start doing more error checking and trapping. Oh what fun it is to deal with systems that don't care about being valid in the slightest and having Perl modules that require it.

Once again, we've made lots of changes at ToolBarn. But now, I'm looking for some feedback. We've got a cache on our local servers of what most of the competing sites are priced at on every product we offer. If we show that, similar to Progressive Auto Insurance, are we asking for trouble? Two thoughts on why we should : 1) Establish ourselves as a destination. 2) Give our customers less reason to leave our site and look around. Two thoughts on why we should not : 1) We may not always have the lowest price. 2) We've got more traffic than most of these sites, so we'd be building their brand for them. Such a tough choice. We'll probably run an experiment at some point and see what it does. What do you think?

Anyone remember when VRML was being touted as the "2D Web Killer"? I do. I laughed back then... of course, it was more because I thought nobody would take the time to learn it and that it'd be tough to get WYSIWYG web designers to ever consider. Well, here we are years later. People are still programming flat (although 2.0 sites try lots of 3D shadows and the like) websites without that whole 3D interactive world. Granted, in 1998 I did play around with some 3D interactive stuff like this , but nothing like some of the VRML sites that came out that you could walk around to different buildings and the like. While I still think there's a place for that (Second Life seems like a possibility, but it's much more flexible in an app you run), I don't see it happening any time soon. Oh, and this post can be credited to my bookcase, where I noticed one of the titles "Java, CGI, VRML and SGML". I think my wife got that one for a web development class once and ...

I started developing a mobile site for our local contractor customers yesterday. We'll see how it does, but so far I've managed to find a neat tool to view what it'll look like. Actually, you can view any site in it as it would be viewed in Opera Mini . For someone developing a site and not wanting to use a mobile device constantly to test it, this is great. Now, I just need to figure out what I want to show them for sure.

I've been rather busy lately. We hired some new programmers at work, which have both been working out great. We're all excited at work, because I'm finally going to have some time to look at strategy again instead of being caught up in details. I've also been busy with the podcasts. They were going well enough that we now have 2 podcasted shows each week on WebmasterRadio.FM (Monday and Wednesday at noon central), followed by a live show on Thursdays. Of course, this means I want to get the process down so it sounds as professional as possible, but it's really tough when trying to get that many done along with having a full time job and two young boys that want to be a part of my life. Top that off with a PC that crashed and it's tough doing much else, so my blogs have suffered. I'm not gone, this blog isn't dead, I'm just busy right now. I'm hoping that all settles down someday, but it probably won't.

We recently got fed up with UPS's server outages (of the 2 IP's we typically use for shipping quotes, 1 was down pretty regularly for a while). So, I decided to populate our shippingrates table with all of the data we could ever need. It all went along fine, until it got to about 10M records. At that point, our site started slowing and slowing, and what I found was the keys weren't able to be used for our standard query. It wasn't that big of a deal with 100k or even 200k records, but when I got up to 10M (it's over 30M now), that caused problems. I did an alter table and made our index include one more field, and it now sorts everything the way I wanted by default. It wasn't actually the select statement causing the problem, it was the ORDER BY portion. It was having to use a filesort, which for a huge table doesn't work real quickly. What other symptoms was this causing? For one, I was seeing the time spent waiting for IO growing. When looking at the outpu...

We recently got SpamAssassin set up with a ruleset that seems to be working fairly well. Not only has this made my inbox much nicer to look at, but it has also created a new game in our office. We now have the "Spam Olympics". What is that you ask? Well, each piece of spam is given a score by the SpamAssassin judging bots. We just watch those scores, looking for the top marks from the algo. Initially, we were looking at 30 somethings as being good, but it quickly moved into the 40 range, and we had a few break 60 over the weekend. Not exactly the most productive game, but at least it makes the SPAM worth double checking for false positives. I wonder how many other offices partake in the "Spam Olympics"...

I tried to take a test tonight. To begin with, the testing platform of choice was Microsoft Word. The background came in a "beautiful" aqua color. You know, this really bad color . Secondly, I had forgotten all of my books at work. That left me feeling a little "disoriented". It's amazing what not having your usual reference materials does. Third, the PC I was using was having some issues. The mouse batteries died halfway though, so I was using a lot of alt+tab, ctrl+tab, and for a new search, I found it simplest just to open a new tab in FireFox. Add in telephone calls, a couple of IM interruptions, and children - well, it just wasn't a good night for a test. Oh, and I didn't get much sleep last night after playing tennis late and then having to get up early for meetings most of the day. I must be ready for a vacation... now that I feel like my brain cells have blown away like seeds of a dandelion.

With everyone being so hot to use Ajax for things now, it seems that people forget how much simpler things can be done if you just stop and use some old-school techniques. I know I like Ajax for some things, but if it isn't necessary, then why introduce the extra bit of complexity? I'm speaking here about the 204 status code. Quoting the W3 204 No Content The server has fulfilled the request but does not need to return an entity-body, and might want to return updated metainformation. The response MAY include new or updated metainformation in the form of entity-headers, which if present SHOULD be associated with the requested variant. If the client is a user agent, it SHOULD NOT change its document view from that which caused the request to be sent. This response is primarily intended to allow input for actions to take place without causing a change to the user agent's active document view, although any new or updated metainformation SHOULD ...

After a couple of "What happened" type moments in the past, we've finally decided to set up a versioning system. At this point, we're trying to set up subversion because Nick has seen good things about it. CVS seems a little out of date, and SVN seems to find strengths in CVS's weaknesses. Yet another adventure in a growing company begins today...

I've discovered something this week that I'm working on fixing. What I discovered is this: real-time lookups aren't a good thing. For years now, we've done real-time lookups whenever someone adds a product to their cart and enters a zip code. This allows us to not worry about rate changes and focus on the products. But what it's also meant is we're leaving money on the table. A quick test over the past few days has shown that nearly 10% of the time, UPS's servers aren't returning data. That doesn't seem good to me. There are days where our customer service reps complain about shipping rates taking a long time to load. Those are the times I knew about. But the 10% figure is even without taking the "Nothing returned" into account - this is just "Something useless returned". So, as a fix, I've queried enough to get all the rates now, and I'm rewriting our shipping calculations to run entirely without querying their servers. W...

I reported not that long ago that we finally got a development server . Well, we've already shut that one off and moved to a new one. Apparently, that was much more of an antique than I thought it would be. We're now running a Hyperthreaded P4 with 2GB of RAM and hardware mirrored SCSI drives. I thought that the old machine would just act like our web servers under heavy load, but when it was taking 2 minutes or longer just to load a couple fields via Ajax calls, I decided that wasn't worth waiting for any more. Now, I'm going to be able to develop a little quicker... and I've got some sweet projects planned out. When I get them a little further along, I'll be sure to let anyone reading here know about them.

I've been working on configuring a development web server for toolbarn.com over the past couple of days. Was it a quad processor, quad core system with terabytes of RAM and storage? Perhaps a Sun T-2000 with more threads than you can shake a stick at? Not a chance. I'm using old hardware to simulate our web servers under a heavier load. That, and it was sitting here not being used. I commissioned a Dell PowerEdge 2300 with 768MB of RAM, dual PIII 600's, and 3 8GB SCSI drives in Software Raid 5. Not exactly high-end stuff that we'd want to run anything serious on, but for developing a website, it'll be just fine. We also set up a PhpWiki for keeping track of development notes internally among the team in IT, and we'll be doing lots of documentation over the next little while. Oh, what fun. I remembered, after getting 403 permission denied errors on all folders, that Apache requires some special tuning of the selinux parameters. Note to self: find this post ne...

I've seen requests for this type of file on a few forums. Well, I just happen to have a pretty large list of those cities and states at my disposal, so I'm posting it on HTMLMonitor.com for anyone to download. If you want any more data to go with it, just let me know. Otherwise, enjoy. US City and State CSV File

I'm starting to work on developing an AJAX-style checkout process and came across something pretty cool. It uses a combination of a server-side Perl script, a Perl module, some JavaScript and a simple HTML form. The first thing I had to do was install the Mail::CheckUser module. The installation on all 5 servers I tried it on didn't want to work via normal methods, and 'make test' always came back saying it had errors, but doing a 'make install' let it install and it works just fine, even with all of the errors reported by the test step. The next thing to do was create a real simple Perl script to check it out. I simply called it validate.pl (since it's part of a form validation routine) and loaded it to a web server with the correct permissions, then replicated it across the rest of our servers. The actual work is done by the following chunk of code. #!/usr/bin/perl print qq(Content-type: text/plain\n\n); use CGI qw (:standard); checkEmail(); exit; sub che...

So we've decided to start looking at new inventory management systems. We're going to take a look at the advantages / disadvantages of custom as well as the good and bad of many existing software. We haven't decided if we're moving to something new or not, but it seems like if we're not seeing what's out there we could be missing some great opportunities. Anyone with suggestions of software to look at, let me know. Comments are fine... just try to provide a URL so we can take a look at it. We're well beyond something like Quickbooks, and we realize that this probably isn't going to be cheap. But if we don't look, we won't know what we are or are not missing.

With our Mini, I have been toying with some ways to search for substrings. What I'm finding is that dashes, dots, etc. don't allow for a substring match. However, there is one interesting observation that I have to note here. The first special character is dropped. That's right, be it a dash or a dot, the first one is dropped and ignored. Even when searching, the same behaviour occurs. What does this mean? How can I use it to my advantage? Well, let's take a look at the UB181 example. If I were to make 2 variations of that in our "Additional Keywords" field (only shown to our Mini), simply making ub-181dz and ub181-dz (adding a dash where a letter meets a number, which is a simple regular expression) will not allow these to match a search for 181dz or ub181. Instead, I'd have to make ub-181-dz for a match of ub181 and u-b-181dz for a match of 181dz. That seems like an odd way to have to handle this, but again - simple regular expressions to make this happe...

So I have some major beefs with the Mini and how Google's algo handles stemming / substring matches. This just isn't working right. For example, one of our products is the "UB181DZ". People commonly search for the "UB181", since this is the base model. Well, the default results were 0 matches. That's not good, especially when we have the UB181DZ in stock, along with the DZK and the DZK-2 (we "invented" these kits by customer demand.) So, a simple fix is to pre-fetch substring matches of any skus. I'm doing a "SELECT sku FROM products WHERE sku like '%$searchstring%';" and sending the search as "ub181 OR ub181dz OR ub181dkz OR ub181dzk-2" for now, but that's just not a good long-term solution. It also doesn't work when they search for multiple words. I'd have to make some funky syntax to get that working, and I'm feeling lazy right now. As an experiment, I thought we could try dashes. The first att...

We bought Google. Ok, so it was just a mini , but it's cleared up my understanding of Google a bit. Upon setting it up, nothing really great came to mind. However, after doing some tweaking (it's powering the search on toolbarn.com right now), it has become clear that the more I learn about this box and it's capabilities the more I understand Google. For starters, I ended up having to cloak some pages to our mini to get our results to come out right. A search for makita drills gave me results of milwaukee drills as well because of our breadcrumb navigation and the cross-linking. Every page on our site was returned for power tools because it's in the main navigation. Some searches return poor results, such as makita 5000, which several people have searched for. I have a temp solution in place for that. So, after playing with it and then sitting back to think about how it works / serves results, I figured something out that may end up being priceless. Searches done on our...

So I started on a site that is going to do some manipulation of MP3 files dynamically, taking a preview out of the song that should be sort of representative of the song. I toyed with a few different ways of pulling out the section I wanted, looking for large dynamic changes, widest frequency range, flattest frequency pattern, breaks in the song, etc. I also tried a few different methods of making the snip. Here's what I found. 1) Regardless how cool it is to be able to detect changes in dynamics, frequency ranges, patterns, breaks, or anything else it just doesn't mean that the section will be representative. 2) MP3 players are fairly bullet proof and don't mind some abuse. There isn't any need to split properly across frames within the file - MP3 players (all that I've tested in Win and Linux) all handle improperly split files just fine. 3) Bitrates don't translate perfectly. Just because you know how many bits per second a file is doesn't mean you can cli...