Posts

Showing posts with the label Scrapers

Scraping at an impressive speed

Image
We started a new blog. It received the first links yesterday, right after the first article was published. Less than 2 hours later, the first scraper had hit it and posted our original content on a spammy site. Impressive.

Scraping our site - for ourselves

Image
After spending some time digging around online, I've noticed a few things. 1) Scraper pages rank well. 2) Our site ranks well. 3) Links to searches on our site tend to make that page rank well. By taking a look at our search logs, it appears that people search the same way on our site that they search on Google. That's good news to me, because we use Google hardware for our search. But what's even better news is that we can capture what product they go to when they search for an item and create a link back to the search results the item was found in for each and every product on our site. So let me back up one step here. Since our product pages are dynamically generated, we can take a look at referrer data (HTTP_REFERER environment variable) and get the search terms out of the URL (if any). We can log these to a table with the brand name and the SKU, then take a look when we display the product page for any queries in that table that we'd like to show. So now, on a prod...

Toys, toys, and more toys.

Image
I enjoy image manipulation. If I can do it via code, even better. For example, the following image will get text added to it because I'm hotlinking (using it from the ToolBarn.com web server from a different site). We're doing that mostly for the benefit of eBay users that decide to use our images. It's resulted in some visits and sales for us, and fewer sales for the eBay users. Of course, nothing is as cool as dynamically created images. When I decide to put a thumbnail of my blog on this page, I created a tool that does that. www.toolbarn.com/web-tools/ has a webpage thumbnail generator now. I've been playing with it for a while, trying to make it display as many pages as possible correctly, and it seems to be working fairly well now. I got a pop-up blocker working, it renders flash properly, and does a few other "Goodies" like timestamping the image and giving a guess as to the PR of the page in the image. Again, simple stuff, but cool none the less. I in...