Tips, Tricks, Tools & Techniques

for Internet Business, Life, the Universe and Everything

RSS Feed

Category: Search Engines

The two most important web metrics for your marketing efforts

1 October, 2007 (13:15) | Search Engines, Testing | By: Nick Dalton

I the previous article you installed two different web analytics tools on your web site. Today we’re going to use the data from those tools to optimize your marketing efforts.

We all want more traffic to our web sites. There are numerous ways to attract visitors to your web site: article marketing, blogs, social bookmarking sites, PPC, link exchanges, etc, etc. Each one has it’s proponents (along with a product that they’re happy to sell you). But how do you know which one works for you? Using real data from your web analytics tools sure beats guessing or trusting the so called gurus. Here’s how.


To know which marketing efforts are most effective we need to know where your visitors are coming from. In web analytics terms this is called the referrer.
In Google Analytics click on Traffic Sources -> Referring Sites -> Pie Chart and you’ll see something similar to this (I’ve hidden some columns for clarity):

Referring Sites Pie Chart

Here you will see which sites are sending you the most visitors. StumbleUpon is as you can see a very important traffic source and I think it requires a separate article to cover properly. (While you’re waiting for that article you can check out ProBlogger’s tips on StumbleUpon.)

If you click on any of the site names you will get a list of the actual URLs within that site that sent you the traffic:

Referral Paths

Click on a URL and you’ll get a link that says “Visit this referring link” and you’ll finally find out why this page is sending you traffic. However if you’re using StatCounter then there is a much faster way to get this information. Select Recent Pageload Activity and you will see a list of entries like this:

Recent Pageloads

The URL in green is the referrer, and if you click on it you will be taken directly to that page. Once I’m on that page I realize that I wrote a comment on Rich Shefren’s Strategic Profits blog and a lot of people are clicking through to my blog from it.

Writing thoughtful comments on other related blogs is one of my most successful ways to get traffic. According to the web analytics data above this strategy accounts for 4 of the top 10 referring sites.

But unless you’re selling ads on your site per impression then the number of page views is not the metric you’re most interested in. You should look at some of the other columns in this table ( Google Analytics -> Traffic Sources -> Referring Sites):

Referring Sites Table

For each referring site you will see how long they stuck around on your site. Social bookmarking sites like Digg and Reddit sent quite a bit of traffic as number 4 and 5 on the list but a majority of these visitors just viewed one page and then left again. You see that by a Pages/Visit number that is close to 1 and a Bounce Rate close to 100%.

Does that mean I should not waste my time on social bookmarking sites? Compared to the time it takes to write an attractive comment on another blog, just clicking on the Digg/Reddit link for one of my own blog posts is a breeze. And if just one of the many Digg drive-bys signs up for my RSS feed or purchases one of my products, it’s well worth the effort. We’ll cover how to track this in a future article.


Search engines are a special case of referring sites. What we care about here are the keywords the visitor used to find your site. In Google Analytics go to Traffic Sources -> Search Engines and select a search engine. You will then see a list like this:

Search Engine Keywords

This is not a lot of organic search traffic as this is not the primary mechanism that I generate traffic for this site. But any free traffic is welcome. So what can we learn from this list.

  1. Four of the top five keyword terms are variations of “article submission software”. Do some keyword research to determine which variation gets most search traffic and then target those specific keywords in your SEO efforts.
  2. People type in the keywords “article submission” into search engines a lot more frequently than the others, and that is why it’s number two on the list. But this site is not really about article submission in general so most of these visitors bounce away quickly in search of something more relevant. This is reflected by the low number of Pages/Visit and the high Bounce Rate. In this case you have the choice of bulking up your site with relevant content for these keywords, or just ignoring that traffic.

An important factor of how much organic search traffic you’re going to receive is of course how well your pages rank. In StatCounter this is very easy to see. Go to Recent Keyword Activity and you will see a list like this:

Recent Keywords

Click on the link in the Query column and you will be taken to the search engine results page. For this particular keyword phrase it turns out that I’m #5 in Google. Cool!

Data is Useless…

…unless you take action on it. For the simple action of installing a web analytics tool on your site you will received all this useful data described in this article. Now you need to make use of it.

  1. Determine which traffic sources send most visitors to your site. Focus on these for your future promotion efforts. Don’t waste any more time on methods that do not result in any significant traffic.
  2. Examine the source pages that are sending you traffic. Identify patterns that work well, e.g. a brief comment with an explicit link to your site, or maybe a simple trackback. Look at headlines; what works what doesn’t. Do more of the “what works” category.
  3. Look at the keywords that are resulting in organic traffic. Are you already targeting these keywords? If not, do some keyword research to determine if they could generate enough traffic to be worth targeting.

Browser toolbars reveal more than you think

27 August, 2007 (06:34) | Search Engines, Security | By: Nick Dalton

All the major search engines provide toolbars that you can download and install in your browser. Each toolbar has some nifty features that are commonly not found in browsers, which makes them compelling enough to download and install. One feature of all toolbars is to be ale to search the web using the search engine that made the toolbar. This is of course the reason for the toolbar’s existence: to funnel more searches to the search engine.

Another common “feature” of search engine toolbars is to report home about each web page that you visit. Even though you can in most cases turn off this feature, the toolbar offers some compelling extra benefit so that most users keep it enabled. (Or they are just unaware of the “call home” feature.)

If we for the moment disregard the privacy aspects of reporting every web page that you visit, there is another implication that most web site owners are not aware of: The web pages reported by toolbars are fed into the search engine’s web crawler. (I don’t have prof that this is the case for all toolbars, but I know it’s true in at least one case. And that’s enough to cause trouble for web masters.)

What’s the problem with that, you say? One example could be that you’re working on a new web site that is not quite ready to be public yet. And you haven’t bothered to password protect it during the development. Who is going to guess your new domain name anyway? As you’re busy developing your site, the toolbar sends the URL of every page – finished or not – to the search engine.

Another, perhaps more serious, example is the thank you page of web sites that sell digital products. When you – or anyone of your customers – goes to the thank you page, the toolbar reports the URL to the search engine. If you don’t have any additional protection on the thank you page it will be included in the search engine index. Then when a potential customer uses that search engine it’s possible that your thank you page shows up in the search results. And it’s very likely that the person searching was looking to buy your product. But now, with direct access to the thank you page the potential customer can download it for free. You just lost a sale.

If you have good web analytics it may be possible to see these direct accesses and calculate how much money you’re loosing. But it’s also very likely that the search engine has cached your page, and possibly even the product download itself. In that case you will never even know that your product was downloaded without payment.

My Digital Security Report has advice on how to protect your digital products from overzealous search engine toolbars.


13 August, 2007 (22:02) | Search Engines, Security | By: Nick Dalton

Back in the days around 3 B.G (Before Google) AltaVista was the new search engine on the block. In an effort to show off the power of their minicomputers, the AltaVista team at Digital decided to crawl and index the entire web. This was at the time a new concept. Many web masters didn’t relish the idea of a “robot” program accessing every page on their web site as this would add more load to their web servers and increase their bandwidth costs. So in 1996 the Robots Exclusion Standard was created to address these web master concerns.

Using a simple text file called robots.txt you can instruct web crawlers (a.k.a. robots) to stay out of certain directories. Here is a very simple robots.txt which disallows all robots (User-agents) access to the /images directory.

User-agent: *
Disallow: /images

By disallowing /images you are also implicitly disallowing all subdirectories under /images, such as /images/logos and any files beginning with /images such as /images.html.

Curiously there was no “Allow” directive in the first draft of the standard. It was added later, but it’s not guaranteed to be supported by all robots. So anything that is not specifically disallowed should be considered fair game for web crawlers.

To disallow access to your entire web site use a robots.txt like this:

User-agent: *
Disallow: /

If User-agent is * then the following lines apply to all search engine robots. By specifying the signature of a web crawler as the User-agent you can give specific instructions to that robot.

User-agent: Googlebot
Disallow: /google-secrets

Since the original spec was published several search engines have extended the protocol. One popular extension is to allow wildcards.

User-agent: Slurp
Disallow: /*.gif$

This prevents Yahoo! (whose web crawler is called Slurp) from indexing any files on your site that end with “.gif”. Keep in mind that wildcard matches are not supported by all search engines so you have to preface these lines with the appropriate User-agent line.

You can combine several of the above techniques in one robots.txt file. Here’s a theoretical example.

User-agent: *
Disallow: /bar
User-agent: Googlebot
Allow: /foo
Disallow: /bar
Disallow: /*.gif$
Disallow: /

This would result in the following access results for a few URLs:

URL Googlebot Other robots Allowed Allowed Allowed Allowed Allowed Allowed Allowed Allowed Allowed Allowed Blocked Allowed Blocked Blocked Blocked Blocked Blocked Allowed

Computer programs are pretty good at following instructions like these. But for a human brain it can quickly get overwhelming, so I highly encourage you to keep it simple. One of the longer robots.txt files I’ve encountered is from – it’s over 300 lines long. The site owner Aaron Wall is the author of the excellent SEO Book; he knows what he’s doing.

For us mortals there is a robots.txt analysis tool in Google’s webmaster tools. Highly recommended. Another good resource for more information on the Robots Exclusion Standard is

Today when companies are spending a lot of money to be included in search engine listings, the idea of excluding your content may seem quaint. But from a security perspective there are many valid reasons for limiting what a search engine indexes on your site. See my Digital Security Report for more information.

What is the difference between on-page and off-page ranking factors?

19 February, 2007 (11:53) | Search Engines | By: Nick Dalton

On-page ranking factors are, as the name implies, limited to what you see on the web page itself (the HTML). Examples are: does your keyword appear in the page title, in an H1 headline, what is the keyword density for the whole page or sections of the page, what is the total length of the page.

There are dozens of software programs that will compute on-page statistics. Most SEOs will use such a program and then apply their experience, magic or crystal ball to come up with recommended changes that they expect will improve the ranking for your page. RaSof is the only software that I’m aware of that uses real statistical data to score on-page ranking factors. (If you know of any others please leave a comment here. I’d love to do a review.)

The best part of on-page ranking factors is that you are in full control over them. Make a change, wait for the search engine to pick up the change and see your page ranking change. You can make random changes or changes based on the latest fads in SEO forums and see what happens. Just like a monkey randomly hitting keys on a typewriter will eventually write Hamlet, you will eventually hit upon the right combination of on-page ranking factors. I would of course recommend the rApogee / RaSof combination instead.

One of the off-page ranking factors that many people do not consider is the complete URL of the page: the domain name, any directory names, the page name, the page name extension. Since the URL is something you do not want to change after you’ve launched your web site or written your blog entry, this requires some upfront planning. Nemeas is a tool that will help you with this.

Links is the off-page ranking factor that most people pay attention to. I’m still looking for a good tool that will evaluate links in a way that is similar to what search engines do. Until I find one (or write one) here’s a simple way to at least get a quantitive measure:

  • Google and MSN – Enter “link:” before your web page in the search box, e.g.
  • Yahoo – Use the same “link:” syntax as above. This will take you to Yahoo Search Site Explorer and show the results there.

Which of the three types of ranking factors should you pay attention to? Of course the answer is all three. But which one is most important? According to James Brausch who compiled the ranking factors data, it varies slightly by search engine. On average it roughly breaks down this way:

  • On-page factors – 40%
  • URL factors – 20%
  • Link factors – 40%

Can rApogee help you with RaSof?

15 February, 2007 (16:35) | Search Engines, Tools | By: Nick Dalton

James Brausch writes: RaSof Price Too Low?

This blog post is both a lesson about pricing… and a warning that one of my most popular services will have it’s priced increased by 10 times at noon tomorrow.
The current price is only $100/month. That price is drawing in tons of very poor customers. The refund request rate is once again at 75%. That service doesn’t even offer a guarantee. There have been four denial of service attacks against the web-site. It’s not a good thing.

I’m writing this blog entry so you recognize the problem when you have it with one of your own products. Many would be tempted to simply withdraw the product from the market. It seems obvious that customers are not happy with it. That’s not the case. I’ve seen this before and I’ll probably see it again. If I’m wrong this time, I’ll post here and give an update… but I’m almost certain this is simply a case of pricing the product too low.

In the spirit of unabashed self-promotion I would like to invite you to take a look at rApogee – a presentation and organization companion for RaSof. I developed it for myself to help me better understand and make use of the Ranking Factors data and the results produced by RaSof. I’m making it available commercially so that others may gain the same benefits as well.

Please note that rApogee does not include RaSof or any Ranking Factors data. You have to purchase the RaSof service directly from James Brausch. (Do it before noon tomorrow!)

Discover How To Present RaSof Data In A Powerful Way To Dramatically Increase Your Understanding Of The Ranking Factors Data

11 February, 2007 (17:42) | Search Engines, Tools | By: Nick Dalton

RaSof is a very powerful tool, but the 1,500 lines of raw output are somewhat daunting for users who are not very familiar with the Ranking Factors data. rApogee takes this raw output and organizes it in a way that you can easlily find sections where you need to improve, shows you which small changes will increase your score, and along the way explains each piece of data.

See the screenshot below for some of the main features of rApogee:

  1. Present the RaSof data in expandable/collapsible groups for a better overview of the data.
  2. Easily compare RaSof data for multiple URLs.
  3. Automagically compare your URLs with the top 10 search results from the selected search engine.
  4. See ranking data for other values for guidance on how to improve the RaSof score of your page.
  5. Hover over data cells to get explanations of what the data means.

rApogee Main Screen Image

Using rApogee is extremely easy. If you have ever used Microsoft Excel; you are already a power user. These two screencasts illustrate, step by step how rApogee works: video1, video2

rApogee is written in Excel and it requires that you have Excel 2000, or later. The advantage of having all your data in Excel is that you can use the full power of Excel to further analyze, organize and track your results.

In addition to Excel you also need a current subscription to James Brausch’s RaSof.

Read more about rApogee.