SEO: Why is a large % of organic (free) traffic not recorded in Analytics?

Google Analytics was once fantastic at showing us detailed information about our Organic (Free) traffic.

We could see how many people arrived at our website. We could see what keywords the user searched for before arriving. As well as a breakdown of other important numbers such as pages viewed, goal conversion etc. So What changed?

Google moved from http to https

Why is this important? The HTTP technical spec (RFC 2616) says when traversing from an HTTPS site to an HTTP site, a browser should not report where it was.

Another way of saying that is, if someone comes from a secure website (begins with https), their browser should never pass on details about where they came from.

This does make some sense. If you have just come from a secure website, the site or page maybe for your eyes only. Perhaps the web address itself is members only or something. The URL may also reveal sensitive information.

Unfortunately Google also moved to secure encrypted (https) search. So now we get no information about where the visitor came from before landing on our website.

Browser privacy

Another key influence effecting Analytics ability to report on organic traffic, is browser privacy settings. In todays world, privacy is high up on most peoples list. People do not like the idea of being monitored. 

Modern browsers now have enhanced privacy settings. Privacy that prevents the browser from sharing the data that is needed for Analytics. 

The organic experiment

There was a renowned SEO experiment carried out by Groupon. Groupon was convinced that the data they were seeing in Analytics, was an incorrect reflection of the traffic on the website.

So they done the last thing that anyone would do to put it to the test. They de-indexed their entire website in Google. That is to say they removed all their pages from Google's search results. Brave or what?

The idea was to see the effects on traffic if the site wasn't in Google ie no organic traffic. To their surprise, they saw that their direct traffic dropped inline with organic traffic. By a staggering 60% in fact!!!

As the site was re-indexed, the direct traffic returned inline with the organic traffic. Thus kindly proving to the rest of us, that the reported "direct" traffic was also made up of organic traffic.

We can assume that this trend of 50-60% is common place amongst most websites.