Digital marketers have a big problem. It’s one they may not even know about. It obscures the facts and confuses the big picture. It causes campaigns to be declared a flop, bonuses to be denied, awkward conversations to be had. The issue? Dark traffic.
What is “dark traffic”?
“Dark traffic” is the term given to data collected about website visitors that is mis-categorised by analytics programs like Google Analytics, as coming to the website directly rather than through another channel such as a search engine or social media post. To fully understand the issue we need to first get our heads around the main channels which drive traffic to websites:
- Organic – this is traffic that has arrived on your website through the use of a search engine but not clicking on a paid advertisement. Technically, any feature on the search results page, such as a “featured snippet” or a Google My Business listing should be counted as organic traffic. This is typically the traffic that comes to your site as a result of good SEO.
- Paid – visitors to your website who have clicked on an advert in the search results (usually denoted as such with an “ad” symbol), or who have clicked on a banner advert or a display advertisement. Essentially, your PPC or paid advertising campaigns that cost you money every time someone clicks on the advert and visits your website.
- Referral – if a visitor arrives on your website through clicking on a link on another website then this is a referral visit. Website “A” has referred the visitor to yours by way of a hyperlink.
- Direct – this is the tricky one. Essentially, direct traffic should be any visitor who arrives on the website through typing the address directly into their address bar or who has a bookmark in their browser that they clicked which took them directly to your website.
The problem with dark traffic
The confusing thing is that, in Google Analytics, in particular, the traffic that is sorted into the “direct” bucket doesn’t always comprise visitors who have typed the address in manually or clicked on a bookmark. In fact, it is the category of traffic that Google Analytics will sort any visitor into if it can’t determine its real source. You see, Google Analytics isn’t omniscient. It can’t always identify how a human arrived on your website and when it can’t it assumes they are direct traffic. It often assumes incorrectly. Direct traffic can actually consist of visitors from a variety of different sources. For instance:
- A referral from a website running on HTTP to your HTTPS website. No referral data is passed when a visitor goes from an insecure website to a secure website, and therefore, Google Analytics cannot determine which website it came from, so it lumps it in with direct traffic.
- Encrypted social. Instant messaging apps like WhatsApp are hugely popular, one of the reasons why is the end-to-end encryption they offer. This security feature, albeit great for the users, is a bit problematic for digital marketers as Google Analytics again cannot determine the source of traffic arriving on a website from a link shared through these encrypted formats.
- Non-web formats. Click on a link from a PDF or Word document and you will end on the website, but Google Analytics won’t be able to tell how you got there. Not too much of a problem you might think because who is using Word documents as part of their marketing efforts? Well, probably not many people. PDFs, on the other hand, are a different story. If your business development team is sending out case studies to potential clients, or invitations to your next event, chances are they may be in a PDF format. Wouldn’t it be great to identify how successful their efforts are?
This dark traffic is a major issue for digital marketers because it stops the credit for traffic going to the right channel and therefore leaves digital practitioners in the dark about how successful their campaigns are. And, consequently the true ROI.
Think of it like this: you are a marketer for a company who is paying a healthy budget for an agency to assist you with your organic search rankings. At your end of year review, your boss raises her concerns that SEO traffic has not increased but direct traffic has, therefore, surely it’s the off-site marketing campaigns that are having the most effect? How can you disprove this?
It’s a problem. Without this clarity in your data, it’s hard to make effective decisions. Sadly, direct traffic isn’t the only issue with misattributed data we’ve come across at Avenue. Out of the box, Google Analytics is a hugely powerful tool but it still needs some configuration to make sure you are getting the clearest picture of your traffic as possible. You see, a quick visit to the referrals section in Google Analytics paints a puzzling picture. Yahoo is in there, but surely that’s a search engine? Why are they picked up as referral sources and not drivers of organic search traffic? This is just one example.
Bringing clarity to dark data
Thankfully there is a solution to the issue of dark data. It sounds convoluted, and to a degree, it can be difficult to get your head around, but there are some quick fixes that will help shine a light on the dark traffic data in your account.
Organic search sources
The first port of call to make sure your SEO traffic is being correctly tracked is the “Organic Search Sources” which sits under Admin > Property > Organic Search Sources.
Organic Search Sources in Google Analytics
From here, you can simply add the referring domain of the search engine that is being recorded as “referral” to the form and Google Analytics should start tracking traffic from that source as organic.
Unfortunately, it doesn’t always work for every search engine. For instance, DuckDuckGo is not counted as an organic source by default, and adding it to the “Organic Search Sources” list also doesn’t work. This is despite Google confirming via Twitter that it had now been added as a default organic source, but as of our tests in February 2019 it was still being counted as direct. Another workaround is required – filters.
Using filters to reclassify traffic
Filters aren’t just for excluding your agency’s IP address from your data. They can also be used to instruct Google Analytics to reclassify data as another channel. This is particularly useful if you are finding the above “Organic Search Sources” solution isn’t working, but can also be used to capture the likes of “m.facebook.com” and “l.instagram.com” as social media sources.
To add a filter to your Google Analytics account you need to navigate to “Admin” again, but this time go to “Filters” under the “View” that you want the data to be changed to.
When you click “Add filter” you will see the toggle option for “Custom”. Select that and then, if you are looking to reclassify a search engine’s traffic from referral to organic, copy the below settings:
Filter Type: Advanced
Field A -> Extract A : Referral (enter the domain of the website you want to reclassify traffic from)
Field B -> Extract B: Campaign Medium referral
Output To -> Constructor: Campaign Medium organic
Then ensure the “Field A Required”, “Field B Required” and “Override Output Field” options are selected.
The same applies for reclassifying referral traffic as social media traffic, just change “organic” to “social media” in the Campaign Medium field. It’s important to note that this is all case sensitive. So if you use “Organic” instead of “organic” you will find your SEO traffic split into two channels in Google Analytics, which will only make the problem worse!
Filters affect data going forwards and not retroactively. Therefore, it is worth remembering that comparing your data with a time period when there wasn’t a filter will mean you won’t get a true “like-for-like” comparison. I would highly recommend marking the date in Google Analytics when the filter was added for reference in future analysis.
Example of a filter in Google Analytics to reclassify msn.com traffic from referral to organic
UTM codes
UTM codes (short for urchin tracking module codes) are appended to the end of URLs in hyperlinks to guide analytics programmes in what the source, medium and campaign any traffic passing through that link should be labelled as.
A quick way to create a UTM code to add to your URLs is by visiting https://ga-dev-tools.appspot.com/campaign-url-builder/ and follow the instructions.
(To reiterate, because this really will cause you headaches, always write the letter case of the labels as you see them in your Google Analytics account – lowercase “g” for “Google” and lowercase “o” for “organic”.)
I would recommend as a minimum you add UTM tracking codes to your Google My Business listings as clicks from these to your website should be counted as organic visitors, not direct as Google currently attributes them. Make sure you add it to other URLs on your listing other than the main website link, such as the link to your menu if you have one. This will give you greater clarity over how your Google My Business listing is used.
I need to make a public service announcement at this stage on behalf of all analytics professionals. Please, please do not add UTM codes to pages within your website. By doing this you will be corrupting the data in your Google Analytics account.
If a visitor lands on your website through organic search and then clicks on a URL on your website with a UTM code on it, their information will then be changed for the rest of their journey through the site. If their visit leads to a conversion you may not be able to attribute it back to the true channel that prompted that traffic.
UTM codes will only affect traffic that has clicked on the link with the UTM code on it. Therefore, it will not affect data that visited your site through that link before there was a UTM code.
Conclusion
To have a successful marketing campaign you need to understand how your audience is interacting with your brand and a core component of that is trustworthy data. Google Analytics is a valuable source of this data but it is imperative that you work to clean up as much “dark traffic” as possible to ensure your budgets and efforts are being allocated to your most profitable campaigns.