Varnish VCL: does not exclude Google Campaign URLS

[iancassidyweb]

Description

Merchants will often use acquisition campaigns to drive traffic to their website and wish to record this information in Google Analytics.

Whilst the standard Varnish 4 VCL file produced by Magento contains exclusion rules for the 'gclid' parameter often seen in paid ad campaigns; it does not contain any exclusion for campaign urls.

This subsequently means that when a user renders a page and includes one of the standard campaign parameters Varnish will miss its cache and create a new version of the cached page.

The cache will also be missed for all variations of campaign url structures leading to an increased varnish cache size and a massively inefficient caching process.

Google have a standard campaign url structure and the builder which can be visited below:
https://ga-dev-tools.appspot.com/campaign-url-builder/

Preconditions

1. Magento 2 CE or EE Version 2.1.7
2. Varnish 4 installed

Steps to reproduce

1. Install either Magento 2 EE or CE version 2.1.7
2. Export the default Varnish 4 VCL from the admin panel
3. Apply Varnish VCL
4. Load the Magento homepage and ensure the X-Magento-Cache-Debug:HIT headers are present
5. Load the same page again by applying a random campaign url i.e utm_source=test
6. You will see the page is not cached by varnish
7. If you load the page again you will see the page has now been cached in varnish
8. Create a variation of the utm_source parameter i.e. utm_source=test2
9. When inspecting the varnish headers you will see the page has not being returned from Varnish by inspecting the X-Magento-Cache-Debug:MISS headers.

Expected result

1. As Magento comes with Google Analytics and AdWords integration the standard VCL file should contain additional exclusion rules for Google's standard campaign URL structure.

Actual result

1. Varnish will create a new copy of the page in its cache for each variant of the campaign url structure.

For Example:

Version 1 = www.mywebsite.com
Version 2 = www.mywebsite.com?utm_source=Facebook
Version 3 = www.mywebsite.com?utm_source=Facebook&utm_medium=Remarketing
Version 4 = www.mywebsite.com?utm_source=Facebook&utm_medium=Remarketing&utm_campaign=Summer2017

[iancassidyweb]

Here is what we are proposing. I thought i'd open this up to a discussion before creating a PR as we're welcome to suggestions on other approaches.

1. The first pass removes all subsequent GA tracking information allowing it to be served from Varnish

2. The clean up url section caters for the stripped parameters so were not serving things such as:

• www.mywebsite.com?&
• www.mywebsite.com?

A Suggestion

# Remove Tracking codes to allow caching
# Google Analytics
set req.url = regsuball(req.url, "&(utm_[^=]+|gclid|cx|ie|cof|siteurl)=[^&]+", "");
set req.url = regsuball(req.url, "\?(utm_[^=]+|gclid|cx|ie|cof|siteurl)=[^&]+", "?");

# Clean up url
set req.url = regsub(req.url, "\?&", "?");
set req.url = regsub(req.url, "\?$", "");

Magento 2.1 Varnish 4 VCL File

This is the standard Magento Varnish 4 VCL file which uses 3 RegEx queries to remove the 'gclid' parameter

# Remove Google gclid parameters to minimize the cache objects
set req.url = regsuball(req.url,"\?gclid=[^&]+$",""); # strips when QS = "?gclid=AAA"
set req.url = regsuball(req.url,"\?gclid=[^&]+&","?"); # strips when QS = "?gclid=AAA&foo=bar"
set req.url = regsuball(req.url,"&gclid=[^&]+",""); # strips when QS = "?foo=bar&gclid=AAA" or QS = "?foo=bar&gclid=AAA&bar=baz"

[mpchadwick]

There should be a field in the admin where you can enter the params you want to strip when generating the VCL. There are a lot more than just google parameters

https://github.com/mpchadwick/tracking-query-params-registry/blob/master/data.csv

[iancassidyweb]

@mpchadwick yea that makes sense to me the challenge with magento is their are so many permutations out their in the wild that they can only blacklist the basics.

Having the ability to extend the VCL easily from the admin would be a great feature.

On a side note, we noticed this because Bronto when sending a campaign out it contains a unique parameter in each email meaning every user that receives the email and loads the website will completely bypass varnish.

Coincidently I saw these two parameters '_bta_tid' and '_bta_c' in your list so I would imagine you've seen the same issue.

It would also be worthwhile to include a mention about this in the Varnish configuration developer documentation as I can see it being a common 'gotcha'

[mpchadwick]

Yes we've been hit by _bta_tid, mc_eid and a few of the other from that list.