New SEO Experiments: A/B Split Testing Google's UGC Attribute

When Craig Bradford of Distilled reached out and asked if we'd like to run some SEO experiments on Moz using DistilledODN, our reply was an immediate "Yes please!"

If you're not familiar with DistilledODN, it's a sophisticated platform that allows you to do a number of cool things in the SEO space:

1. Make almost any change to your website through the ODN dashboard. Since the ODN is a cloud platform that sits in front of your website (like a CDN) it doesn't matter how your website is built or what CMS it uses. You can change a single page — or more likely — entire sections.
2. The ODN allows you to A/B split test these changes and both measure and predict their impact on organic traffic. They also have a feature called full-funnel testing allowing you to measure impact on both SEO and CRO at the same time.

When you find something that works, you see a positive result like this:

SEO experimentation is great, but almost nobody does it right because it's impossible to control for other factors. Yes, you updated your title tags, but did Google roll out an update today? Sure, you sped up your site, but did a bunch of spam just link to you?

A/B split testing solves this problem by applying your changes to only a portion of your pages — typically 50% — and measuring the difference between the two groups. Fortunately, the ODN can deploy these changes near-instantly, up to thousands of pages at a time.

It then crunches the numbers and tells you what's working, or not.

Testing Google's UGC link attribute

For our first test, we decided to tackle something simple and fast. Craig suggested looking at Google's new link attributes, and we were off!

To summarize: Google recently introduced new link attributes for webmasters/SEOs to label links. Those attributes are:

• rel="sponsored" - For paid and sponsored links
• rel="ugc" - For links in user-generated content (UGC)
• rel="nofollow" - Remains a catch-all for all followed links

On the Moz blog, all comments links are currently marked "nofollow" — following years of SEO best practices. Google has stated that using the new attributes won't give you a rankings boost. That said, we wanted to test for ourselves if changing these links to "ugc" would impact the rankings/traffic of our blog pages.

To be clear: We are not testing if the pages we link to change rankings, but instead the source page that hosts the link — in this case, the blog pages with comments.

Here's an example of a comment the ODN modified.

After we set the test running, 50% of blog posts had comments with "ugc" links, while 50% kept their original "nofollow" attributes.

Experiment results

We expected a "null" test — meaning we wouldn't see a significant impact.

In fact, that's exactly what happened.

If we detected a significant change, the probability cone at the bottom right would have pointed more dramatically up or down.

In fact, at a 95% confidence interval, the test predicted traffic would either fall 26,000 visits/month or gain 9,300 visits/month.

Hence, a null result.

This validates Google's statements that using the "ugc" attribute won't give you a ranking boost.

What should Moz test next?

While "null" tests aren't as fun as a positive result, we have a lot of cool A/B SEO testing ahead of us.

The great thing is we can now test out changes with the ODN, and when we find one that works, pass that to our developers to make the changes permanently. This cuts down on needless development work and stops the guessing game.

We have a Trello board set up for test ideas, and we'd love to add some community ideas to the mix. The ODN is currently running on the Moz Blog and Q&A, so anything in these site sections is fair game.

We're also looking at experiments where we use Moz data to inform these decisions. For example, a Moz Pro crawl identified that the Moz Blog titles currently use H2 tags instead of H1. Google recently indicated this likely shouldn't impact rankings, but wouldn't it be good to test?

What wild/clever/ridiculous/obvious SEO things should we test? With each good test, we'll publish the results. Leave your ideas in the comments below.

Big thanks to the Distilled Team, including Will Critchlow and Tom Anthony, for embarking on this journey with us.

And if you'd like to learn more about DistilledODN and SEO split testing in general, this post is highly recommended.