Organic Growth Update Aug. 2024: A/B Testing & Regex
SJNF, CO
Kids*, trips, trips with kids, some much needed time to recalibrate. The feed looks weak but life has been full.
One thing I’ve learned when working on sites above the 10M pages indexed range as well as on sites nowhere near that many pages indexed but that also have much more (and higher quality) traffic is how little you want to mess around with third party, usually javascript-based A/B testing tools on organic pages.
If you’re at an early startup in the middle of finding product market fit via ads I feel like you should probably be trying to improve your product and apply some of this effort to UXR instead of iterating on landing pages anyway, but note this doesn’t apply to paid landing pages if you split yours off from your organic pages.
Anyway, a few years ago the team I was on used Adobe Target to try to add a bunch of content to some kind of major category page that wasn’t controllable from the sites’ CMS and that had been given a bummer of a folder by an architect that had probably retired 10 years before conventions on URL structures and folders were established. This company’s dev teams were pretty siloed and design teams were mostly off on their own crusades doing qualitative UXR with not a lot of metric-based directive, especially as far as landing pages went. This was an SPA, so the pages were completely javascript and were semi-able to be rendered, but a prerendering service was in place to get search bots the cached HTML.
The page got decent traffic for some hefty single-keyword searches but conversion and on-page engagement signals were trash. This was a place perceived scrappiness was a big thing, so we thought we’d just use our A/B testing tool to try to lead the user in with a little content, populate some links to other product categories, bag some conversions, reduce bounce a little bit. WCGW?
Granted, this was an ecomm site so authoritative that it could go out and rank on page 1 for competitor keywords by finessing their names into a blog title, with content objectively worse than you get out of an LLM on a first pass right now.
Fast forward about a week and the page dropped out of the index, consequently also dropped out of the brand’s sitelinks from its knowledge panel (where it was getting a lot of its traffic from, it turned out). The obvious in retrospect culprit was the SPA javascript page and the cached HTML page being different between each other, and so Google basically just nah.
Now fast forward several years and whether or not it’s considered quasi-cloaking or just bad form I don’t know, but it still happens and there’s still an algorithmic cost. The literature is clear: basically only use javascript for A/B testing small things like styling elements, subtle layout changes, button and text color.
You’ll pay the price if you don’t, and if you have no other way to get the updates you need made I’ve become a fan of product-led SEO and would have loved if some of the execs I’ve worked under in the past had this in front of them. (Marketing, engineering leadership up to the C-suite.) I’m extremely spoiled to work with a crack team of devs now, but it’s feeling more and more true that there are no shortcuts in the organic growth game, and these kind of misadventures can sting potentially for the time between Core Updates if they’re not resolved.
That’s where regex-ing testing out, especially on sites that have scale in Google’s index come in. God help you but get into GA4 if you have to or else Search Console, map out your control and variant pages to roughly 50% of your sessions/clicks and then hard code whatever new redesign, iteration, or content across all of them. ChatGPT or your LLM du jour can pretty much do all of the actual regex syntax—I was glad to read I’m not the only one who does it this way.
And if you need/want to test out changes to a single, let’s say heavily-trafficked landing page, just wait to have them built the right way and measure results on a pre/post basis, or run tests or a paid variation of the same page that you bid on in the same search results as your primary organic keywords.
Yeah it’s maybe slightly less convenient than hopping into your testing platform and getting some quick changes up, but it seems like the algorithm has gotten pretty Soup Nazi where if you step out of line once you honestly might not come back—especially if there are other anchors on your site’s performance like generally poor third party reviews or low/no overall brand sentiment on the UGC sites. Live and learn.
On a different note these are some of the more interesting times that I can remember for the industry; looking forward to revisiting the current moment in another year to see how the competitive landscape in Search is divided up, if at all.
*+ 100-pound doberman