Highwire collaborated with a client to create a real-world test of AI-created, human-edited content. Can the machine drive value as well as volume?
It’s easy to understand the rabid interest in AI-generated content. The promise of high-quality words and images in a fraction of the time — and at a fraction of the cost — is a powerful lure at a time when B2B marketing spend remains relatively static. But can AI move the needle for marketing? Does it drive traffic, generate leads, or create opportunities?
As a leading strategic communications, and digital marketing agency, Highwire has a responsibility to provide the highest-quality content in the industry. As we always say, “A great story can change the world.” So, in partnership with a client, we created a controlled study to put the value of AI content under the microscope.
Rising interest in AI meets reality check
From virtual assistants like Siri and Alexa to customer service chatbots and algorithmic social media, AI is a ubiquitous part of our lives. In the last few years, however, there’s been a growing demand for AI-generated written content.
Google trends shows “AI content” rising in popularity around the time of ChatGPT’s launch, peaking in late 2023.
At first, AI content looks like the ideal solution for marketers. A simple prompt spins up advanced AI engines and produces hundreds of words on any subject in seconds.
But it would be irresponsible not to mention the potential risks of replacing human creativity with computer-generated output. Highwire’s AI framework and Risk Maps highlight these, from a lack of quality control to subjective context, nuance, and bias. As a best practice, all AI content should be edited by a human expert.
Another consideration is that AI can’t make its own conclusions; by design, it can only restate what has been said before. And in B2B marketing, establishing your brand’s unique perspective is critical if you want to be seen as a leader in your field.
It was in this context that we embarked on a controlled AI assessment.
The experiment: Determining the value of AI written content
We wanted to see if generative AI could help close the search visibility gap between a scrappy startup and its biggest, best-resourced competitor.
That content gap was huge. Our analysis determined a need to create approximately 250 more articles, taking years at the current pace of our client’s SEO program. Instead, we brainstormed an experimental solution: Increasing output with AI tools ChatGPT and Frase.io, the latter of which bills itself as “the fastest and easiest way to create content that ranks on Google.”
A crucial step was ensuring that every AI article received a human editing pass, ensuring brand safety while meeting our goal of closing the gap within the next year. We limited this revision period to one hour per article, spent spotting grammatical errors, double-checking cited sources, and ensuring that each article aligned with the client’s style guide.
The plan was to create the 250 articles the client needed to close the competitive gap in about a quarter of the time it would normally take. Every article Highwire creates for its clients is the culmination of hours of strategic planning, research, skilled journalistic writing, and thoughtful editing. Our hope was that AI could streamline this process, allowing our client to get more bang for their buck.
We sought to answer three questions:
- Between human and AI-generated content, which type ranks faster for target search keywords?
- Which type of content ranks higher?
- Which type of content leads to more down-funnel engagement?
The results: AI content isn’t a silver bullet
In order to minimize risk, we began the AI experiment with a three-month trial period. We published content at a 3:1 rate of AI-to-human writing. By the end of Q1 2024, we were able to definitively answer the three questions above:
- Which type of content ranks faster? Human content, by a factor of more than 2x.
- Which type of content ranks higher? Human, with an average rank of 16 positions higher. That’s almost two full search result pages.
- Which type of content leads to more down-funnel engagement? Human, with zero new contacts created by AI content.
Out of the 2,600 sessions generated by this content on our client’s blog in Q1 2024, 59 were the result of AI content. Two months after publishing, AI content continued to grow at a slower rate and generated fewer sessions. While human-drafted content contributed three top-10 keywords, two top-five keywords, and one top keyword in a 60-day period in 2022, 60 days’ worth of AI content resulted in zero top 10 keywords.
In summary, human-drafted content significantly outperformed AI content in search rankings.
Additional warning signs
Our experiment isn’t an outlier, and in fact there are dangers beyond under-performance for those who rely on AI-driven content alone. Data we found via the SEO tool ahrefs shows that websites which rush to publish large volumes of AI-generated content frequently experience drops in rankings and traffic.
In 2023, a popular content platform published 130 blogs in two weeks and saw a 30% drop in overall traffic — and it still hasn’t recovered from the hit. Similarly, an industrial supplier published a whopping 500 articles in 30 days and saw traffic decline by 50%. It also lost out on a number of prime first-page search result rankings.
All of the above is in keeping with Google’s recently publicized position on “spammy” AI articles that were clearly created to game the SEO process. The search giant explains how its most recent update will further suppress “low-quality, unoriginal” search results. If you’re publishing “content that only exists to summarize other content,” as The Verge put it, you’re going to find it increasingly more difficult to hit those front-page search results.
There are a number of factors contributing to this decline, both in our own experiment and others, but perhaps the most significant reason is that AI isn’t capable of coming up with new ideas. No matter which tool you use, AI-generated content can only summarize content that already exists.
You can’t establish yourself as a leader in your field if you’re not saying anything new.
The real value of AI in content
This doesn’t mean that AI has no place in content marketing. We’ve found plenty of areas where artificial intelligence can be genuinely useful:
- Brainstorming: If you’re unfamiliar with a topic or just don’t know where to start, services like Frase.io can point you in the right direction. From there you can expand on the AI’s basic conclusions with your own data and actionable insights.
- Short form copy: It’s clear that AI can’t replace human input on long-form articles, but shorter blurbs like meta descriptions and social media copy are another story. AI can’t come up with new ideas, but it can summarize your completed work.
- Grammar check: One of our most-used tools at Highwire PR is Grammarly, a cloud-based “writing assistant” with AI-supported features. Naturally, Grammarly doesn’t replace the work of skilled editors, but it can help writers throughout the draft process by making suggestions based on your own internal style guide.
To get the most out of AI technology, we have to acknowledge its strengths and shortcomings. AI may be cheaper and faster, but it doesn’t provide equivalent value. And once you’ve lost your place in Google’s rankings — and the trust of those who turn to your brand for expertise — there’s no telling if you’ll ever get it back.
To learn more about how Highwire PR can help you tell your story, get in touch.
