Testing the value of
AI-generated content
Highwire collaborated with a client to create a real-world test of AI-created, human-edited content. Can the machine drive value as well as volume?
Challenge
Our client wanted to see if generative AI could help close the search visibility gap between a scrappy startup and its biggest, best-resourced competitor.
Could AI-generated content be a viable and cost-effective solution?
Solution
Partnering with the client, Highwire created a controlled study to answer three questions:
- Between human and AI-generated content, which type ranks faster for target search keywords?
- Which type of content ranks higher?
- Which type of content leads to more down-funnel engagement?
Strategy
During a three-month trial period, we published content at a 3:1 rate of AI-to-human writing. We used tools like ChatGPT and Frase.io to create AI-generated content, gave each draft a brief edit, and analyzed the results.
Campaign Details
After a three-month trial, human-generated content consistently outperformed AI content.
- Which type of content ranks faster? Human content, by a factor of more than 2x.
- Which type of content ranks higher? Human, with an average rank of 16 positions higher. That’s almost two full search result pages.
- Which type of content leads to more down-funnel engagement? Human, with zero new contacts created by AI content.
Out of the 2,600 sessions generated by this content on our client’s blog in Q1 2024, 59 were the result of AI content.
Two months after publishing, AI content continued to grow at a slower rate and generated fewer sessions. While human-drafted content contributed three top-10 keywords, two top-five keywords, and one top keyword in a 60-day period in 2022, 60 days’ worth of AI content resulted in zero top 10 keywords.
In summary, human-drafted content significantly outperformed AI content in search rankings.
The experiment revealed risks in relying solely on AI-driven content, as it could lead to severe underperformance. Supporting data indicates that our result isn't an outlier; websites with large volumes of AI-generated content often suffer drops in rankings and traffic.
While Google's recent updates penalize "spammy" AI content, AI still holds value in areas like brainstorming, short-form copy, and grammar checks. However, this study underscores the importance of human creativity and expertise in crafting compelling narratives that resonate with target audiences, establishing thought leadership and in generating new ideas.