AI-Generated Content’s Report Card: Is it Actually Effective?

Brand Perception: AI Content and the “Uncanny Valley”

The “uncanny valley” is the unease people feel when something artificial comes close to being lifelike but falls short. The term was coined by a Japanese robotics professor. One theory is that it’s a survival instinct—many predators and poisonous plants use mimicry to attract prey. As a result, we might be predisposed to feel aversion towards this mimicry.

Despite our evolutionary gifts, we’re not especially good at detecting AI-generated content. A recent study out of Penn State University showed that humans might only be slightly better than a coin flip (53%) at sniffing it out.

But there might be a more important question to consider. How do a reader’s attitudes towards content change when they learn that it’s AI-generated? According to researchers at the University of Tokyo, a little more than half of the attitudes skewed negative. But it depends on context and competence.

Savvy readers who understand AI technology are generally more inclined to forgive AI-generated content if it meets their needs. Less-savvy readers might immediately retreat to a preexisting negative bias. In addition, the kind of content impacts attitudes.

Here’s a fictional (but plausible) example.

Imagine a nonprofit that offers mental health services is promoting its grief support group. They publish an AI-generated blog titled, “How to Navigate Your Grief Journey.” The blog reads a little distant, clinical, and matter-of-fact. They use an AI-generated image of someone being consoled. At the same time, the nonprofit uses AI to create a landing page for the support group, with a brief description, date/time, and a registration form.

In this scenario, the AI-generated blog is more likely to evoke negative attitudes. A reader might wonder why the nonprofit used AI to draft content about a sensitive topic. Conversely, a reader is more likely to “forgive” the landing page for being AI-generated because it provides basic, functional information about the support group. People don’t mind if a machine designs a registration page as long as it works and connects them to what they want.

Let’s do some quick math to sum it all up.

An average user detects AI-generated content ~50% of the time. When they do detect it, they have a ~50% chance of having negative attitudes. That means you have a ~25% chance of evoking negative attitudes in your audience when you publish AI-generated content.

Why does this matter? Your brand is the impression you make on your audience. It’s the total of all the subtle visual and verbal cues that say, “trust us, we’re for you.” This perception develops and strengthens over time. AI-generated content can diminish that trust. Once you lose it, it’s difficult to get back.

Final grade for brand perception: C

Simply put, a ~25% chance of diminishing brand perception is not a risk to be taken lightly.

Recommendation: A hybrid model that skews more human-generated is best. Do not publish any AI output verbatim. This is doubly important when the content is supposed to feel compassionate and relatable. And unless it’s a well-defined part of your brand strategy, we don’t recommend disclosing whether content was AI-generated or AI-assisted.

AI Content and Search Performance

A recent study by Semrush—one of the most widely used SEO tools—found that human-generated content is 8x more likely than AI-generated content to take the top spot in search results*. This also means that human-generated content is more likely to appear in Google’s featured snippets or AI overviews, as those typically come from top-ranking results.

*There’s a catch hidden in “purely AI-generated.” In this study, the content had to explicitly be labeled as AI-generated. The reason content needs to “self-identify” as AI-generated is that even the best AI detection tools are quite unreliable (and, as we’ve already established, we are too).

There are a few plausible explanations for this

1. The first comes down to time. Until 2023, SEO was built solely on the text-based craft of human copywriters. For decades, experienced content writers have collaborated with SEO specialists to build the experience, expertise, authoritativeness, and trustworthiness (EEAT) that Google prizes. Many of the top search results today built their SEO standing over time. AI-generated content, on the other hand, is still relatively new and has a lot of catching up to do.

2. The second reason is, similarly, a function of time. Good content writers have learned to avoid the penalties associated with cheap, spam-like patterns. Google can detect keyword stuffing, inaccuracies, a lack of unique insights, and repetitive passages. And their search algorithms are always improving. Google doesn’t explicitly penalize or reward AI-generated content. But they do have a problem with content slop generated at scale, whether by AI or by humans. Be very careful if you’re using AI tools to generate large quantities of content.

3. AI-generated content can be derivative. That’s simply how the models work—they regurgitate the ideas and words from similar content. This can lead to keyword stuffing, inaccuracies, copycat insights, and repetitive passages. While you might be able to give the AI output a more relatable voice with some prompting, you can’t really force it to generate novel, unique insights. So it’s hard to maximize EEAT with content that’s purely AI-generated.

Where AI might be more effective for SEO

An organization’s best-performing blogs or web pages might be 5+ years old. As part of a content refresh, AI-powered tools can help revise key pages to reflect the latest discourse on a topic.

For example, let’s say a university conducts an SEO audit across its whole system of websites. The report shows that search volume for the word “cost” has decreased, while search volume for “ROI” has increased. A university could look at its most-visited pages and blogs related to cost and tuition, pop the web copy into an AI writing assistant, and prompt it to revise the copy with an emphasis on ROI rather than cost.

Sample prompt: Revise this article so that it focuses on the ROI of a college degree instead of the cost. Emphasize job placement and long-term earning potential. Minimize the number of times that the total dollar amount of tuition is mentioned.

Final grade for SEO: C-

While AI can generate tactically sound first drafts of content, human-generated content—with unique human insights—performs better over time for SEO.

Recommendation: We recommend a hybrid model that skews heavily toward human writers. While AI-powered tools can accelerate production and optimize content, the “spin” or “angle” of your content must be unique to your organization. In addition, AI tools cannot yet conduct the deep keyword and website analytics research required to fully leverage SEO.

Does AI Content Get Clicks and Convert?

Ultimately, so much of content marketing comes down to whether or not something works. We’re talking about metrics. Open rates. Click-throughs. Renewals. Subscriptions. Registrations.

Here, the evidence is mixed. But the mix does suggest a best practice.

According to a head-to-head study conducted by LinkedIn, human-generated copy gets slightly higher click-through and conversion rates.

• AI-Generated Copy: Average CTR of 3.8%, engagement rate of 12.7%.

• Human-Written Copy: Average CTR of 4.5%, engagement rate of 15.3%.

AI-generated copy shows promise, to be sure, but the seemingly small percentage difference in CTRs and engagement can translate into thousands of conversions.

AI-generated copy can shine in small doses

Still, AI is generally a good tool for the smaller, tactical components of conversion copywriting. Email subject lines and send times. A/B testing. Paid ads on social media or Google. That’s because these components have fairly clear-cut best practices. AI writing assistants can operate like the headline analyzers of the early 2010s, but on omnichannel steroids.

In these cases, the derivative nature of AI outputs might be a good thing. It’ll (mostly) follow the optimal character counts, structure, and platform-specific recommendations. And it can produce lots of variations, which is useful when an ad placement allows for many options—e.g., search ads, Meta ads, LinkedIn ads, etc. Of course, as the LinkedIn study indicates, it’s still important to have a human copywriter do quality control and ensure brand or strategic alignment.

The promise and pitfall of AI-generated emails

Another recent study by the University of Chicago found that AI-generated emails are particularly effective for e-commerce marketing. The study’s test subject was an online wine retailer. With the help of a consulting firm, they developed a custom LLM for the experiment to closely approximate the workflow of the company’s three human copywriters. The AI-generated emails outperformed the human-written emails by about 10% in sales.

But there are four important caveats to this study. First, it only looked at email marketing. Second, the study does not identify the cost of the consulting firm and the custom LLM, which could range from $100K to $1M+. Third, the LLM integration was trained on thousands of emails written by the wine retailer’s copywriters over many years.

And fourth, this study is less relevant for mission-driven organizations—associations, universities, nonprofits, and government agencies. Selling wine to people who enjoy drinking wine is more straightforward than promoting certifications, education, philanthropy, or civic engagement. The pain point for a wine buyer is simple: “I want wine that tastes good to me and fits my budget.”

The pain points and customer journeys for mission-driven organizations are much more complex. They demand more nuanced, long-term persuasion with a distinctly human touch.

For example, the recruitment journey for prospective university students begins when they’re in middle school. Their needs and expectations evolve as they grow and mature. Strong university marketing must adapt over this multi-year recruitment journey. The same goes for associations, nonprofits, and government agencies.

Final grade for conversions: B-

AI-generated conversion copywriting provides a serviceable start.

Recommendation: A hybrid model with some human intervention can convert effectively. Strong writing teams should leverage AI-powered tools to generate variations of smaller content components at scale—email subject lines, ad headlines, UX copy, CTAs, etc. Just be sure your human writers have a clear, deep understanding of audience pain points and incorporate them into their prompts.

Final Report Card for AI-Generated Content

We give it a respectable C

It’s a powerful tool, but certainly not (yet) as plug-and-play as every tech CEO would want you to believe.

The organizations getting the most compelling results with AI-generated content have strong foundational, human-crafted ingredients required for any effective marketing. That means messaging platforms, brand guidelines, research-backed audience personas, competitor analyses, keyword research, etc.

Without these, even the strongest prompt engineers will be left ambling through an AI chat trying to refine and refine and refine.

That’s why organizations today can’t just take any platform off the shelf and expect it to dramatically improve speed, quality, and ROI in content production. Where AI tools do excel in content marketing is generating tactical copy at scale and adapting copy to different audiences.

We expect that in the coming years, as users become more skilled and platforms become more powerful, the report card of AI-generated content may start coming up aces.

For now, reach out to explore how we can help your organization make the grade when it comes to leveraging AI tools.