SEO for AI search and LLMs: what's working now
Something caught my attention recently: 25% of the top 1000 websites are blocking OpenAI from accessing their content. At first glance, this seems like a smart defensive move - protect your content from being scraped and potentially used to train AI models.
Let’s get this straight: these websites might be shooting themselves in the foot.
Search is changing. AI-powered search keeps taking bigger bites out of Google's pie. And as this happens, those websites blocking AI crawlers are creating an opening - a gap that other websites can fill.
Let me explain why this matters.
When someone asks ChatGPT or another AI assistant a question, it doesn't just make up answers from thin air. It pulls from its training data, yes, but newer versions also actively browse the web for current information. If your website blocks these AI crawlers, you're essentially making yourself invisible to this growing traffic channel.
An opening others will take
While major players opt out, there's room for others to step in and become the authoritative sources that AI models reference.
This isn't just about being visible to AI - it's about understanding how these systems work and positioning your content accordingly. Research shows that AI models favour content that demonstrates clear expertise, maintains consistency across platforms, and covers topics thoroughly.
How to position your content for AI search
Here's what works:
1. Write for humans, structure for AI
AI models prioritise content that demonstrates two key qualities: it needs to be thorough yet accessible. Here's how to achieve this balance:
Content structure:
Break complex topics into clear subsections
Start with a broad overview, then drill down into specifics
Add summaries, tables, and bullet points to improve data extraction
Use FAQ sections under blog posts and landing pages to directly answer common queries
Content quality:
Provide unique perspectives backed by data or personal experience
Include specific examples and case studies
Avoid fluff and focus on factual, actionable information
Update content regularly to maintain relevance
For example, if you're writing about SEO tools, don't just list features. Instead, provide concrete examples of how each feature solves specific problems, include real usage scenarios, and back up claims with data or user testimonials.
2. Build real authority
AI models, like humans, trust sources that other trusted sources reference. Some platforms carry more weight with AI systems:
Wikipedia and established knowledge bases:
Create or improve Wikipedia pages about topics in your industry
Focus on factual, well-referenced contributions that follow Wikipedia's guidelines
Include relevant citations to authoritative sources
Technical communities:
Share open-source tools or documentation on GitHub
Contribute answers on Stack Overflow within your expertise
Participate in technical forums specific to your field
Industry publications and forums:
Write for established industry blogs (for example, in SEO, target sites like Search Engine Journal or Search Engine Land)
Participate meaningfully in Reddit communities related to your field
Contribute to discussions on platforms like Quora with detailed, experience-based answers
3. Get the technical part right
Schema markup implementation:
Use Organization schema to identify your brand, logo, and official website
Implement Product schema for your tools or services
Add FAQ schema to your content to improve AI's understanding of your answers
HTML structure optimization:
Use proper heading hierarchy (h1 through h6) to create a clear content structure
Place your primary topic in the h1 tag with relevant keywords
Group related content using semantic HTML5 elements like article, section, and aside
Add descriptive captions to images and figures using figcaption tags
Technical foundations:
Ensure your robots.txt allows AI crawlers access to important content
Set up clear XML sitemaps to help AI systems discover and navigate your content
Use canonical URLs to prevent content fragmentation
Keep your domain name consistent and preferably short
Looking back to see ahead
This reminds me of when mobile browsing took off. Many websites dragged their feet on mobile optimisation, giving mobile-first companies a chance to pull ahead. The same thing's happening now with AI optimisation.
Search success isn't just about Google rankings anymore. It's about showing up wherever people look for answers - whether they're typing into a search box or asking an AI assistant.
What this means for you
The websites blocking AI crawlers think they're protecting their content. Instead, they're creating space for new voices to become trusted sources as AI search grows.
AI is already changing how people find information online. The real question is: will people find you?
Want to talk more about getting your content ready for the AI-driven future of search? Connect with me on LinkedIn or read my other articles about digital strategy and content.
