If you ask three people for their opinion on AI, you might get as many as ten opinions. The thoughts vary depending on use such as security work being highly approved of making the internet safer. Experts have mixed opinions on using it to write code or create media such as images. Now, we’re seeing those opinions made into hard filters like the one Cloudflare is introducing in September.
The old internet had many bots we saw online that were usually divided into search engine bots like GoogleBot, Bing, or other then “bad” bots that scraped using resources without sending traffic. With the introduction of AI, we now have search engine bots mixed in with AI training bots, agents searching on behalf of users, data scrapers, and more. What should a webmaster going to do to sort through all these? That is where Cloudflare proposes a new solution.
Cloudflare: new AI traffic filters
The new Cloudflare filters that will roll out in September attempt to more carefully control AI training. They do this by offering settings blocking dual use spiders that both train AI and index for search engines. The changes also impact BingBot and others which may be important depending on your website’s audience.
Cloudflare’s controls give website owners the choice that was previously hard to setup or control. If you want to be included in the index and are okay with your site content used for AI training, you can select permissive controls. If not, you can block them accepting that you may not appear in some searches or agentic references. This can be very important for professional writers, photographers, and other content creators who would rather not have their work included in training.
Google: Search engine traffic declines
According to a recent article by NeimanLab, Google search engine referral traffic has declined so much some publishers are considering opting out of Google. Only a few years ago, that would have been unthinkable with sites doing everything possible to bring Google’s spiders to their sites in hopes of reaching a wider audience. SEO remains a big business, but Google’s place as the sole way to search has slowly been replaced with other resources such as the now famous ChatGPT. Does that mean Google isn’t worth the trouble of SEO in 2026?
For many small websites, Google remains an incredibly valuable source of search engine referral traffic along with agent referrals from AI agent searches. A hybrid strategy of allowing search engine spiders access to public pages while keeping more important content protected from training allows an agile website to remain in the index without providing the important premium content to an overview or training data set. Only human readers and visitors can access those pages. The value remains on your site instead of going to Google, Microsoft, or others. This is sometimes implemented with a login, but could be implemented with the new Cloudflare controls as well.
We hope this has helped point you to a better way to WordPress. Stay connected to us here on Eleven2 for more on making your WordPress blog better. If you have any comments, let us know below or contact us via email
© 2023 Eleven2. All Rights Reserved.