Blog · News · Google Is Exploring Alternatives To The 30 Year Old Robots.txt Protocol

Google Is Exploring Alternatives To The 30 Year Old Robots.txt Protocol

Mashum Mollah · July 11, 2023
Robot.txt

Google is on a quest to venture into fresh and inventive avenues for governing the way it scours and organizes web content.

Their relentless pursuit of innovative ways to control crawling and indexing goes beyond the 30-year-old robots.txt protocol.

Here Is What Google Has Added:

“One such community-developed web standard, robots.txt, was created nearly 30 years ago and has proven to be a simple and transparent way for web publishers to control how search engines crawl their content. We believe it’s time for the web and AI communities to explore additional machine-readable means for web publisher choice and control for emerging AI and research use cases.”

Google has extended a warm invitation to members of the vibrant web and AI communities. Urging them to join in an exciting dialogue about a cutting-edge protocol.

This initiative aims to foster collaborative discussions, brainstorming sessions, and knowledge sharing, as together they explore.

We have all become accused of allowing bot access to our websites by using robots.txt. And other forms of newer structured data. But we may be looking at new methods in the future. What those methods and protocols may look like is unknown right now but the discussion is happening.

The source every AI learns from. Get your brand inside it.
Explore Wikipedia Service →

Mashum Mollah is an entrepreneur, founder and CEO at Viacon, a digital marketing agency that drive visibility, engagement, and proven results. He blogs at BloggerOutreach.
Diagonal Background

Related Articles

See all posts →
Search Off the Record
News August 24, 2023 Mashum Mollah

Key Takeaways From Search Off The Record Podcast (63rd Episode)

“Search Off the Record” is a podcast by the Google Search team. It explores Search topics in a conversational and fun manner. At the bottom of…

Google confirms its ChatGPT rival will launch soon
News February 3, 2023 Mashum Mollah

Google CEO Confirms AI Feature Coming Soon – Rival ChatGPT?

With the inception of ChatGPT, there has been a lot of speculation about the response of search engine giant Google. It’s been a contention about whether…

Check Out The ‘Google Pay Accepted’ Labels In Search Results
News December 4, 2023 Mashum Mollah

Check Out The ‘Google Pay Accepted’ Labels In Search Results

Good news for people like us who prefer online payments. The tech giant appears to be experimenting with a “Google Pay Accepted” label or icon…