What you need to know
- Reddit has blocked Bing from crawling the social media site.
- The change took place on July 1, 2024, after which Microsoft’s AI tools had to stop crawling Reddit to index content.
- It appears that all search engines apart from Google are now unable to scrape Reddit for data, which means those search engines cannot show results from Reddit’s website.
Bing will no longer show search results from Reddit. If you navigate to Microsoft’s search engine and try to filter results to be from Reddit or simply include Reddit in a search query, you will not be presented by any results from Reddit’s website from the past week. The same thing happens if you try to get Reddit results from DuckDuckGo or other non-Google search engines.
Reddit now includes a robots.txt file that says sites are not allowed to scrape data from the website. You can check out the specific page that mentions scraping to see the code in question.
Microsoft confirmed the change in a statement to Search Engine Land:
“We respect the robots.txt standard. Bing stopped crawling Reddit after they implemented their updated robots.txt file on July 1, which prohibits all crawling of their site.”
Despite the change file, Google is able to display search results from Reddit. A Reddit spokesperson told The Verge that Bing being blocked is not related to Reddit’s partnership with Google:
“This is not at all related to our recent partnership with Google. We have been in discussions with multiple search engines. We have been unable to reach agreements with all of them, since some are unable or unwilling to make enforceable promises regarding their use of Reddit content, including their use for AI.”
Note that the spokesperson did not deny the existence of a partnership between Google and Reddit, just if the partnership is the reason for Bing being blocked. Google’s deal with Reddit was reportedly worth $60 million and provides Google with real-time access to content from Reddit.
A blessing in disguise?
Bing, and other search engines, have always been handy for searching topics on Reddit. In the past, I’ve found searching for content and adding “Reddit” to the search or specifically filtering results to be from Reddit has been effective. Search engines are often better at finding content than Reddit’s built-in search functionality.
While crawling Reddit may be useful for finding specific Reddit threads or content, Bing not having access to Reddit may be a blessing in disguise when it comes to generating responses with AI. When Google gained the rights to use Reddit content for its search engine, Google recommended people eat glue to keep cheese on a pizza. Reports that Google recommended a depressed person jump off a bridge were worrying. Google also suggested people consume rocks daily, which is not sound health advice.
Thank you to Liam (@djsonicwarrior on X) for the tip about Bing being blocked by Reddit.
