Twitter Ad Revenue Play: Now Allowing More Search Engine Indexing

Twitter has updated their robots.txt file to allow search engines to crawl more of the site. Specifically, Google, Bing, Yandex and others will now have access to Twitter’s search results pages, which display Promoted Tweets where available.

The modification was first noticed by The Sociable, who offered a look at Twitter’s robots.txt file from September 11th:

It now says:

Allow: /search
Disallow: /search/users
Disallow: /search/*/grid
Disallow: /*?
Disallow: /*/with_friends
The change will allow search results pages to show up in Google’s index, though user search is still disallowed. For example, the results page for a search on the Olympics becomes indexable and therefore could show up in Google search results to users seeking information on the event.

With the new indexing direction, searches for keywords like “Ford” could potentially direct Google/Bing searchers to Twitter’s internal search for the term. This is different from the way Real-Time Search operated in that users were apt to see actual tweets on the topic displayed in Google’s results. They would see Google’s ads on the search results page, then click on a Twitter result and go straight to a user page, where there are no ads. With this change, the results appearing in search engines would direct the user to a Twitter search results page, which displays a Promoted Tweet embedded in the feed (when available).