In a blog post last month, Google announced that it would extend certain commitments it made to the FTC in 2012 that were set to expire relating to, among other things, the scraping of third-party content for use on certain Google “vertical search” properties such as Google Shopping. The announcement came days before the commitments were set to expire on December 27th and months after Yelp had claimed that Google was not living up to its promises by allegedly scraping Yelp local business photos for use in certain Google results (e.g., local business listings).
In 2012, the FTC investigated allegations that Google scraped content, such as user reviews and star ratings, from competing websites to improve its own “vertical” offerings, such as Google Local and Google Shopping. Unlike “horizontal” search which scours the entire web for results, “vertical” searches are more focused – such as News, Images, Shopping, Travel, or in the case of Yelp, local businesses. Following the closing of the investigation, Google had agreed to refrain from scraping online content from certain websites that focus on specific categories such as shopping or travel for use in its own “vertical” offerings (e.g., Google Shopping, Google Flights, Google Hotel, Google Local). Under the commitment – which Google has now extended — Google promised to give websites an opt-out mechanism to keep content from specified domains out of Google’s vertical search offerings linked to google.com in the U.S, while still having them appear in Google’s organic search results. As part of the extended commitment, Google reserved the right to display content that was sourced independently or to otherwise crawl and index web content to innovate in search. In a related commitment, Google also agreed to amend its AdWords terms to give online advertisers more flexibility to simultaneously manage ad campaigns on Google’s AdWords platform and on rival ad platforms.
Google’s announcement that it would extend its prior commitments to the FTC is notable, but is limited in scope to content on Google’s vertical properties. Such scraping of content and reposting should be differentiated from other issues that website owners or individuals face, namely, the desire to remove a URL from Google Search results or take down an offensive or infringing page from the web. Indeed Google has an entire help page devoted to how to “get something off Google” and Google will remove search links to content that, among other things, violates its policies, constitutes a legal violation or violates copyright.
Moreover, this announcement does not address the unwanted crawling or spidering of websites by third parties other than Google. In those cases, site owners may try to use certain technical means as well as enforceable agreements to communicate their intentions regarding crawling and scraping to third parties.