Brave Sells Search Index To AI Companies

Brave, the company behind the Brave browser and Brave Search, is facing a lawsuit from the New York Times over its practice of selling its search index to AI companies. The lawsuit claims that Brave copied over 30 billion webpages without identifying itself or providing website owners with a mechanism to block its crawler.

According to Search Engine Journal, Brave sells access to its search index through a product called the Brave Search API, which is used by some of the biggest names in AI. The index holds over 30 billion pages, with millions of new pages indexed each day.

The New York Times complaint names five AI companies that Brave supplies, including Anthropic, Perplexity, and Cohere. However, none of these companies are being sued. The complaint also points to evidence that Anthropic's AI assistant, Claude, may have used Brave Search in the past.

Brave's crawler does not have a name, which makes it difficult for websites to block it without blocking Google. The company's help page explains that its crawler follows Google's rules and does not advertise a differentiated user agent. This means that websites can either allow both Google and Brave to crawl their site or block both.

Websites can check if their pages are in Brave's index by using the site operator on Brave Search. The index is a stored copy of webpages, not a live look at the website. The lawsuit may have implications for websites' control over their content and how it is used by AI companies.

A court will decide whether Brave can sell its index to AI companies, which may set a precedent for other search engines and AI companies. The outcome of the lawsuit may impact how websites are able to control their content and how it is used by AI companies.