The limits of search neutrality

Search engines are the gateway to the internet for the vast majority of its users. They have unparalleled influence over the content and services that people discover on a daily basis, and their ability to rank results according to perceived relevance is of great value to consumers. However, search engines' ability to steer internet traffic also puts them in a unique position to potentially manipulate, restrict or otherwise alter search results for their own benefit.

The issue of neutrality has already arisen in the context of network neutrality, according to which there should be no restrictions on individuals' access to the networks that make up the physical infrastructure of the internet, and no discrimination between the types and sources of data travelling across networks. So, for example, many proponents of net neutrality oppose ISPs throttling upload and download rates, or blocking access to certain sites and services. (See Practice note, Net neutrality and Legal update, BEREC publishes final report and new consultations on net neutrality.)

Although closely related to the principle of network neutrality, search neutrality focuses on issues of equal access to the internet's navigational structure and access points. Advocates of search neutrality maintain that search results ought to be comprehensive, driven by relevance, impartial and not manipulated for commercial gain. However, the vastness of the internet and increasingly complex search algorithms mean that achieving so-called neutral search rankings is an exacting task in practice.

One problem is that relevance is an inherently personal and subjective attribute. Search engines must discriminate between websites to provide users with a list of "relevant" results; they are anything but impartial. In reality, search engines can merely perceive relevance by providing an opinion about the likely significance of search results to users' queries, and their ability to do so effectively is a key measure of their success.

There is a self-evident risk of search engines abusing their position as the internet's gatekeepers for their own commercial gain, but proving that search engines have deliberately manipulated results to their advantage will be difficult in all but the most blatant of cases.

This raises the question of whether regulators are actually in a better position to determine the relevance of a particular set of search results? Even if they were, they too would need to express a view as to why other results might be irrelevant, so shifting questions of relevancy from search engines to government-sponsored regulatory bodies.

Search neutrality advocates cite openness and transparency as one means of ensuring that search results remain "neutral", but this approach is not without its problems. Websites compete fiercely for hits and achieving higher search rankings has spawned an entire industry devoted to search engine optimisation or SEO. Imposing transparency requirements on search engines may uncover instances of search result manipulation, but it is also likely to allow spammers, and anyone else with the necessary technical know-how, to exploit search algorithms, potentially filling search results with spam blog comments, hacked websites and link farms (a group of automatically generated websites that link extensively to each other).

Regulatory scrutiny of the way search engines rank results is gaining momentum, and Google in particular has come under pressure from competition regulators. In November 2010, the European Commission initiated proceedings against Google under Article 102 of the Treaty on the Functioning of the European Union. The Commission has identified four areas of concern about Google's business practices which might amount to abuses of dominance: giving preferential treatment to its own vertical search services; copying content from competing vertical search services; exclusive agreements between Google and its partners for search advertisements, requiring the partners to obtain all or most of their requirements for search advertisements from Google; and restrictions on the portability of online search advertising campaigns (see European Commission: Vice President speech on the Google antitrust investigation). Google has recently written to the Commission with an offer of compromise which addresses the Commission's four areas of concern.

In the US, the Federal Trade Commission is also conducting anti-competition investigations into whether Google promotes search results that favour its business.

A complicating factor for the objectives of search neutrality is the fact that search engines are being placed under increasing pressure to police online piracy more actively (see further PLC Media's article post, The music industry, digital innovation and piracy). For example, since July 2011, Microsoft has asked Google to remove more than 3,000,000 URLs on grounds of alleged copyright infringement (see Google: Transparency Report). Cary Sherman, CEO of the Recording Industry Association of America (RIAA), supports search engines adopting voluntary marketplace best practices to prevent internet users being directed to sites that are dedicated to violating property rights (see RIAA: RIAA CEO to tout "transformed" music business to congressional panel reviewing "future of audio").

Search engines' unique ability to direct internet traffic dictates that some form of oversight is desirable. However, given the complexity of search algorithms and the far-reaching effect that strict regulation might have on the way the internet works, it is important for regulatory bodies to take account of the possible knock-on effect of over-regulation. For example, any large online intermediaries with popular platforms such as social-networking or mobile app sites that drive internet traffic could have claims of non-neutrality levelled against them. If the neutrality principle were applied in a strictly neutral way across the internet, it would risk regulating a substantial proportion of the internet ecosystem, potentially curbing competition and innovation in the process. On the other hand, if applied over-selectively, it risks being perceived as a device for businesses and other online interest groups to target their competitors.