In a move with potentially far-reaching implications for the search
market, Alexa Internet is opening up its huge web crawler to any
programmer who wants paid access to its rich trove of internet data.
Alexa, a subsidiary of Amazon.com that is best known for its traffic rankings, on Monday unveiled Alexa Web Search Platform, a set of online tools for searching, indexing, computing, storing and publishing vast quantities of net data.
Alexa claims it’s the first time that developers, students and
startups will be given inexpensive access to an industrial-scale web
crawler — the same technology used by industry giants like Yahoo (Yahoo Slurp) and Google (Googlebot).
"It sounds innocuous but it’s big," said Alexa CEO Bruce Gilliat.
"We’re giving access to billions of pages and computing resources….
Users have never had this opportunity before. Big industry has ruled
search, because it was the only player with access to the tools."
Alexa spiders 4 billion to 5 billion pages a month and archives 1
terabyte of data a day. The new platform will allow developers to build
their own search engines.
"If it is what they claim it is, it strikes me that this is nontrivial news," said search industry pundit and author
John Battelle. "Anyone can crawl the web, but crawling and maintaining
an index at scale is very difficult and very expensive. They are
providing convenient access to something that was very dear."
Battelle said the move, if it pans out as promised, could have a big
impact on the search industry, and could possibly lessen Google’s
growing dominance in web search.
Alexa’s offering may help "create an ecosystem (in search) where something can occur outside the Googleverse," he said.
