Scroogle may be shutting down :(

Scroogle may be shutting down :( « on: Today at 10:01:55 AM »

Quote

---

I discovered the following just now. Very annoying, I had only started using scroogle a few weeks ago. I really don't want to go back to sucking at Google's teat...


"We regret to announce that our Google scraper may have to be permanently retired, thanks to a change at Google. It depends on whether Google is willing to restore the simple interface that we've been scraping since Scroogle started five years ago. Actually, we've been using that interface for scraping since Google-Watch.org began in 2002.

This interface (here's a sample from years ago) was remarkably stable all that time. During those eight years there were only about five changes that required some programming adjustments. Also, this interface was available at every Google data center in exactly the same form, which allowed us to use 700 IP addresses for Google.

That interface was at www.google.com/ie but on May 10, 2010 they took it down and inserted a redirect to /toolbar/ie8/sidebar.html. It used to have a search box, and the results it showed were generic during that entire time. It didn't show the snippets unless you moused-over the links it produced (they were there for our program, so that was okay), and it has never had any ads. Our impression was that these results were from Google's basic algorithms, and that extra features and ads were added on top of these generic results. Three years ago Google launched "Universal Search," which meant that they added results from other Google services on their pages. But this simple interface we were using was not affected at all.

Now that interface is gone. It is not possible to continue Scroogle unless we have a simple interface that is stable. Google's main consumer-oriented interface that they want everyone to use is too complex, and changes too frequently, to make our scraping operation possible.

Over the next few days we will attempt to contact Google and determine whether the old interface is gone as a matter of policy at Google, or if they simply have it hidden somewhere and will tell us where it is so that we can continue to use it.

Thank you for your support during these past five years. Check back in a week or so; if we don't hear from Google by next week, I think we can all assume that Google would rather have no Scroogle, and no privacy for searchers, at all.

— Daniel Brandt, Public Information Research, scroogle AT lavabit.com



P.S. Many thanks for all the support emails. A word to those who are emailing us with links to simplified Google pages with just the initial search entry box on them: Sorry, but this doesn't help us. It's the simplicity of the pages with search results that matter. Scroogle uses it's own gateway pages, but we parse out the links from Google's results page before we pass them back to you.

See the "sample" link in the second paragraph of this page, where you have to mouse-over each link to see the snippet. Results like this were not too difficult to parse. More importantly, they were stable. The query to Google that produced results like this is illustrated by the URL behind the "Next" link at the bottom of that sample (which no longer works, obviously). The parameters in URLs like that could be manipulated for language selection and number of results per page, just like for all Google searches.

Whether Google cares to restore this simple interface is a question of fundamental corporate policy. Frankly, we've always felt that we'd reach this point with Google sooner or later. Years ago it was Microsoft's decision in a different context, before web search engines even existed:

From Barbarians Led by Bill Gates: Microsoft from the Inside by Jennifer Edstrom and Marlin Eller (New York: Henry Holt, 1998), p. 117. Eller was Microsoft's lead developer for graphics on Windows from 1982 to 1995:

"Microsoft didn't want a lot of other companies writing code that could compete. It wanted to keep the barriers to entry very high. The idea, in fact, was to keep raising the bar, putting in more layers of software and APIs, which developers would then have to support. Microsoft wanted to make it so gnarly that anybody who couldn't devote a team of one hundred programmers to every Windows application would be out of the game."

Microsoft made a decision for corporate protectionism, and even today many people will never forgive them for their past arrogance. Let's hope Google doesn't do the same thing. It's not as if Google needs the money — but then, neither did Microsoft."

https://ssl.scroogle.org/cgi-bin/nbbwssl.cgi