Google Reveals New MapReduce Stats

An updated version of Google's paper about MapReduce (available at ACM and mirrored here) provides new information about Google's scale. MapReduce is a software framework used by Google to "support parallel computations over large (...) data sets on unreliable clusters of computers". Google uses it for indexing the web and computing PageRank, for processing geographic information in Google Maps, clustering news articles, machine translation, Google Trends etc.

The input data for some of the MapReduce jobs run in September 2007 was 403,152 TB (terabytes), the average number of machines allocated for a MapReduce job was 394, while the average completion time was 6 minutes and a half. The paper mentions that Google's indexing system processes more than 20 TB of raw data. Since 2003, when MapReduce was built, the indexing system progressed from 8 MapReduce operations to a much bigger number today.

Niall Kennedy calculates that the average MapReduce job runs across a $1 million hardware infrastructure, assuming that Google still uses the same cluster configurations from 2004: two 2 GHz Intel Xeon processors with Hyper-Threading enabled, 4 GB of memory, two 160 GB IDE hard drives and a gigabit Ethernet link.

Greg Linden notices that Google's infrastructure is an important competitive advantage. "Anyone at Google can process terabytes of data. And they can get their results back in about 10 minutes, so they can iterate on it and try something else if they didn't get what they wanted the first time."

	Aug. '04	Mar. '06	Sep. '07
Number of jobs (1000s)	29	171	2,217
Avg. completion time (secs)	634	874	395
Machine years used	217	2,002	11,081
map input data (TB)	3,288	52,254	403,152
map output data (TB)	758	6,743	34,774
reduce output data (TB)	193	2,970	14,018
Avg. machines per job	157	268	394
Unique implementations
map	395	1958	4083
reduce	269	1208	2418

{ The screenshot illustrates a Google rack from 2007. I don't remember the exact source of the image, but it's likely to be a presentation. }

Obligatory Quotes

From New Yorker's "The Search Party":

"I worry about complexity. I admire Steve Jobs. He has been able to keep his products simple." (Sergey Brin)

"Microsoft is a bit of an unusual company. It doesn't seem to like any of us being successful in the technology space." (Sergey Brin)

"How many people do you think had embarrassing information about them disclosed yesterday because of some cookie? Zero. It never happens. Yet I'm sure thousands of people had their mail stolen yesterday, or identity theft." (Sergey Brin)

"There are almost three billion mobile phones worldwide, and Schmidt expects a billion more in the next four years. If the phones use Google software to sell advertising, Schmidt thinks that over time it is 'mathematically possible for Google to become a one-hundred-billion-dollar corporation.' Two vital markets are television, which is 'easily attainable,' and mobile phones, which are 'more personable' and more 'targetable' than most advertising. To achieve this goal, Google would need to claim ten per cent of all global advertising, which now amounts to just under a trillion dollars."

"What sets Google apart, Schmidt told me in another conversation, is that although people like him always assumed that 'Google would be an important company, the founders always assumed that Google would be a defining company.' He remembers a day in 2002 when he walked into Page's office and Page started to show off a book scanner he had built. 'What are you going to do with that, Larry?' Schmidt recalls asking. 'We're going to scan all the books in the world,' Page replied. Eventually, Google began to do just that."

Headers and Footers in Google Docs

The latest version of Google Docs lets you add headers and footers in your documents. Just go to the Insert tab and click on the appropriate link. Google Docs still doesn't provide support for pagination, so you can't add any information about the number of pages in your header or footer. In fact, there doesn't seem to be a way to enter dynamic content.

In November last year, Zoho Writer added a special view for pagination and more advanced headers & footers. While the page view is still read-only, Zoho gives some idea of your document's layout. In Google Docs, the only way to see an estimation for the number of pages is to select File / Word count.

If you try to print a Google Document, you'll have another surprise: Google will export your document to PDF without mentioning that or adding a link to a PDF reader. In the print settings you finally have the option to add page numbers, but make sure there's no conflict with your document's header or footer.

An Unfulfilled Wishlist

In December 2006, I posted a list of the Google features I wanted to see in 2007 and you added 273 comments with even better ideas. Very few of your wishes were fulfilled in 2007:

* Using your own domain at Blogger.
* Ability to search feeds in Google Reader.
* An orkut redesign! Who came up with that hideous colour scheme and those pathetic-looking buttons? And the silly bevel effect added to all photos looks ridiculous!
* IMAP in Gmail
* A presentation app for Google Docs
* Google Talk on your Google Personalized Homepage
* More and better "mobile" versions of Google services
* How about the ability to open (in Docs & Spreadsheets) a Doc or Spreadsheet that is sent to you as an attachment to your Gmail account?
* GPS support in Google Maps for mobile phones
* Google Desktop for Mac and Linux
* I want Google to make their own Wikipedia.

The most frequently requested features and services were:
* GDrive (Google online storage)
* synchronization with mobile devices and Outlook for Google Calendar, Gmail Contacts
* task management (To-Do lists)
* Google Operating System (Linux-based)
* project management tools
* a lot of updates for Google Talk (Mac/Linux versions, groups, phone calls)
* Google web hosting

GDrive continues to be in the works, but it should be launched this year, according to a WSJ article. Google Calendar should add a task management feature, while the synchronization with mobile devices is currently limited to Blackberries. Google Talk will either completely move online or surprise us with some major new features; Google Operating System, project management and web hosting don't seem to be in Google's plans. The conclusion is that Google still has a lot of things to add from the wishlist for 2007.

A Search Engine from Every Collection of Links

So you've found a great web page that lists a lot of interesting links about a certain topic and now you're ready to explore them, but you don't have enough time to read everything. A good option would be to create a custom search engine, but it's too time-consuming. A faster way is to create a custom search engine on-the-fly from all the links included in the page you've found. That means you'll have a search engine restricted to all the domains, subdomains and web pages linked from your page.

For example, del.icio.us search results pages are a good place to obtain a list of authoritative sites on a topic like AJAX. Once you have the del.icio.us URL, paste it at Google's on-the-fly search engine factory and enter your query.


Other good starting points could be: Wikipedia articles, other wikis, directories, scholar papers, pages with many references etc. Maybe Google should create a separate search engine for pages that have a lot of external links related to a topic (like these pages).

Multi-Touch Interface for Google Earth

Google Summer of Code Blog mentions about an application created by Pawel Solyga "that allows you to control Google Earth using two finger gestures on [a] multi-touch table". I couldn't find too many details about TouchEarth, but it's an interesting application of the mostly-unknown Google Earth COM API.



I remember that one of the demos of Android (03:54), Google's mobile platform launched in November last year, showed a touch interface for a spinning globe.

YouTube Feeds

YouTube has never offered too many feeds and they were usually difficult to find. The new YouTube API changed that, but the feeds are still inaccessible from the interface. This why I decided to compile a list of the most useful YouTube feeds you may want to add in your favorite feed reader:

1. Search feeds
http://gdata.youtube.com/feeds/api/videos?orderby=updated&vq=mapreduce
(replace mapreduce with your query)

2. Search in a category
http://gdata.youtube.com/feeds/api/videos/-/Film/?vq=michel%20gondry&orderby=updated
(Film is the category, while michel gondry is the query - you need to replace the space with %20. Other categories: Music, Comedy, News, Sports, Autos, Howto.)

3. The latest videos from a channel
http://gdata.youtube.com/feeds/api/users/radiohead/uploads?orderby=updated
(replace radiohead with your channel)

...or only the videos that match your keywords:
http://gdata.youtube.com/feeds/api/users/radiohead/uploads?orderby=updated&vq=jigsaw

4. Feeds for favorite videos
http://gdata.youtube.com/feeds/api/users/garyferro/favorites?orderby=updated
(replace garyferro with your favorite username)

5. Your subscriptions
This is not a feed that comes directly from YouTube, it's a Yahoo Pipe.
http://pipes.yahoo.com/pipes/pipe.run?YouTubeUsername=google&
_id=28303b479f11e24199be4cafced31ad9&_render=rss&itemLimit=20
(concatenate the two lines and replace google with your username)

The feed above shows the latest videos from the first 25 subscriptions. For those who have more subscriptions, I created a new version of the pipe that shows the videos from up to 500 subscriptions:

http://pipes.yahoo.com/pipes/pipe.run?YouTubeUsername=MotherTalking&
_id=58e4f59f9e5e3282aaffdcbaf05ba68d&_render=rss&itemLimit=50
(concatenate the two lines and replace MotherTalking with your username)

6. Playlists
http://gdata.youtube.com/feeds/api/playlists/50653251EDB4E764
(the code in italic can be obtained from the playlists's URL (e.g.: Eric Speaks)