Monday, November 14, 2011

Hadoop - Still full of potential

Japan's K-Computer has 705K cores stayed on top of the Top 500 supercomputer list. Now that is scale! I am reacquainting myself to Hadoop to which I was introduced in 2006 and find it is still using stateful nodes maxes out around 3-4K and does not use network storage. Security is an after thought and it is not yet suitable for its killer application interactive behavioral processing or the future killer application real-time stock data processing. If I were to add cloud related requirement of multi-tenancy, virtualization, billing etc etc. you see how underdeveloped this system is and how big its potential is.

Prompt is the new Config, Workflow is the new Product

 Prompt are dynamic configurations that can adapt to incoming requests. Today configurations are static files which may miss a lot of corner...