Monday, November 14, 2011

Hadoop - Still full of potential

Japan's K-Computer has 705K cores stayed on top of the Top 500 supercomputer list. Now that is scale! I am reacquainting myself to Hadoop to which I was introduced in 2006 and find it is still using stateful nodes maxes out around 3-4K and does not use network storage. Security is an after thought and it is not yet suitable for its killer application interactive behavioral processing or the future killer application real-time stock data processing. If I were to add cloud related requirement of multi-tenancy, virtualization, billing etc etc. you see how underdeveloped this system is and how big its potential is.

Expert Systems to Generative AI — tiny steps that caused giant leaps in productivity

In the beginning, we wrote the rules by hand. The expert systems of the 1980s and 90s were magnificent and exhausting. You found the best pe...