Skip to main content


Pets and Cattle

Any pet owner will understand the pain felt by the owner of the pet that died on United Airlines. What essentially happened is that flight attendant dealt with the pets as if it were cattle. Your onboard baggage is cattle and the airline is optimized for cattle. This is not a note on UAL or pets, but on the notion in cloud computing that enterprise applications are pets and should be converted to cattle so they can leverage the cloud computing infrastructure.

The market for enterprise software is around $280B and market for infrastructure that includes servers, network and storage is roughly $60B, $40B and $20B. As cloud takes a large upfront cost hit on the infrastructure they want to host workloads that are more sticky that the ones that drive revenues today. Today's workloads are mostly transient. New workloads start on the cloud and migrate out of it as soon as they are viable (business modelwise). So, it is understandable that cloud computing giants are pushing the pets vs. c…
Recent posts

Too many Dashboards

It seems we have a dashboard for every metric and dashboards to pick dashboards. This reminds of the early 2000s when widgets were introduced and we placed a widget for every metric, data stream on the desktop. We have distributed our IT systems and that enables us to monitor everything, but dashboard is not the answer. In fact with so many dashboards I kind of miss the good old monolithic system :)

What we need is an automaton which processes these metrics and takes automatic decisions. Dashboard seems too much like a business process. What would be great is if we get a alert saying "metric reached threshold and controller took some action".. kind of like what we get from our banks when a charge is made or fraud prevented. What would be neat is if we replace a whole bunch of dashboards with a few controllers that execute some policy that was recommended real-time by ML.

One horse or 10K Chickens?

Would you like to ride a carriage pulled by 10K chickens or single horse? The answer is not easy. It is technically cool to distribute a job over many compute units and when done successfully it replaces the workload on a more hospitable economic curve (cheaper, faster, better..). However, not all jobs can be distributed and a complicated system which does distribute it over cheap compute units ends up being so complicated that focus shifts from the job to managing this system.

We just spent over 6 mos trying to distribute a webApp and came to the conclusion that it is not about the app but about the data. We need to distribute the data and have compute thread scheduled on the data. But wasn't that the whole point of OOP?

Anyways, the search for distribution introduced me to blockchain. Here is a distributed network that kind of mimics human networks. It keeps people honest and has potential. stay tuned...

Network Data Analytics for Recommendation Tool

Yeah so first Happy New Year

Can analytics data collected from the network actually be used for driving IT infrastructure sales? This is what I was thinking about when scuba diving on the coral reefs over the break. As an aside I try to think of something pleasant when scuba diving as I tend to panic and hit the button to rise to the surface.

So back to the question. The answer IMHO is 'yes'. If we collect the right telemetry of an application distributed in the datacenter, we can answer a query like "What resources (IT) are used to create this latency profile". For example, if we know that a certain install of SharePoint has a distribution of response times that are satisfactory to the administrator then a simple query like the above should give us a list of inventory that made this possible. This inventory can include switches, servers, storage software etc. Evolve this tool further and this product can also give you TCO of a infrastructure by recording this data …

Developer will own Security Operations

I have a sneaky feeling that in coming year, the developer will strike big and get operational control of security in a datacenter and enterprise as a whole. Earlier this year, I warned that this should not  happen. Read This.

But I feel now, it is too late. Here is why. We have moved from securing perimeter to interfaces and now are talking about process jails. Some folks call it micro segmentation moving to nano segmentation.  From a ops person POV this means several orders of magnitude increase in number of endpoints that he has to identify and operationalize. i.e. he cannot do it. It will have to be done by software. And the developer owns software.

Yup, software is eating the world. It just ate the security ops.

Spark - Yup it is still all bout APIs

Having spent over 6 months now reading and practicing with code snippets on the big data ecosystem, the epiphany came to me that all I was doing was learning a new API. In fact, I was learning three new APIs: RDD, DataFrame and DataSet. It wasn't so obvious when I started reading about Apache spark. See the beauty of APIs is that it speaks the language of a developer and as a developer at heart and training I can easily understand what is being said.

All the stuff I had to read to get to this epiphany about Scale, R, NumPy and in-memory databases, keeping data in CPU registers and not in L1/L2 cache was just all confusion that kept me to from getting to the core. It took six months to weed through so much garbaget to get here.

Ok, so these three APIs help you deal with data. And depending upon the data, you pick one of these. The more structured your data the more you think of datasets and dataframes over simple RDD. And that is all there is to it.

Machine Learning, Deep Learning and Streaming Data Processors

When AlphaGo beat the human last week using ML to process a small set of board positions which its computing power could process, it proved that Deep Learning ML (that which uses algorithms vs. simply data) has arrived. But can the same machine analyze a streaming set of unrelated mouse clicks to identify a "hack"? Could it have blocked the hacking of NY Fed and saved Bangladesh $100M of lost funds?

As I refresh my understanding of ML - this time with Spark MLLib. I am thinking that this use case of streaming data analyzes with ML or Deep Learning is the NBT (Next Big Thing). To run this type of computational jobs, one requires a cloud because no small cluster will do and it needs a special fabric of the network because nothing enforces better than a policy on the network. Next generation compute systems and specially memory/microprocessor arch has found its killer app in streaming data processing just like GPUs found video games. This time the games are played by hackers an…