One horse or 10K Chickens?

Would you like to ride a carriage pulled by 10K chickens or single horse? The answer is not easy. It is technically cool to distribute a job over many compute units and when done successfully it replaces the workload on a more hospitable economic curve (cheaper, faster, better..). However, not all jobs can be distributed and a complicated system which does distribute it over cheap compute units ends up being so complicated that focus shifts from the job to managing this system.

We just spent over 6 mos trying to distribute a webApp and came to the conclusion that it is not about the app but about the data. We need to distribute the data and have compute thread scheduled on the data. But wasn't that the whole point of OOP?

Anyways, the search for distribution introduced me to blockchain. Here is a distributed network that kind of mimics human networks. It keeps people honest and has potential. stay tuned...

Network Data Analytics for Recommendation Tool

Yeah so first Happy New Year




Can analytics data collected from the network actually be used for driving IT infrastructure sales? This is what I was thinking about when scuba diving on the coral reefs over the break. As an aside I try to think of something pleasant when scuba diving as I tend to panic and hit the button to rise to the surface.


So back to the question. The answer IMHO is 'yes'. If we collect the right telemetry of an application distributed in the datacenter, we can answer a query like "What resources (IT) are used to create this latency profile". For example, if we know that a certain install of SharePoint has a distribution of response times that are satisfactory to the administrator then a simple query like the above should give us a list of inventory that made this possible. This inventory can include switches, servers, storage software etc. Evolve this tool further and this product can also give you TCO of a infrastructure by recording this data overtime and using it for TCO calculation. Telemetry needs to meet AI/ML for this to happen.


So the question is why isn't anyone doing this?

Developer will own Security Operations

I have a sneaky feeling that in coming year, the developer will strike big and get operational control of security in a datacenter and enterprise as a whole. Earlier this year, I warned that this should not  happen. Read This.


But I feel now, it is too late. Here is why. We have moved from securing perimeter to interfaces and now are talking about process jails. Some folks call it micro segmentation moving to nano segmentation.  From a ops person POV this means several orders of magnitude increase in number of endpoints that he has to identify and operationalize. i.e. he cannot do it. It will have to be done by software. And the developer owns software.


Yup, software is eating the world. It just ate the security ops.

Spark - Yup it is still all bout APIs

Having spent over 6 months now reading and practicing with code snippets on the big data ecosystem, the epiphany came to me that all I was doing was learning a new API. In fact, I was learning three new APIs: RDD, DataFrame and DataSet. It wasn't so obvious when I started reading about Apache spark. See the beauty of APIs is that it speaks the language of a developer and as a developer at heart and training I can easily understand what is being said.

All the stuff I had to read to get to this epiphany about Scale, R, NumPy and in-memory databases, keeping data in CPU registers and not in L1/L2 cache was just all confusion that kept me to from getting to the core. It took six months to weed through so much garbaget to get here.

Ok, so these three APIs help you deal with data. And depending upon the data, you pick one of these. The more structured your data the more you think of datasets and dataframes over simple RDD. And that is all there is to it.

Machine Learning, Deep Learning and Streaming Data Processors

When AlphaGo beat the human last week using ML to process a small set of board positions which its computing power could process, it proved that Deep Learning ML (that which uses algorithms vs. simply data) has arrived. But can the same machine analyze a streaming set of unrelated mouse clicks to identify a "hack"? Could it have blocked the hacking of NY Fed and saved Bangladesh $100M of lost funds?

As I refresh my understanding of ML - this time with Spark MLLib. I am thinking that this use case of streaming data analyzes with ML or Deep Learning is the NBT (Next Big Thing). To run this type of computational jobs, one requires a cloud because no small cluster will do and it needs a special fabric of the network because nothing enforces better than a policy on the network. Next generation compute systems and specially memory/microprocessor arch has found its killer app in streaming data processing just like GPUs found video games. This time the games are played by hackers and loss is measured in hundreds of millions.  

Microservices and Container - Not perfect together

There is a continuous turf battle going on since 1990s between developer and deployer (a.k.a Admin). Currently the admin controls the security, scale and size of the application. The developer controls the content, architecture and interfaces to other systems. Two new emerging technologies empower (or shift power) these two constituencies. Microservices empower the developer while container empowers the deployer/admin.

If either wants to take off, they need to find other partners not each other. If microservices want to take off it needs to get the container monkey off its back. It shifts the responsibility of security and scale on the developer and away from the deployer. Developer should ideally be limited to interface design and selection of abstractions that make logic easy to codify. If the security and scale fall in the hands of developer it is going against the grain of the evolution of application where binding is done as late as possible and certainly not at design time. A datacenter administrator will be very uneasy deploying applications where developer has coded in the security policy and scale limit.  

Container has a similar story. Container is not a disruptive technology like it is portrayed to be nor is its effect as tectonic as virtualization or  Java/JVM. It needs to find a partner in a deployment technology like a vagrant or rails or PaaS. Currently its main value proposition is the time to bring up a execution environment. But to get that one has to give up security, naming, directory and identity. In other words, what it is doing is pushing those decisions to the developer who is the most ill equipped to handle those.  

JESOA

In continuing my education on Microservices, I came across a blog which gave a formula for SOA. In essence the formula says remove ESB, SOAP, Persistence (Reliable Messaging) and centralized governance and add containers  + PaaS. That is the microservices definition.

There was a initiative a while back called JEOS for "Just enough Operating System". May be we should try that for Microservices equation as a function of SOA. It is just enough SOA.

Microsoervices

In trying to understand the difference between Microservices and Web Services (Circa 2002), I cam across this definition.

"..., the microservice architectural style is an approach to developing a single application as a suite of small services, each running in its own process and communicating with lightweight mechanisms, often an HTTP resource API. These services are built around business capabilities and independently deployable by fully automated deployment machinery. There is a bare minimum of centralized management of these services, which may be written in different programming languages and use different data storage technologies." source: M Fowler


When web services started a similar definition was put forth. The main focus then was to get away from Java RMI into a light weight HTTP based communication and bring in language independence (then championed by Microsoft's CLR). As we implemented web services it became quite obvious that HTTP was quite heavy weight and language independence was not very economical. In fact what worked was language homogeneity with standardization on language that could shed weight for simple tasks and leverage a framework for heavy lifting. 

One of the biggest innovations in Java was the built in packaging and (later) deployment mechanisms. In fact most of the appeal of Java was in its "platform" features and not language features. This whole CNA biz smells a lot like language once again taking control of deployment and not leaving it to operations. We are seeing a application ops profession asserting itself. 

IoT is a Cloud Network

Cloud Native, SDN and SOA are all techniques not technologies. IoT is not a technique it is a use case that needs to use the above mentioned techniques to enable a mesh of connections that is manageable, secure and most of all just works. IoT could use any infrastructure including the carrier network but it is most likely end up using the cloud as a IaaS. IoT is a cloud networking problem that needs some connectivity middleware to sit on top of a virtual network. IoT is an application designed using SOA that will provision its network using SDN and will use ephemeral compute threads that have preexisting binding to the language runtimes. 

sdn and CMS

Only a few years ago, datacenter architects picked their overlay tunnel technology and created a list of stacks with which to build out their datacenter network. The cloud management system or even "the cloud" was an after thought. Today the tables have turned. We have datacenter architects debating the merits of various CMSes like OpenStack, vVMware (vSphere, vCAC, NSX) and a very distant third or fifth CloudStack. Within their CMS they are asking for support from one or more SDN stacks. The days of stanalone SDN stacks are gone. The battle today is between an open ecosystem like OpenStack vs. multiple closed ecosystems.

So what are the SDN stacks being evaluated on by these cloud datacenters?

First is ability to scale. And by scale, I don't mean just the overcoming the vlan exhaustion issue with annoating BGP or encapsulating L2 in L3 etc. Scale means the performance of the cloud network scales with the number of nodes in the datacenter. The nodes are server nodes. The cloud network does not scale with number of switches. Scale means your automation system can manage the configuration of a 50 node cloud as easily as 5K node cloud.

Second is heterogeneity. This one is a quite a beast because it requires supporting all the major hypervisors, authentication systems, SIAMs,  best of breed appliance (virtual and physical). From a cloud vendor's perspective this is where the R&D dollars are mostly spent i.e. in creation of a heterogenous ecosystem. Not proprietary ones like iCloud.

Third is security. Not just network security, or long expensive compliance test but application data input validation, fraud prevention, almost waf like.

Killer App for Overlay Networking/SDN

SDN has been searching for a killer app since its birth in the midst of protocol and encapsulation debates of 2011. It wasn't monitoring, flow management or physical network orchestration for a controller. It turns out it is container networking.

Container is challenging the VM or a group of them is a unit of application. Its value proposition is removal of the virtualization tax and being open source it does not cost a whole lot to try out. The schedulers (k8, mesos etc.) seem to be maturing fast enough but the networking behind is still quite elementary.

Using offloads that accelerate encap/decap of an overlay network on the JEOS (Just enough OS), containers with virtual interfaces can outperform hypervisor based VMs and integrate better with orchestration technology like gubernetes.

Utility Pricing in Cloud

The license management technology is still behind the times when it comes to cloud deployment. Most of the license enforcement mechanisms are based on hostname, ip address, usernames and are licensed based on seats.

There is pretty much no utility based pricing anywhere in the cloud. Either it is a flat subscription or a host based or site based license. If we want to regulate the internet as a utility, we need to figure out a way to charge the user for software used as a utility. There is nothing that I can see inside software system that even meters usage. Without this there is no way to bill a user based on usage. A utility based smartmeter for software is something that the industry needs if cloud biz model is to take off.

SDN and SmartNIC

SDN or programmable network is quickly being split into three separate problems. First is programmable management plane, second is programmable control plane and programmable data plane. A lot of time is spent on discussion of programmable management and control. But very little time is spent on programmable data plane.

Whatever overlay (over L2 or L3) is used for multi-tenancy, unless all the tenants are in the same host, the packet needs to leave the host. This host interfaces with underlying fabric using a NIC of variable complexity. Today the fabric is not multi-tenant and NIC is not really programmable.

The multi-tenant fabric is a focus of academic research. As always they will come up with a new tag that needs to be inserted after existing fabric tags. But its adoption will require the NIC based programmable silicon. This NIC needs be truly programmable (in  a programmable FPGA sense of the word and not SR-IOV VF sense of the word).

SOA and SDN - Lots in common

A decade and a half ago SoA drove an architectural change in software systems with the primary goal of loosening the tight bindings that existed between distributed components in a system. Then the tight binding was the language used (primarily Java). SoA was not a product, but it drove the roadmap of almost every product in the software industry. It created new products as well. Tight bindings made the software systems unscalable and brittle. SoA succeeded in that today's systems are more distributed and more scalable. Today's programmer has a choice of language. SDN too is not a product, it is an architecture. It too aims to loosen the tight bindings in the network systems - in this case a tight binding between the interface and the implementation (resulting in monopolistic markets). But the impact of SDN on the networking industry at least so far (4 yrs since OpenFlow) seems very limited. SoA drove a new serialization, new interface definition, new directory and a new wire protocol. SDN so far has done nothing of the sort. This could be because SDN is looking for a killer app. It is certainly not vanilla network management where it is applied today. SoA did not take off until a single webpage relied on 300 or more services to render the page. This technology was rapidly adopted in the social networking space. We are not seeing anything like this in networking yet. Cloud Networking or NFV may be those killer apps that SDN needs, but so far, we have not seen them drive any fundamental innovation into networking. SDN today is just a paper tiger. It needs a killer app and a real product to succeed. OpenFlow is increasingly looking like CORBA and not SOA.

Who is the cloud customer?

Will the real cloud customer please stand up? Most of marketing theory strives to correctly identity the "real" customer. Once the customer is identified it is a downhill task to cater to her/him. The problem is the real customer is always hiding. You would too if everyone wanted your money. Just like you don't answer the telemarketing calls at home and want to get on the do not call list, the real customer uses proxies to engage the vendors. In a cloud the customer is CIO who is still spending 84+% of his budget on fixed costs. That leaves him a mere 15+% to play with on his/her own discretion and most say of that only 5% is actually spent on innovation. His budget is under review every year from the CFO who has to keep showing a growing bottom line inspite of a flat to downward slopping top line. If you follow the S&P 500 companies where revenue growth for the last 3-4 yrs is under 5% (they are taking ORCL to the cleaners for missing even that today) and the divvys are expected to replace the lost income from the bonds (thanks to the pump from Fed), you will see very little room for the CFO to maneuver. Cloud offers the CIO more bang for the buck and hence this interest in cloudification of all assets. That pressure to reduce the fixed cost is driving software defined everything. Software can replace most of the fixed costs ('admins'), it can disintermediate the admin. This trend is driving a whole new industry in the area of configuration management and ever more user friendly self-service portals. So if you are designing a system that connects a super user friendly (read simplistic) to a sophisticated back-end (read complex system), you have to use a language that is different from the one you may have used to specify a menu driven desktop system.

Protocol vs. API: OpFlex

An API and Protocol both enable communication between two (or more if bus is involved) endpoints that are well defined and have address reachability. But API wins over protocol because of the flexibility that it provides over a static protocol. This is especially true in infrastructure management. With an API interface, one can support multiple management models including but not limited to programmatic (rpc, messaging), declarative (all those ini files or config-t command on Cisco IOS).


A key initiative in the cloud/virtualization industry is to open source every component in VMware's ecosystem. It started with hypervisor, then jumped the stack and moved to cloud management suites. What is not standard and open source yet is a set or inventory of managed objects. I am talking like managed objects that can be queried like exists at a much primitive level in SNMP. vCenter has such as inventory of managed objects and the industry should endeavor to create a standard and open source version of this inventory of managed objects.


Recently I came across a draft from Cisco on Opflex. I think the authors are attempting to create open src version of inventory of managed objects, but doing so in a very inflexible way i.e. creating a protocol for communication between endpoints. It would have been more useful had this initiative tried to create a framework with abstract objects that could be customized for the use case. The framework could offer a few basic services and define a few primitives. One of that service could have been endpoint registry, another could be endpoint profile. While it is positioned as a distributed system, it includes a domain which bring a domain controller into the picture.


What the cloud/virtualization industry needs is not another protocol but a SNMP like system that is open source but works at a much higher layer of abstraction instead of just device. We need to create a open source version of inventory of managed objects that can managed virtual resources that could run on any hypervisor.

L3 to Server

There is a trend towards moving switching (L2) out of the hypervisor and onto a NIC. After multiple false starts, it looks like it might actually happen.

Overlay networks essentially empower a virtual management tool or orchestrator to perform the control functions of a L2 network. The tool knows the full lifecycle of a VM and does not really need to learn any mac addresses. Coupled with a intelligent NIC and some standard based overlay (MacInIP), this tool can remove the need for a L2 switch in Hypervisor and facilitate bring L3 network directly to a server. If this happens, it will shift the network edge inside a server and shift the market power away from pure networkers to server vendors.

NfV Packet Processing

One of the key pillars on which the vision of NfV rests is optimized (meaning fast) cpu based packet processing. Sometime referred to as datapath processing in the cpu-mem complex without the need for any asic acceleration. No requiring any asic in the data path makes the cpu fungible (changed at will) and dramatically brings down the cost of network functions.

As we look at optimizations, we may also want to looks at why we have the ethernet I and II packet types and why the MTU min/max/jumbo packet sizes were arrived at. Do we really need the ethernet packet when the conversation is between two virtual machines on the same node?  Even across the host boundaries, but within the same data center, it is possible to have a conversation without having to create ethernet frames.

Hopefully we won't take existing VNFs and simply make OVFs out of them and run on a favorite hypervisor. Real NfV would involve refactoring the VNFs to use packet gateways for certain function and use standard RPC/IPC mechanisms for others.

Hypervisor based Overlays or Web Server based Overlays?

There  is a characterization of overlays as an abstraction as defined in computer science. An abstraction can be compiled down to the physical sometimes using an intermediate format. (like objects to byte code). Overlays are not abstraction, they are mark-ups and create a new layer for processing.  Mark-up are hints to higher layer software on how to process the content. There are services that these overlays need from the underlay and the exchange points are implemented in gateways.

Lot of complexity is being built into overlay networking because the design center of overlay networking is the hypervisor. By making hypervisor as a key node in the federated controller architecture, we are leaving out alternate mechanisms that enforce tenant/workload isolation like lpars/containers etc.  which don't need or require hypervisors.

Packets, Flows and Messages

I came across this post while doing the daily reading on SDN  https://devcentral.f5.com/articles/the-bifurcation-of-the-network-flows-versus-messages#.UjDVZ7Hn_mE. This resulted in a flashback as I recall sometime in 2004/5 time frame I was trying to explain to folks that message based switches are the way to go for then emerging service oriented network.

I would articulate the messages vs. tuple (flows) vs. header (packet) a little differently. In a service oriented network (and it is not SDN), the brain or intelligence bubbles up into a overlay network of proxies. Those proxies communicate among each other using messages. The brawn sinks down into a dumb data plane. The mapping between the two is done by "binding" to a transport. In those days HTTP was the transport but it was very apparent that it could not serve the purpose due to lack of asynchronous MEP (exchange). Today we are closer to realizing that vision.

Alas.. but we got hit on the head with murphy's law. In came SDN trying to disrupt the data plane by creating a centralized control plane. Except the control plane is not as intelligent as a the control plane of the original SON concept. It still works on packets. Fortunately, the byproduct of SDN hype is the real fruit of all this labor. That by product is programmability of the network device (not the network just the device). Today a message processor can read a route sent over the overlay of proxies and program the control plane on the switch to change the flow direction. This was not available in 2004/5. This was the original network virtualization concept. Virtualization because in the message processor you can have multiple tenants (although in those days we called it applications) and each one sees a different topology for its packets.

Big learning from that experience ... it hurts more to be early more than to be late.