Prompt is the new Config, Workflow is the new Product

 Prompt are dynamic configurations that can adapt to incoming requests. Today configurations are static files which may miss a lot of corner cases. Enter an LLM and now we can reason with APIs and dynamically workaround blocks. This one ability of the LLMs can create panic among all gateways to enterprise. A gateway allows/deny based on a static policy. What if we embed this intelligence in them so they adapt continuously to incoming traffic. This is atleast one reason why firewalls/cybersec products are receiving scrutiny from CIOs. A prompt injection can completely change the security posture of an enterprise. 

When you have a wild and powerful entity on the loose, you need to build harness to channel this energy. That is the workflow. More and more startups are now focusing on workflows instead of just functionality. They called this harness a wrapper around LLM. These wrappers are the new gateways into the enterprise. 

Token Pricing: Reasoning Tax,, GPU Utilization & GPU Recency

 According to newly published TPI (Token Price Index), average cost of 1M tokens is just over $2. It rose over 75% in 1 year. So does a model serving provider make money off this pricing? Yes and No. The marginal cost of power + infra + facilities ranges between 6 cents to 1.60 cents per million token. The key factors driving costs higher are (a) low utilization of GPU (currently around 15%), (b) reasoning tax (the token generated inside to support customer tokens (c) GPU recency. 

On the latest GPU, the cost of generating token is lower creating an incentive for providers to move up to the latest GPU. But this also means they pay for supply constraint which will likely stay forever because the GPUs that are one generation behind are not used. So the amortization on those GPUs which assume 100% consumption for the full 6 years is just too low i.e. margins are artificially high. 

The GPU utilization relies on many techniques at deployment time but batching is the one which makes a meaningful impact. Assuming 100% of the input is batched is euphoric. We are average 15-25% on recent GPU and almost 1-2% on older GPUs. No one wants to run on legacy GPUs. Net net this factor is also artificially inflating margins. 

The reasoning tax comes from ordinary queries but especially from agentic AI. Agents generate output that is sent back to the model which then starts generating reasoning tokens which are currently not charged to the customer. These reasoning token cost is eaten by the provider and is referred to as "Reasoning Tax".