Cracks in the Walled Garden

4 min readSep 11, 2020

There are serious and systemic issues with AWS Services

I love AWS. It’s an indispensable tool for many start ups. However I also have two eyes and can see. And what I see is a walled garden of open source tools repackaged and sold at a premium.

The Open Source Problem

Developers are not adopting AWS Services in place of native open source technologies. Below are just a handful of match ups, all of which AWS is essentially loosing.

Most of these match ups involve AWS simply repackaging the competitor and selling it with a side of vendor lock in. Developers are not only unimpressed with AWS’s repackaged solutions, they are actively finding ways to implement open source stacks at the lowest possible value stage (EC2 and S3). I find this ironic because these are the same technologies I started using on AWS nearly ten years ago. If you are going to convince software developers to invest in learning your technology, it must be worth their time. As of today AWS implementations are no better (and often worse) than the open source version, and developers arguably have a more marketable job skill if they invest in open source. AWS needs to immediately expand its open source footprint, and offer technologies that are far superior than today's open source solutions.

One notable exception here is AWS Firecracker. It’s an open source micro VM that powers Lambda. It has enjoyed a lot of success. More like this please.

The Service Limits Problem

The fine print on service limits is the death nail for many projects. From Lambda concurrency, to SQS retention period, to the number of Cognito user pools, all services have limits that can, at any moment, kill your project. Worst of all these limits are arguably arbitrary, and exist solely to drive larger customers into value added pricing (hence the designation between hard and soft limits). One needs to only look at Lyft as a prime example of how one can find themselves in the AWS value added pricing trap. Developers who deal with even moderate scale would rather move to a lower value stage (such as EC2) than operate within poorly documented and incredibly disruptive limits. AWS should immediately remove most soft limits, provide clear and transparent ratcheted pricing, and DOCUMENT HARD SERVICE LIMITS IN A STICKY HEADER (ALL CAPS).

The Redundancy and Clutter Problem

AWS has too many services with overlapping functionality that don’t fit a cohesive whole. Some examples include:

These are just a few examples, but AWS is replete with overlapping functionality among its services, all which seem to solve a portion of a use case well, and a portion very poorly (or not at all). This is likely a direct result of AWS’s commitment to letting small teams drive the creation of products from the SA level out. This is a great model, but it is severely lacking governance. It’s not enough to just listen to your customers, you need to consider the developers implementing your countless solutions to the same problem. AWS should work to consolidate services into units that produce the largest number of reference architectures with the least number of services possible.

The Training Problem

As a developer it’s easy to feel like AWS is leaving you in the dust. This is a direct result of having too many services that all solve similar use cases continually iterating. In addition AWS has some truly horrible documentation that is the rabbit hole of all rabbit holes. Learning how to do “hello world” might take you through six different services, IAM roles and policies, CloudFormation, the and the CLI. Your little sip from the AWS tap quickly turns into a waterboarding. Onsite training is good, and can lead to more focused effort that solves an actual problem, but most people simply can’t afford onsite instruction. AWS needs to streamline the learning curve and provide better educational materials that are free of charge.

Conclusion

AWS, if you are listening, help developers help you. Invest in open source. There is a massive opportunity cost to pay for services that you must support (with rigid HA requirements) that do not enjoy widespread developer adoption. In point of fact the worst possible outcome in the SaaS world is to find yourself tied to a loss leader sinking to the depths. Your lack of unique technology in big data, the control plane, container orchestration, and messaging should be a slap in the face to your technology teams. Your engineering teams are doing a fantastic job developing switches, chip sets, fiber optic cables, etc., but you have largely abandoned innovation in software stacks. While your profits are huge, there is an opportunity cost racking up in the balance sheet, and it’s going to bite you in the ass one day.