Tales of a journey in diagramming AWS infrastructure

Tales of a journey in diagramming AWS infrastructure
Number 1, 1950 (Lavender Mist) by Jackson Pollock.
Updated on August 19, 2023.
Developed an online “CloudFormation to draw.io” converter, available for you to try here.

I recently embarked on a daring quest of re-documenting my entire cloud infrastructure - a fully serverless platform consisting of a mountain of CDK projects and CloudFormation templates, all deployed across dozens of accounts managed by our organization. I knew that manually creating diagrams for each component would be time-consuming and laborious. But I was determined to work smart, not hard.

In this article, I share my insights on the significance of infrastructure visualization in building a thriving cloud environment. Join me as I explore various graphical, diagram-esque application design tools on AWS, from the well-forgotten, yet still very much alive CloudFormation Designer to the recently introduced, yet all kind of familiar, Application Composer . Delving into hidden features of diagrams.net , I uncover how you can effortlessly build impactful cloud infrastructure diagrams.

Unlocking the power of tiny icons: The significance of diagramming your infrastructure

When people ask how much time I spend creating diagrams for my cloud infrastructure, I usually answer with a shrug and a “probably too much.” But the truth is, I absolutely love it. After days in YAML purgatory, writing and staring at configuration files, creating diagrams to illustrate the newest parts of my applications is my “very own therapeutic way” of reconnecting with my inner self. It’s like a digital coloring book, but instead of painting between the lines, it’s all about picking the right hex code and aligning AWS icons to the pixel. It is also an enjoyable opportunity for me to very discretely promote my own secret agendas: … Let’s just say that I do tend to prefer more lively, vibrant colors for my favorite serverless services; over dull, tarnished colors; that seem a little bit more fitting for our old-school EC2 instances and security groups - It’s all about the subliminal messaging, people.

But creating diagrams isn’t just about the mental health benefits. They are a crucial, yet too often overlooked, step in building a successful, well-managed, well-governed, well-architected cloud infrastructure for your organization. “Conway’s Law” reminds us that the architecture of a system reflects the communication structure of our organization: Clear and efficient communication is essential to a successful cloud infrastructure, and diagrams help us avoid misunderstandings, keep everyone on the same page, and streamline workflows, ensuring that new systems seamlessly integrate with the existing setup. In other words, without the right tools and effort to communicate clearly and efficiently with your team, your cloud infrastructure will end up like that Jackson Pollock painting, in no time.

Here are three reasons to start diagramming your way to the top:

  • Diagrams are a prerequisite to going serverless: Is there anything more thrilling than building complex, event-driven, serverless architectures? In 2022, serverless technology won some crucial battles in the computing war, and I saw more and more organizations recognizing the benefits of going “wide” rather than “tall.” However, “Marcoservice” architectures do have their drawbacks: One of their biggest challenges is their lack of observability and explainability: The complexity of interactions between serverless application components can get overwhelming faster than you think, even for the most accomplished architects. Even the best, well-designed serverless architecture won’t mean much if it’s as confusing for outsiders as a Rubik’s cube in the hands of a toddler.

  • Diagrams lead to better architectural decisions: Diagrams help you avoid misunderstandings and keep everyone on the same page, by communicating more effectively with your team members or stakeholders. This will, in turn, streamline your workflows and pave the way for future growth, ensuring that newer systems get well-integrated into your existing infrastructure.

  • Diagrams are your #1 free tracing and debugging tool: Diagrams will preemptively save you from many headaches: With a clear overview of your applications, you can identify blindspots and areas of improvement efficiently. No. No amount of X-Ray implementation will fill the void in your heart from not having clear diagrams. (Just draw the darn things.)

The current state of graphical application architecture services on AWS

Now I’m no fool: the “normally constituted human being” you are, is probably resenting drawing infrastructure diagrams - at least equally as much as writing infrastructure code. On that particular note, I’ve seen firsthand how maintaining infrastructure code can be a major obstacle to serverless adoption in small and medium businesses - paradoxically, organizations that would benefit the most from the cost savings and scalability of serverless technologies. This is most certainly why AWS has been working hard to streamline infrastructure code writing for years now and make it more accessible to a broader audience; whether it was with CDK yesterday, in an attempt to appeal to the software developers community; or with graphical programming solutions tomorrow.

At the latest re:Invent conference, they announced with great pomp Application Composer - a graphical tool that can help you develop serverless applications visually, with no code required (results may vary).

What caught my attention from this announcement was a peculiar sense of déjà vu 🐈🐈. Graphical programming tools have always promised a lot, but they often lack the level of automation and personalization found in dedicated IDEs or diagramming software. Yet, deep down, we all secretly wish for that magical unicorn that could eliminate the need for writing YAML files and effortlessly create fantastic diagrams. The idea of a tool that could make the process of visualizing my cloud infrastructure less daunting sounded too good to be true, but I couldn’t resist giving it a shot. Intrigued by the possibilities, I started exploring my options available on AWS for graphical programming.

AWS CloudFormation Designer

CloudFormation Designer is a quirky little feature of AWS CloudFormation that has been around for ages. Yet it has remained largely unknown from ignored by the general public, as far as I’m concerned. It made its maiden debut in 2015 , a time when Lambda just landed on planet earth, and API Gateway was still the new kid on the block. The idea behind it was to provide users with a more intuitive way of creating and editing CloudFormation templates, through visual diagrams instead of clunky configuration files: An ambitious task. Too much maybe. So much so that the product team in charge, probably having their doubts, thoughtfully included an embedded YAML editor <textarea /> to the thing right off the bat, always lurking, just a click away, waiting to be used.

AWS CloudFormation Designer
Spaghetti code illustrated.

On the bright side, it does deliver on what was advertised (… I guess?): Feed the Beast a new or existing CloudFormation template, and you’ll be able to drag and drop building blocks and arrange them to your heart’s content on a white canvas. Some come equipped with visual indicators, hinting on what to use them with: A Bucket Policy will politely remind you it should be hooked to a Bucket resource, that kinda thing. Cloudformation Designer draws up dependencies automatically between your resources; and once you’re satified with your display, will save all your layout information as a header to your CloudFormation YAML file: adding a glittery bow on top of an already well, well-over-decorated Christmas tree.

        width: 60
        height: 60
        x: 240
        'y': 450
      z: 1
      embeds: []
        width: 60
        height: 60
        x: -250
        'y': 350
      z: 1
      embeds: []
        id: b6363c4f-515d-4a81-bf61-eadd2fc5e1d3
        selector: 'g:nth-child(1) g:nth-child(4) g:nth-child(1) circle:nth-child(1)     '
        port: 'AWS::RefLink-AWS::S3::Bucket-Bucket'
        id: 41af82d8-1b9e-419b-98e3-500fc5588a0b
      z: 3
Oh. Dear.

Credit where it’s due: CloudFormation Designer does feature every CloudFormation-deployable resource in existence for you to drop onto your canvas. But it’s probably not the thrilling experience you think it would be. Most resources you’ll find down there won’t show you a vast display of love and attention: They’ll just append a three-liner to your configuration file: an auto-generated resource ID, the resource corresponding type, and an empty Properties object. From there, it’s back to opening half a dozen tabs of CloudFormation documentation in your favorite browser. No custom icon either - or anything else that could give your infrastructure diagram a little extra “Pizzazz!” to that matter. In fact, the service still uses the ancient and decried AWS17 icon set, which might hint at when the last update rolled out.

Sure, I wasn’t expecting much from this gimmicky feature of CloudFormation. The canvas engine feels, rightfully, as temperamental and unpredictable as a seven-year-old on a sugar rush: it will simply have it its own way. No room for compromise or personalization. Designer must have been an inspiring idea back then, but was since left to gather dust, presumably by lack of public adoption. Still, there’s something charmingly retro about giving it a whirl before someone inevitably pulls the plug on it (GCP would have probably deprecated this thing two days before its launch). And who knows, you might discover that those AWS17 icons are the new retro chic . (I kinda do.)

AWS Application Composer

Fast-forward seven years, for the unveiling of the Application Composer service at the 2022 AWS re:Invent conference, to great fanfare. The service is designed to provide users with a more intuitive way of creating and editing CloudFormation templates serverless applications through visual diagrams rather than configuration files. (Sounds familiar already?)

AWS Application Composer
Only a handful of services are supported yet, but Application Composer is showing great promises from a UX perspective.

The service allows users to drag and drop building blocks onto a canvas and arrange them as desired. CloudFormation Designer Application Composer then saves all layout information to your CloudFormation templates files metadata, like a glittery bow on top of an already well-over-decorated Christmas tree, making it easier to group resources together when the project is loaded again.

      Label: Group
        - CodesQueue
        - CodesTable

Yes: it does kind of already feel like a “Governance & Management” rendition of the Groundhog Day movie.

Jokes aside, it’s not the first time AWS has revamped existing services entirely from the ground up. But I find this particular “2023 glitch in the Matrix” all the more interesting, given how much those two features look alike and aim to solve the exact same problem, only with a seven years age difference.

However, despite its fresher looks and more intuitive interface, Application Composer in its current state (at the time of writing, the service is still in preview) still lacks key features that would make it the go-to tool for building, visualizing and maintaining workflows on AWS, the one everyone wants it to be. The most obvious is its limited support to only a handful of services and resources: it’s not ready yet, but I’ll give it some time, and sincerely hope it has a brighter future than its alter-ego.

One thing this strange redundancy reveals, however, is that graphical infrastructure programming is a hard thing to get right, especially in an ever-evolving environment such as. Cloud. Although Application Composer represents a significant improvement over CloudFormation Designer, at least from a UX perspective, there is still a long way to go in providing the necessary functionality for creating, managing, and visualizing complex Cloud infrastructures. The question remains whether folks at AWS will commit to giving the necessary time, effort and attention to the service; that Designer would have needed at the time. Until then, YAML and infrastructure diagrams will still have a long way to go.

Seamlessly converting CloudFormation templates to draw.io files

Building graphs out of CloudFormation templates is not a novel idea. CloudFormation templates are nothing more than a linear representation of an oriented graph inside a JSON structure, with each resource representing a node and intrinsic functions references giving you the relationships. Big boys have had their go at it, and you may also stumble across a couple of open-source initiatives trying to achieve this goal. Yet, after scouring the boards and trying out many options, none offered the level of flexibility, personalization, and, more importantly: simplicity; I so desired.

I set my sights on a target: the ability to automatically import my CloudFormation files to draw.io . I did not want anything too fancy or smart: I only needed a good enough base from which to take over and arrange the visuals myself, add comments, and print out to a PNG file.

I am pretty sure draw.io doesn’t need any introduction: It is one of the most popular online open-source diagramming tools around there, known for its ease of use, high degree of personalization and versatility. To me, what sets draw.io apart is its ability to integrate with third-party tools such as Confluence, Jira, or GitHub, making it an excellent choice for teams to collabore, share diagrams, and maintain version control. (The only thing I wish it had, would be a native integration with S3.)

At first hand, that task may not be as simple as it seems: .drawio files are saved in a not-so-friendly XML format; inheritance of the JavaScript graph library the tool is built upon. Sure, transcribing CloudFormation JSON to mxGraph XML would have been a viable option, but it did feel like a whole amount of work, test, and hassle, that I was really not feeling putting in.

Creating diagrams with ease using the draw.io CSV API

Thankfully, the folks at draw.io had thought of everything, and have a solution for this problem as well . One of the lesser-known features of the application is its CSV API , which allows you to create diagrams with minimal effort out of tabular data. The best part? You don’t have to compromise on style and personalization: Simply add your formatting options to the file header, and draw.io will take care of the rest.

# label: %component%
# style: shape=%shape%;fillColor=%fill%;strokeColor=%stroke%;verticalLabelPosition=bottom;aspect=fixed;
# namespace: csvimport-
# connect: {"from":"refs", "to":"id", "invert":true, "style": \
#            "curved=0;endArrow=none;endFill=0;dashed=1;strokeColor=#6c8ebf;"}
# width: 80
# height: 80
# ignore: id,shape,fill,stroke,refs
# nodespacing: 40
# levelspacing: 40
# edgespacing: 40
# layout: horizontaltree
## CSV data starts below this line
1,Habit Tracker HTML App,#ffe6cc,#d79b00,mxgraph.aws4.mobile,
2,UI Assets,#277116,#ffffff,mxgraph.aws4.s3,1
3,User Authentication,#C7131F,#ffffff,mxgraph.aws4.cognito,1
4,API Gateway,#5A30B5,#ffffff,mxgraph.aws4.api_gateway,1
5,AWS Lambda,#277116,none,mxgraph.aws4.lambda_function,4
An example of draw.io CSV file for a Habit Tracker app that uses AWS.

To my delight, draw.io even included an example of a CSV AWS infrastructure diagram in one of their blog posts , which would convert into this sharp visual:

Pro tip: You can create direct viewer links for your draw.io CSVs using querystrings.

It seemed like the magic recipe was right there in front of us the whole time:

  1. Parse template resources in your CloudFormation definitions;
  2. Traverse their properties;
  3. Figure out their dependencies;
  4. Return as CSV line to append to your outfile;
  5. Profit.

The rest is just a matter of generating the file formatting, but nothing a configuration file and a couple lines of scripting could not handle.

An online converter to try out yourself!

I tinkered around and developed a handy little script I packaged into this web-based tool , that allows anyone to easily convert their CloudFormation templates to “draw.io-compatible” CSV files . With this online converter, you can create impactful and accurate visual representations of your what is exactly deployed by your templates in a matter of seconds, making them easier to understand and manage for anyone: Feed it a CloudFormation template file, and it will automatically generate the required declarations for you.

This diagram is drawn from this CSV file, generated by the converter out of this website’s CloudFormation template.

From a formatting perspective, everything is configurable through a verbose - yet, intuitive configuration file that mimics the styles properties of the draw.io API: you can tweak the appearance of both nodes and edges; or change colors and bindings of each single AWS icon, to make your chart as visually impactful and easy to understand as possible. I’ll probably be adding more to it in the months to come, so if you feel like contributing , you’re more than welcome. No more excuse to not draw those darn things and turn your dull YAML files into a work of art.