05-18, 11:00–11:25 (Europe/Vilnius), Saphire A - Python
Understanding software architecture and how the data flows within software components is a vital step toward building and maintaining software systems. Architecture diagrams help enable this through digital graphical designs mixed with human-computer interaction. Furthermore, these visualizations not only help system architects, but also developers, project managers, and even customers. The complexity in designing them arises not only from the fact that such systems are an intangible conceptual entity, but also, most importantly, that they are ever-evolving.
While we are searching for life on Mars, our software diagrams remain manual and lifeless. Imagine a life where you update the code for your software, and the architecture view gets updated automatically and is ready to be interacted with. Let's use Graph Databases, pandas, and Python to add life to them and make them interactive.
Before I dive into how to make "auto-generative software architecture visualizations," let's first see if you have been in any of the following situations:
- You often wonder before you go to sleep, "A short time ago in a galaxy far, far away, I updated my team’s arch. diagram. So why is it stale… again?"
- Your team grew multifold over the pandemic and so did your components and ownership. Has it become harder to onboard new engineers due to the ever-changing arch. diagrams, dependencies, and owners?
- Your new release didn't go as planned because something broke in your new code. Wouldn’t it help if you had a holistic and interactive view to navigate between components while understanding how they communicate with each other? When was the last time you used your arch. diagram as a debugging tool?
- You managed to draw the data-flow diagram and you thought “this is majestic work right here!” But the feedback you received was, "it’s too detailed" or "it’s not detailed enough." While you’re trying to figure out the fine line between details, your hairline is starting to show!
If you’ve fallen victim to any of these scenarios, behold! Your prayers have been answered!
In the first half of the talk, we’ll describe an approach that can automatically identify the software design and data flows within the system. To achieve this, we use algorithmic scrapers and metadata profiling techniques that integrate with distributed trace, code structure, language dependencies, contribution ownerships, and other sources to avoid the toil of manual updates of software architecture diagrams.
In the second part of the talk, we will dive deeper into the details of how to use Python data engineering libraries to enrich the collected data and process it to store it in the graph database. The graph structure not only signifies the relationship between components in a real-world manner, but also helps in generating multiple views of the software architecture in an easy and comprehensible fashion. Now, instead of the architecture and data flow diagram being a static JPEG, those auto-generative views can be compiled into interactive and immersive UIs.
My team built this tool for our organization, Bloomberg, but the approach we took is germane to all tech organizations. We want to share the challenges and lessons we learned during our journey to help you build a similar tool for your organization because why not “Code More, Draw Less”?!
Beginner
What topics define your talk the best? –python, open source, design and architecture, data engineering, images and videos, fun
Yash Saboo is a Software Engineer at Bloomberg, where he is focused on building highly-available, consistent, and reliable workflow platforms for Bloomberg’s Fixed Income Indices. Previously, he was a Computer Science grad student at the University of Illinois Urbana-Champaign. His work generally lies at the intersection of distributed systems, databases, and machine learning.
Kang Min is a software engineer with over three years of experience in technology industry. He has worked for IBM Quantum as a backend software engineer in the past and is currently working as a full stack developer for Bloomberg Indices.