News for hackers. Stuff that matters.
Re:Visualize AWS Cloud Diagrams by viraptor on Tuesday September 22, 2020 @09:43AM (#24554290)
I may be missing some complexity that's not clear from the description, but since all the resources should be available in AWS Config already, why is it using so much extra infrastructure? Why isn't it a "point me at AWS Config dump, I'll give you an explorable static page/SVG" script instead? (The cost part would have to reach out to billing and/or cloudwatch, but still...)

Or expressed a bit differently: what are the infra + 2 data stores doing that's a major improvement over processing raw AWS Config + CUR data which should easily fit in laptop memory. We're talking megabytes of data for a point in time snapshot, right?

Re:Visualize AWS Cloud Diagrams by tidepod12 on Tuesday September 22, 2020 @10:54AM (#24555121)
Speculation on my part, but this seems to be much more geared towards data analysis of your environment rather than just having an SVG for reference.

I'm assuming like most things AWS, this is not targeted at your small-sized startup running 2 VPCs, a handful of subnets, and 10-20 EC2 instances. This is likely built specifically for major enterprises running a hundred thousand instances, with hundreds of VPCs, thousands of subnets, hundreds of thousands of security groups, etc. In such environments, a static SVG isn't feasible and you might want something that allows more analytics instead, which I think is the entire point of using Neptune.

Re:Visualize AWS Cloud Diagrams by viraptor on Tuesday September 22, 2020 @06:39PM (#24560843)
I should've been more clear, but don't confused a static page and SVG with static content. I was thinking closer to a one page interactive app generated from a data snapshot.
Re:Visualize AWS Cloud Diagrams by toksvaeth on Tuesday September 22, 2020 @10:34AM (#24554828)
Hi,

That's a good question.

Whilst Perspective sources a lot of data from AWS Config, it also supplements and enriches the data from other sources where resources are not supported or where we wanted to show more detail. We'll be adding in other data sources in future, too.

Additionally, the AWS Config query language is quite helpful but doesn't expose a graph of the resources. Our approach is to pull the individual relationship data out and represent those in a graph DB (Amazon Neptune). Backing on to a graph database gives us the ability to let users navigate through the relationships of interest on the fly, rather than dumping what would inevitably be a complicated mess on the screen if we just showed everything.

Using this approach is also one of the ways we've tried to solve the UI/UX challenges presented in this space. Rather than say "show me all the things" that would lead to a complex mess of interconnections (and probably explode browsers) we instead encourage a workflow that starts with a resource or workload and explores outwards from there. It also allows us to implement some interesting features in future -- we intend to keep working on the project.

The ElasticSearch cluster is used by the discovery process to locate resources based on the metadata we collect; it's part of the way we enrich the data and relationships beyond what we get from Config.

Re:Visualize AWS Cloud Diagrams by viraptor on Tuesday September 22, 2020 @06:41PM (#24560857)
As with other comment, I should've been more clear. SVG / generated page doesn't mean no filtering. Just a point in time snapshot which doesn't require API calls once generated.