nathanwallace

A similar site I've enjoyed for >15 years (!!) is https://techmeme.com/

I also use it's sister aggregator site for political news every day - https://www.memeorandum.com/

Readers may also enjoy Steampipe [1], an open source tool to live query 140+ services with SQL (e.g. AWS, GitHub, CSV, Kubernetes, etc). It uses Postgres Foreign Data Wrappers under the hood and supports joins etc with other tables. (Disclaimer - I'm a lead on the project.)

1 - https://github.com/turbot/steampipe

Steampipe project lead here - thanks for the shout out & feedback multani!

I agree with your comment about JSON columns being more difficult to work with at times. On balance, we've found that approach more robust than creating new columns (names and formats) that effectively become Steampipe specific.

Our built-in SQL client is convenient, but it can definitely be better to run Steampipe in service mode and use any Postgres compatible SQL client you prefer [1].

You might also enjoy our open source mods for compliance scanning [2] and visualizing clusters [3]. They are Powerpipe [4] dashboards as code written in HCL + SQL that query Steampipe.

1 - https://steampipe.io/docs/query/third-party 2 - https://hub.powerpipe.io/mods/turbot/kubernetes_compliance 3 - https://hub.powerpipe.io/mods/turbot/kubernetes_insights 4 - https://github.com/turbot/powerpipe

Beyond the Steampipe CLI (which has an embedded Postgres), you can use the steampipe plugins as native Postgres FDWs [1], SQLite extensions [2] or command line export tools [3]. We're definitely interested in bringing the plugins to duckdb as well, but haven't had a chance to focus there yet!

1 - https://steampipe.io/blog/2023-12-postgres-extensions 2 - https://steampipe.io/blog/2023-12-sqlite-extensions 3 - https://steampipe.io/blog/2023-12-steampipe-export

Fair point. I guess that I'm disclosing that I lead the project and disclaiming the self-interest in posting? :)

Steampipe is most commonly used for DevSecOps. It works with any BI tool for dashboards (it's just Postgres), but is really great when paired with Powerpipe [1] for security benchmarks, visualizing infrastructure and much more [2]. It can definitely be used for a wide range of data scenarios, e.g. there is an OpenAI plugin [3].

1 - https://github.com/turbot/powerpipe 2 - https://hub.powerpipe.io 2 - https://github.com/turbot/steampipe-plugin-openai

Thanks for leaning into Steampipe!

thank you! We believe that SQL makes the simple stuff easy and the hard stuff doable :). Hopefully our repos & hubs have good enough examples to get you started most of the time!

doh! thanks for the heads up :-(. Fixed to be https://hub.steampipe.io/plugins

Readers may also enjoy Steampipe [1], an open source tool to live query 140+ services with SQL (e.g. AWS, GitHub, CSV, Kubernetes, etc). It uses Postgres Foreign Data Wrappers under the hood and supports joins etc with other tables. (Disclaimer - I'm a lead on the project.)

1 - https://github.com/turbot/steampipe

Readers may also enjoy Steampipe [1], an open source tool to live query 140+ services with SQL (e.g. AWS, GitHub, CSV, Kubernetes, etc). It uses Postgres Foreign Data Wrappers under the hood and supports joins etc with other tables. (Disclaimer - I'm a lead on the project.)

1 - https://github.com/turbot/steampipe

For runtime cost analysis, you could try Steampipe [1] with it's Powerpipe "thrifty" [2] mods. They run dozens of automatic checks across cloud providers for waste and cost saving opportunities.

If you want to automatically make these changes (with optional approval in Slack) you can use the Flowpipe thrifty mods, e.g. AWS [3].

It's all open source and easy to update / extend (SQL, HCL).

1 - https://github.com/turbot/steampipe 2 - https://hub.powerpipe.io/?objectives=cost 3 - https://hub.flowpipe.io/mods/turbot/aws_thrifty

In addition to Postgres FDWs, Steampipe plugins are also available as a SQLite extension [1] or a CLI export tool [2] for lighter weight use cases. (Although a local Postgres has a surprisingly small footprint!) Building plugins as DuckDB extensions would be cool too, but we haven't done that yet.

1 - https://steampipe.io/blog/2023-12-sqlite-extensions 2 - https://steampipe.io/blog/2023-12-steampipe-export

Readers may also enjoy Steampipe [1], an open source tool to live query 140+ services with SQL (e.g. AWS, GitHub, CSV, Kubernetes, etc). It uses Postgres Foreign Data Wrappers under the hood and supports joins etc with other tables. (Disclaimer - I'm a lead on the project.)

1 - https://github.com/turbot/steampipe

Steampipe does live queries against the API endpoint and stores results in an in-memory cache. You can modify the cache expiration to adjust that behavior across queries [1]. If you want to do change data capture to act on results then check out the query trigger [2] in our Flowpipe open source project [3].

1 - https://steampipe.io/docs/guides/caching 2 - https://flowpipe.io/docs/flowpipe-hcl/trigger/query 3 - https://github.com/turbot/flowpipe

Readers may also enjoy Steampipe [1], an open source CLI to live query Google Sheets [2] and 140+ other services with SQL (e.g. AWS, GitHub, etc). It uses Postgres Foreign Data Wrappers under the hood and supports joins etc across the services. (Disclaimer - I'm a lead on the project.)

1 - https://github.com/turbot/steampipe 2 - https://github.com/turbot/steampipe-plugin-googlesheets

Readers may also enjoy Flowpipe [1], a cloud scripting engine to build workflow pipelines using HCL (like Terraform). It's as-code, self-hosted and open source. Our design choices were a bit different - focus on DevSecOps, composable OSS mods [2], first-class support for retries etc [3], and the flexibility to run function code or containers when required [4].

1 - https://github.com/turbot/flowpipe 2 - https://hub.flowpipe.io 3 - https://flowpipe.io/docs/flowpipe-hcl/step#retry 4 - https://flowpipe.io/docs/flowpipe-hcl/step/function

Perhaps give Flowpipe [1] a try? It provides "Pipelines for DevOps", including a library of AWS actions [2], that can be run on a schedule (e.g. daily) or a trigger (e.g. instance start) to do things like turn off, update or delete expensive assets. It can also be combined with Steampipe [3] and the queries from the AWS Thrifty mod [4] to do common queries. We'd love your feedback if you give it a spin!

1 - https://github.com/turbot/flowpipe 2 - https://github.com/turbot/flowpipe-mod-aws 3 - https://github.com/turbot/steampipe 4 - https://github.com/turbot/steampipe-mod-aws-thrifty

Readers may find Steampipe's [1] AWS Thrifty Mod [2] useful. It will automatically scan multiple accounts and regions for 50 cost saving opportunities - many of which are looking for over-provisioned or unused resources. For example, it's crazy how much you can save by doing things like just converting your EBS volumes to the newer gp3 type. Combine with Flowpipe [3] to automate checks and actions. It's all open source and extensible.

1 - https://github.com/turbot/steampipe 2 - https://github.com/turbot/steampipe-mod-aws-thrifty 3 - https://github.com/turbot/flowpipe

Steampipe [1] is an open source project [2] that includes an embedded Postgres to instantly query cloud, code & more with SQL. This release expands our plugin ecosystem [3] to be a full Zero-ETL platform. Steampipe plugins can now run natively in your own Postgres as Foreign Data Wrappers [4], as SQLite extensions [5] or as simple data export tools [6]. Please give it a try, we'd love your feedback and contributions!

1 - https://steampipe.io 2 - https://github.com/topics/steampipe 3 - https://hub.steampipe.io/plugins 4 - https://steampipe.io/blog/2023-12-postgres-extensions 5 - https://steampipe.io/blog/2023-12-sqlite-extensions 6 - https://steampipe.io/blog/2023-12-steampipe-export

Thanks for your kind words - so great to hear that :-). We're really excited about Flowpipe and think it makes "cloud scripting" really fun & easy through pipelines - please let us know how you go and what we can improve!

Readers may enjoy Flowpipe [1], an open source cloud scripting engine we launched this week. It allows creation of pipelines in HCL to connect HTTP queries, run containers, execute lambda-compatible functions, query databases, etc all from your own machine / CLI. It can also be combined with Steampie[2], our open source project to query cloud resources (139+ plugins) with SQL. We're more DevOps focused, but Yahoo Pipes was part of the inspiration for our naming of these "pipes" :-)

1 - https://github.com/turbot/flowpipe 2 - https://github.com/turbot/steampipe

Most of our team just use their text editor to do formatter via plugins. I also find this online SQL formatter to be useful - https://extendsclass.com/sql-formatter.html

We use our open source project Steampipe [1] to query data using the GitHub plugin [2] and mods [3]. We have also opened up the custom mod we built for the specific charts & reports [4].

I hope they help - let us know if you give them a try!

1 - https://github.com/turbot/steampipe 2 - https://hub.steampipe.io/plugins/turbot/github 3 - https://hub.steampipe.io/mods?q=github 4 - https://github.com/turbot/steampipe-mod-community-tracker

Fair point! Rapid response is definitely valuable for thoughtful issues or PRs. For "questions" we try to just ask for more clarifying details or do a brief pointer to documentation. I find it can also be useful to deliberately slow down responses when getting caught in a low value conversation - kinda like rate limiting undesirable traffic to your website <grin>.

If the projects were my own, I'd consider a monorepo. We use this approach for Steampipe samples - https://github.com/turbot/steampipe-samples

If it's a collection of changes, small improvements, etc to existing projects and repos then personally I'd go for separate forked repos. Then you can track your changes relative to the original project source code and (hopefully) contribute back PRs etc more easily.

As always - there are pros & cons to both - just a matter of choosing the approach that feels best 51% of the time :-). Of course, it's minor in general compared to the value of just keeping on moving on your projects and work!

Keeping docs with the code is definitely helpful, we've found that works well for our plugins and mods. All pull requests are then reviewed for consistency and documentation at the same time [1] - e.g. does this new table have good column definitions and clear examples?

1 - https://steampipe.io/docs/develop/plugin-release-checklist

Monorepos have advantages and we considered that approach - easier mgmt, linear change timeline, consolidation of GitHub stars, etc. But on balance we felt our plugin and mod model benefited more from the separation - specific audience, clearer release history, simpler clone & install (mods), encourages contribution of plugins via repo ownership. Cross-repo tooling helps to offset the negatives of managing many repos, and as you say, the lessons we shared come also from managing our larger repos (e.g. the Steampipe CLI).