Re:What the Heck is a Data Mesh?
by datameshlearn on Thursday July 29, 2021 @07:30PM
(#28002805)
Feel free to throw in the data mesh community Slack[1]. There was an interesting approach that sounds kinda similar re schema contract management from FindHotel that they posted a few weeks ago re data mesh[2].
1 https://launchpass.com/data-mesh-learning 2 https://blog.findhotel.net/2021/07/the-evolution-of-findhote...
I have a dumb question. Could I use flow to import a text file into a postgresql database? The text file is not append-only.
There's a lot of tools to import logs into stuff like kafka but not to import whole files (that can change) to a database.
Re:What the Heck is a Data Mesh?
by jgraettinger1 on Thursday July 29, 2021 @06:03PM
(#28002040)
Yep. You can, for example, have it watch file(s) in S3, and every time a file changes it will flow its records through into a table it creates in your DB, keyed on your (arbitrary) primary key.
Any way to watch local files too? S3 might be fine, but just asking.
We're still building, but it's a GitOps workflow tool that tightly integrates schema definition (JSON Schema), captures and materializations from/to your systems & SaaS, rich transformations, catalog and provenance metadata tracking, built-in testing, and a managed runtime. All with sub-second latency.
Flow's runtime uses nascent but really promising open protocols for building connectors to the myriad systems and APIs out there. We're seeing Airbyte's work (itself built off of Singer) as the best steps in this direction and are leaning into that effort ourselves.
[1] github.com/estuary/flow