photoft

http://github.com/lyft/amundsen and https://github.com/apache/atlas

nice post for data quality integration with https://github.com/lyft/amundsen

Amundsen is a metadata driven application for improving the productivity of data analysts, data scientists and engineers when interacting with data. It does that today by indexing data resources (tables, dashboards, streams, etc.) and powering a page-rank style search based on usage patterns (e.g. highly queried tables show up earlier than less queried tables). Think of it as Google search for data. The project is named after Norwegian explorer Roald Amundsen, the first person to discover South Pole.

It includes three microservices and a data ingestion library.

amundsenfrontendlibrary: Frontend service which is a Flask application with a React frontend. amundsensearchlibrary: Search service, which leverages Elasticsearch for search capabilities, is used to power frontend metadata searching. amundsenmetadatalibrary: Metadata service, which leverages Neo4j or Apache Atlas as the persistent layer, to provide various metadata. amundsendatabuilder: Data ingestion library for building metadata graph and search index. Users could either load the data with a python script with the library or with an Airflow DAG importing the library.

there are four repos (3 for microservices, 1 for data ingestion library): https://github.com/lyft/amundsenfrontendlibrary https://github.com/lyft/amundsensearchlibrary https://github.com/lyft/amundsenmetadatalibrary https://github.com/lyft/amundsendatabuilder

We did a presentation in Strata SF 2019 last week which detail could be found at https://conferences.oreilly.com/strata/strata-ca/public/sche.... The slide could be found at https://www.slideshare.net/taofung/strata-sf-amundsen-presen... .