Etai
Is an All-in-One Database the Future?
Just posting a thought experiment here.As data challenges become more defined and difficult, purpose-built databases have emerged to optimize specific problems, leading to a proliferation of specialized solutions. This has resulted in complex and messy data infrastructures, patched together by third-party data pipelines and event streaming products.Companies rarely rely on a single primary data...
Auto-populate your data dictionary with dbt metrics
Steps to managing and prioritizing data requests
As an early data hire at a fast-growing company, one of the first things that you’ll likely encounter is a backlog of questions from employees. This backlog, alongside all the other tasks associated with reporting, maintaining data and creating new pipelines can feel extremely overwhelming. These long lists of tasks and questions prevent many data teams from being proactive about analysis....
How to create a data catalog, a step by step guide
Simple data cataloging starts with a great organization. A data catalog is a collection of metadata and documentation that helps make sense of the data sprawl that exists in most growing companies. Getting together and starting to use a data catalog is a simple process, but starting to get adoption and having the dictionary exist as part of your workflow is a little bit more difficult.An...
We're also considering open sourcing that part of the product, but haven't made a firm decision on that yet. Would love to chat if you're open to it. We're definitely looking for people in product. Feel free to send me an email to [email protected]
We have pretty advanced RBAC in Secoda. You can make anyone a viewer, guest, admin or editor in the workspace. Viewers and Editors are only able to see the information. Secondly, we allow you to create "groups" for different functions in the organizations (ie. marketing, sales etc.). You can choose to share any resource with a specific user or group. This works similar to the RBAC that Notion uses, which only means that the right people are seeing the right information in Secoda. Lastly, we allow data teams to create "collections" of information, which can be shared with specific groups or specific users. Without sounding bias, I think this is where Secoda excels as a product.
2. What about PII? Some data needs to be stored, but cannot be viewed except for very, very few people and with a strong audit tail. This is a more specialized case for #1.
We have an ability to auto tag PII on a table and column level. Any PII data won't be viewable without permission from the admin.
3. How do you see the tool "spread" the most within companies? I would assume that easy sharing is how people learn about this, then try it themselves... but would love to hear what you actually see.
Usually the Slack integration is the best way to spread Secoda. With our Slack integration, any employee can search for information by pressing /secoda in Slack. You can also push information from Slack to Secoda and vice versa. This exposes Secoda to new employees in the place they work.
Launch HN: Secoda (YC S21) – Searchable Company Data
Hey HN, we’re Etai and Andrew, and together with our team, we’re building Secoda (https://www.secoda.co). We do search (that actually works!) for your company data. The name Secoda stands for "Searchable Company Data".As a company grows, so does its data. Tables, metrics, queries, and dashboards often become isolated and are difficult to find. Even with great practices, organizations still...
We're happy to help you or whoever would set it up and show you how other teams have been using the tool. Feel free to shot me an email at [email protected] if you'd like any help along the way
Companies store a growing amount of knowledge in BI tools, data warehouses, data pipelines, queries and documentation. Because these tools are not connected, it has become more difficult to manage all of this. Even with great practices, organizations still struggle to get value out of their data - up to 73% of all enterprise data goes unused. One of the big contributors to this problem is that organizations create data silos by not documenting and centralizing their data knowledge in a single place where every employee can access information about data.
Today, most data teams end up documenting all this data with Google Sheets or Confluence, which get outdated quickly. Because data documentation is outdated and hard to find, employees struggle to discover, understand and use it. This overwhelms data teams with repetitive questions about how to use and where to find company data.
In our last roles, Andrew and I had a hard time understanding context around different data resources. It was difficult to understand which table to use, what dashboard to trust, who to talk to about a particular metric or why we changed our pricing model. All of this data knowledge was in our data teams head and it made it really difficult to try to work with data. It would take around 2 weeks to get an answer to any data request because the data team was so backed up with questions. This sucked.
Secoda is unique because it's focused on helping the data team curate knowledge for the less technical employee. Data teams can use the tool to curate knowledge for specific departments or roles so that only the right people are able to see the data knowledge that they should see. We currently integrate into data warehouses, BI tools, dbt as well as Airflow and once teams connect their data to Secoda, they can get a comprehensive view of all their data knowledge in one place.
We’d love to hear your feedback or experience with the problem that we're solving and would be thrilled if you would sign up at https://secoda.co to let us know what you think!
How to Measure ROI on Data Discovery
Our team wrote this article about how to measure the ROI on data discovery as a tool to help teams justify the costs of data discovery and the benefits to management. You can check out the full article here: https://www.secoda.co/blog/measure-roi-on-data-discovery and calculate your own ROI on data discovery here: https://www.secoda.co/measuring-data-debtTLDRSolving data discovery starts by...
Table Lineage in Snowflake, Redshift and BigQuery
We just released a V1 of table lineage for Snowflake, Redshift and Bigquery, with support for Postgres, MySQL and Microsoft SQL Server coming soon. We wrote our own SQL parser to get this feature running so it's something we will be improving over time to handle more and more complex queries. If you're interested in trying it out, you can sign up for free at https://www.secoda.co
Staying Ahead of Data Debt
Problem 1: Data resources and BI tables were once created but unused by anyone in the organization. Problem 2: Everyone seems to have different ways of calculating similar metrics. Problem 3: My data documentation is outdated the second I finish documenting it.Data teams work as fast as possible to make sure that the business team is using the right information to make decisions. This pace and...
Show HN: A self service data discovery tool
Most Commonly Used Data Engineering Tools
There was a real appetite for data driven decision making across the company, but there was no good way to share tribal knowledge that had been built up over time, so people's intentions fell short. We realized that other mid sized tech companies were also creating a tremendous amount of data (we had over 500 different tables, visualizations and reports across the company of 25 people), but they didn't have a great way to make all that data accessible. This problem was so frustrating that we decided to leave our roles to solve this problem for other companies.
Secoda's product is an automated and collaborative data discovery platform that breaks down data silos within an organization and makes it easy to find, understand and use data. The product integrates with a company's existing data stack, such as Snowflake, Big Query, Redshift, Mode, Airflow, and builds up a graph of metadata that can be easily searched by employees in the organization.
We are building collaborative documentation tools such as discussions, tagging, and Slack integrations to make it simple to build up a knowledge base around company data.
Thanks for taking the time to stop by and we would love to hear any feedback you might have!
Andrew and Etai