That's a massive waste of your time and effort. The maintenance costs become massive as well should you choose to create your own system. At the VERY least you should leverage existing free open-source tools such as Metabase or Superset or Dash, or free tools such as Google Data Studio or Mode Analytics, if you're not going to spend cash to get a tool like Periscope Data/Looker/Tableau. I mean this gently, but you likely underestimate the complexity of a reliable reporting/analytics infrastructure. Think about it this way - these tools are either collaborated on by a large open-source talent pool, or are created by teams of dedicated software engineers just as talented as you.
I've worked with quite a few companies in an analytics consulting type role, and your "I can do the same thing with scripts" statement is one I've heard countless times. The long-term maintenance costs and technical debt (and "rigidity cost") of rolling-your-own analytics far outweighs the cost of a true analytics platform.
If you decide to roll-your-own anyway, look at tools like DBT and Airflow to reduce long-term maintenance costs.
Yeah I work at a company in the analytics space and see that all the time. It peaks the curiosity of people who are software developers (yet their core competency at their job is something else). They think its a fun work side-project and go after it. Write some python scripts to do ETL and process the data...make a backend with pg, a web server, then do charts in d3.js.
A year later they have a bunch of nice demos to their bosses but nothing that they can actually use in production because it crashes, there's no UI for interactive queries, no reports for people in the business groups, no user management etc. Then they drop it because they're busy with their actual job. So the cost of that engineers time to do something that didn't work was about $20-30k over a year. While the product they could actually use in production was around the same price.
If you want analytics directly on the stream, then there are plugins available to support reading the query results of something like Kinesis Analytics or Confluent's KSQL.
Grafana/Kibana are the actual charting tools
The auto-chart generation is nice. But what about Tableau makes it more likely to be accurate? Aren't you just as likely to make an error on the SQL in Tableau than if you didn't use Tableau?
The only exception to "using any of these is better than creating your own" is large companies like Google and Facebook, where they have entire teams of engineers who are dedicated to creating an in-house SQL+Visualization tools. It is absolute hubris for one engineer to think they can make a robust analytics platform!
Since people aren't typing code, it can be more accurate to use, and it provides visual results beyond just a table that can be useful in detecting anomalies in your data.
If you are good, you aren't "scripting", you are making a rad MVC system.
Salesforce puts all the MVC together in nasty, nasty ways.
If this is something you guys are interested in, I started a company called Retool (https://tryretool.com) that is essentially Excel for developers. Imagine if every Excel cell — instead of being a cell — were instead a React component. So you drag and drop these components around, and you can connect them to any back-end datasource (postgres, APIs, etc.). So you could drag on a table and have it pull data from `select * from users` from postgres, and then drag on a button and have it `POST` the selected row back to your API, in order to ban a particular user. The goal is to let end users build CRUD apps (like Salesforce) around their existing datasources quickly.
If you guys have any feedback... I'd really appreciate it. We're just starting out, and really curious to get any feedback from developers. Thanks!
That made it not viable for me. Building from scratch was way cheaper with Upwork. Anyway, Product was cool.
Do you mind if I email you? (I can't see your email in your profile, but my email is [email protected] if you're interested in reaching out.) I'm really curious — just to learn — what kind of pricing plan might work for you. Would a per-app model work for you, for example? Thanks!
It was a light hearted jab at the description that made it sound like something completely original and never tried before :)
I hope he does well, those tools are super useful.
If I had a dollar for every time someone showed me a wrongheaded graph they were using to make business decisions, I could retire.
Is that fully accurate, though?
Where does that leave data scientists / data analysts? I know SQL very well, and I know python's data stack (numpy, pandas, matplotlib, plotly, seaborn, various stats toolkits). I have a strong understanding of the "programming ecosystem" i.e. concepts, terms, definitions, and so on. I understand (basic) computer architecture, I've used and am familiar with (basic) shell/terminal, and services like Docker/Heroku on the command line, and can certainly use GUI cloud tools for AWS, GCP, etc. I can read and understand code and how systems fit together. I've worked alongside engineers of all types.
But I'm not a software engineer. I don't tell people I "program" because my strongest skill is SQL and generally people do not refer to that as "programming".
1. It's hard to get considered to be a SWE in the first place when your job titles are more in the realm of data analyst. They'll toss out my resume for a fresh grad, much less someone with experience, without a second thought.
2. If I were to make the switch I'd likely have to start at level 0 on the scale ladder. I've already career changed once into tech, and at this point I do not wish to "reset" my experience another time
Never start a land war in Russia, and never neglect your data infrastructure if data is in any way a key business differentiator/fundamental in your market.
Massive customization, and then you are bound by this broke ass Object model in which to get it all done, between Apex and VisualForce nauseating crap.
I don't want to pay hundreds of thousands of dollars for the right to do write database driven web pages.
I thought that the point of Tableau was to provide a tool that end-users (who can't program) could use to interactively explore their data. That's not something you can replicate with a bunch of scripts.
The issue in practice is almost always getting the data in a workable state so that you can manipulate it easily in Tableau. In my experience in smaller and mid-sized places, Tableau tasks get punted to analytics and data science, because they are needed anyway to get and transform the data in the first place. And these people usually prefer and are capable of using more technical tools than Tableau. I know I would rather use Shiny or Dash.
Maybe that's not a difficult problem in larger corps.
Except it's not just a few weeks of dev time that makes up the overall cost. Consider infrastructure, maintenance, updates, support, training, etc,. Those things start to add up and you don't get the benefit of scale/community if you do it on your own. There's also the opportunity cost of building your own system when you could buy something existing and use that time to work on other things.
And you should know, that the ratio between those two is something like 1:1,000,000
So, for the most of the world, tech is essentially "magic" to them.
To me, Salesforce looks like a big shared Excel file with a bunch of sheets. Tableau... well I can do the same thing with some scripts or spin up a web server.
To others, this tech is just magical. Pay the money, do the integration and it just works... And clearly people will pay a lot of money for things that "just work".