News for hackers. Stuff that matters.

lenguyenthedat

Re:Ask HN: Who is hiring? (February 2016) on Monday February 01, 2016 @08:45PM (#11017165)
Singapore | Data Scientist / Engineer (Python / AWS / Docker / Airflow) | Onsite

https://honestbee.workable.com/jobs/168844

honestbee is the first concierge grocery delivery service for all your household and grocery needs in Singapore. Our mission is to be the largest online grocery and home goods store in Asia which boasts the largest variety and fastest market-to-doorstep delivery service available.

We are a technology startup with an aggressive growth plan to launch our business across all major cities in Asia. We are constantly searching for dynamic and motivated individuals to be part of our big family. You will be working with a high performing team that is constantly implementing new ideas and improving processes. If you have the passion for technology and thrive in a startup environment, we want you!

Currently offering unlimited supply of caffeine, great sense of accomplishment and endless creativity freedom to all selected talents.

Re:Data Infrastructure at IFTTT on Friday October 16, 2015 @12:11AM (#10397423)
Amazon Kinesis is a great tool, the main problem is that it's relatively new while Kafka have been around for a few years already.
Re:Building Analytics at 500px on Monday June 22, 2015 @11:36PM (#9762469)
Nice post! We went through a very similar process back in Zalora (http://www.zalora.com) building our own Data Warehouse and Analytics function for over 300 internal users at the time.

Back when Redshift was on its initial beta release (2012), there almost wasn't any ETL / charts tool available for us, so we ended up building most of the tools and libraries ourselves. A few of them were open source too:

- https://github.com/zalora/redsift A web-interface SQL tool for Redshift, letting the user to query and also export them to S3 (and send an email alert to user once done.)

- https://github.com/zalora/postgresql-user-manager for managing user privileges

- https://github.com/lenguyenthedat/aws-redshift-to-rds for copying Tables from Redshift to RDS (postgresql)

- https://github.com/zalora/kraken a bit similar to luigi or airflow.

Things has changed quite a lot since then, there are a lot of great solutions to our problems that are either free or very cheap and production-ready:

- redash.io (https://github.com/EverythingMe/redash) - web-interface SQL tool with visualization - FREE

- redshift_console (https://github.com/EverythingMe/redshift_console) - redshift ops tool - FREE

- flydata (https://www.flydata.com/resources/flydata-sync/sync-rds-mysq...) sync live data from MySQL to Redshift - Subscription base

- dreamfactory (https://aws.amazon.com/marketplace/pp/B00GXYDK18?sr=0-3&qid=...) for providing REST API interface to your database (supports redshift and other databases) - FREE

We didn't use Tableau in Zalora either (due to pricing and the number of users that we have in-house), and ended up building our own customized data dashboards with d3js and a few other different frameworks.

However, as long as you are ok with the price, Tableau is pretty good. It's being use widely in my current company (http://commercialize.tv) :

- You can minimize processing from Tableau server by just create another data mart layer from Redshift with your ETL tools / scripts, having Tableau connecting directly to it.

- Visualizations / Charts creation process is pretty much straightforward. The end-result will look exceptionally comparing to other solutions that we have tried.

- They also have a really good and active community.