News for hackers. Stuff that matters.
Re:Ask HN: What is a broke person's way to learn data engine... by diehunde on Sunday May 05, 2019 @01:35AM (#19831348)
AWS, GCE, Azure, all of them offer credits for new accounts to experiment. You might be limited in some resources but you can learn a lot. Also, a lot of data tools such as Spark, Airflow, Kafka, etc can be deployed in Kubernetes, which can be run locally using Minikube. Just that and read a lot of blog posts.
Re:Ask HN: What is a broke person's way to learn data engine... by karterk on Saturday May 04, 2019 @10:53PM (#19830818)
Most people (regardless of where they are in the world) are not going to be able to afford a cluster of any meaningful size. So, the best option is to learn these tools by running them on your local machine. You can use Virtualbox or Docker to set up virtual hosts within your machine to simulate a cluster.

Also, data engineering is a vast field. Pick an area that you would like to pursue first and go deep: Machine learning (feature extraction+training), Hadoop/Spark job processing, event streaming & aggregation etc.

Then try to find a entry level job that would allow you to apply what you learned and also get to expand your knowledge of running actual clusters.

Re:Ask HN: What is a broke person's way to learn data engine... by usgroup on Monday May 06, 2019 @10:15AM (#19839578)
You need representative data; the sort you’ll be working with. There’s plenty of it for free on AWS. E.g. GDELT.

You then need use cases: things to do to the data. From this you learn how to process it using whatever tools you like.

How to set up clusters ... worry about that less. It’s more and more commoditised over time and it’s the admin part of data engineering anyway.

Re:Ask HN: What is a broke person's way to learn data engine... by analognoise on Saturday May 04, 2019 @01:04PM (#19827711)
We might be able to give you more targeted advice for your country.

I'm a massive advocate of community colleges over "boot camps" and other such educational vectors, but I'm not sure what infrastructure is already available in your country that would help you get on that path.

Re:Ask HN: What is a broke person's way to learn data engine... by pendergast on Saturday May 04, 2019 @01:09PM (#19827748)
Thanks for replying! I'm Indian. I only have access to a laptop at the moment. I'm not sure that educational programs in data engineering are so accessible to me, so I was looking to go the self learning route.
Re:Ask HN: What is a broke person's way to learn data engine... by whb07 on Saturday May 04, 2019 @04:10PM (#19829012)
what is data engineering to you? You don't need a cloud provider to setup a "cluster". Look at kubernetes where you can run a "cluster" in 1 machine that will have multiple pods/services running.
Re:Ask HN: What is a broke person's way to learn data engine... by Dikesm on Saturday May 04, 2019 @04:42PM (#19829260)
Hi, I can help you with logistics and learning. Can you ping me on [email protected].