News for hackers. Stuff that matters.
Re:Ask HN: As a data scientist, what should be in my toolkit... by arca_vorago on Tuesday February 20, 2018 @11:07AM (#16421339)
I have to admit this scares me just a little bit. I'm a senior sysadmin who is trying to lateral transition into data science, but I'm no math whiz, I'm just good at pragmatic use of tech stacks and have a generally analytical mind. If you are a math undergrad how could I ever expect to know more math than you? Of course a standard deviation should be easy, but your comment on math just stuck out to me.
Re:Ask HN: As a data scientist, what should be in my toolkit... by dsacco on Tuesday February 20, 2018 @01:06PM (#16422407)
> If you are a math undergrad how could I ever expect to know more math than you?

Read through, and do all the exercises in, one textbook each for:

1. Calculus

2. Linear Algebra

3. Abstract Algebra

4. Analysis

5. Topology

6. Probability Theory

7. Number Theory

...more or less in that order. Make sure your calculus book covers single variable and multivariable calculus. Supplement with applied mathematical statistics. Do that, and you have the equivalent of a mathematics undergrad (as far as relevant courses are concerned).

You could even do this with something like UIllinois’ NetMath program, or some courses on Coursera. You can swap out Number Theory for Complex Analysis or deeper Probability Theory and it’d be more relevant.

Re:Ask HN: As a data scientist, what should be in my toolkit... by alexbeloi on Tuesday February 20, 2018 @02:12PM (#16423092)
I would amend as follows:

Skip abstract algebra, topology and analysis. If you find yourself in the same room as a number theory book, walk away slowly without making eye contact lest it cast a spell on you.

Re:Ask HN: As a data scientist, what should be in my toolkit... by dsacco on Tuesday February 20, 2018 @02:16PM (#16423156)
If you skip analysis and at least elementary topology your understanding will be limited to discrete probability, at best. If you skip abstract algebra, you’ll miss out on a lot of buildup to advanced linear transformations and operations in vector spaces.

Sure, skip number theory. Like I said, you could swap that out.

Re:Ask HN: As a data scientist, what should be in my toolkit... by alexbeloi on Tuesday February 20, 2018 @03:02PM (#16423639)
You could also spend months of your life learning the intricacies of measure theory, but then every measure in probability is a positive sigma-additive unit measure.

One can learn the necessary topology, analysis (etc.) in the relevant places (and the relevant depths) that they come up.

Re:Ask HN: As a data scientist, what should be in my toolkit... by jeffreyrogers on Tuesday February 20, 2018 @01:44PM (#16422768)
That doesn't seem like a good use of time. I've tried reading through and doing the exercises in an abstract algebra textbook. It's a lot of work and the applicability to real world problems is virtually non-existent. I think a more targeted approach would give you a better return on your time.
Re:Ask HN: As a data scientist, what should be in my toolkit... by dsacco on Tuesday February 20, 2018 @02:01PM (#16422979)
Sure, I agree. Abstract algebra isn’t directly helpful. But:

1. The context is knowing more math than someone who has an undergraduate degree in it,

2. Abstract algebra is part of such a degree, and contributes significantly to overall mathematical maturity, and

3. You can avoid some subjects in the short term, but in the long term you can’t progress further without a reasonable mastery of algebra and analysis.

Probability theory and linear algebra are heavily used in data science. You won’t be as competitive a candidate for a job if you don’t have a firm grasp of both subjects. At a certain point, linear algebra ceases to be distinct from abstract algebra, and those exercises you were doing become applicable to real world results.

Re:Ask HN: As a data scientist, what should be in my toolkit... by arca_vorago on Tuesday February 20, 2018 @02:11PM (#16423088)
Thank you for this list, going on the todo, along with every other relevant comment on this thread. (emacs org mode is my ds notebook and todo app)
Re:Ask HN: As a data scientist, what should be in my toolkit... by red_hare on Sunday February 25, 2018 @06:21PM (#16461810)
Honestly, the market is so oversaturated with PHDs who are switching to DS I don't see how anyone can transition into it from a different role. I'm speaking for myself as well as someone with a math background, most people don't consider a math degree and 4 years of applying math models as a data engineering experience enough to be a "data scientist". They just weed out anyone without a PHD.

But, what you're describing I would consider "data engineering" (at least how I have been hired to do it). Working through the business problems and pragmatically facilitating data, pipelines, databases, and models to solve those problems. It's less established and less "hot" but, IMO, it's a much more valuable job to most businesses.

Re:Ask HN: As a data scientist, what should be in my toolkit... by shepardrtc on Tuesday February 20, 2018 @11:41AM (#16421644)
Maybe consider being a data engineer or a systems engineer? There's a pretty big demand for people that can set up, maintain, and assist the data scientists with the more complex tech stacks out there. In a former job as a systems engineer, I set up Hadoop clusters and helped manage data going into and out of it. And if you do decide to continue learning to become a data scientist, you'll already have a solid footing on the tech they actually use.
Re:Ask HN: As a data scientist, what should be in my toolkit... by arca_vorago on Tuesday February 20, 2018 @02:15PM (#16423128)
That might be an option to learn from, but it's not my end goal. As a senior sysadmin who was working for and reporting to PHD execs, I saw directly how what was needed was someone to do the data science and then bring convincing results and reports to the execs, essentially distilling the knowledge and wisdom of what needed to be done. I really want to fill that disconnect. (eg one of my failures as a sysadmin was me focusing too much on the technical, and now I want to expand and play the business board room politics game, but with data science)
Re:Ask HN: As a data scientist, what should be in my toolkit... by sah2ed on Tuesday February 20, 2018 @11:41AM (#16421648)
I think it is a reasonable expectation to require a certain baseline of expertise, considering the first thing the poster admitted to was: "I'm a data engineer for a startup that's trying to hire its first data scientist."

Much like how the early hires at Twitter were not deeply experienced in high availability work -- segregating the architecture of a predominantly RoR code base to be resilient at scale, which lead to countless "fail whale" outages, before they eventually landed someone who helped them re-think their architecture to use RoR for what it's good at while introducing the JVM and other languages to handle other aspects of their workload.