News for hackers. Stuff that matters.
Re:Ask HN: As a data scientist, what should be in my toolkit... by ms013 on Tuesday February 20, 2018 @05:29AM (#16419238)
Mathematics. Which branch of math is domain dependent. Stats come up everywhere. Graphs do too. In addition to baseline math, you really need to understand the problem domain and goals of the analysis.

Languages and libraries are just tools: knowing APIs doesn’t tell you at all how to solve a problem. They just give you things to throw at a problem. You need to know a few tools, but to be honest, they’re easy and you can go surprisingly far with few and relatively simple ones. Knowing how, when, and where to apply them is the hard part: and that often boils down to understanding the mathematics and domain you are working in.

And don’t over use viz. Pictures do effectively communicate, but often people visualize without understanding. The result is pretty pictures that eventually people realize communicate little effective domain insight. You’d be surprised that sometimes simple and ugly pictures communicate more insight than beautiful ones do.

My arsenal of tools: python, scipy/matplotlib, Mathematica, Matlab, various specialized solvers (eg, CPLEX, Z3). Mathematical arsenal: stats, probability, calculus, Fourier analysis, graph theory, PDEs, combinatorics.

(Context: Been doing data work for decades, before it got its recent “data science” name.)

Re:Ask HN: As a data scientist, what should be in my toolkit... by p33p on Tuesday February 20, 2018 @08:27AM (#16420105)
And don’t over use viz. Pictures do effectively communicate, but often people visualize without understanding. The result is pretty pictures that eventually people realize communicate little effective domain insight. You’d be surprised that sometimes simple and ugly pictures communicate more insight than beautiful ones do.

I don't necessarily agree with this. Yes, a sound understanding of the domain and knowledge of the mathematics and statistics are vital to gaining insights. But. I would make a very clear distinction between exploratory data viz and explanatory data viz. Data visualization when presenting those insights is an important part of driving decision making.

Re:Ask HN: As a data scientist, what should be in my toolkit... by barskern on Tuesday February 20, 2018 @10:31AM (#16421047)
> And don’t over use viz. [...]

I don't fully agree with this neither. Especially for mathematical concepts, visualization can give insight into how theorems are constructed and combined. This can prove to be vital when applying concepts and theorems to new problems.

I would especially like to bring forth 3blue1brown[1]. He is a creator of videos which beautifully visualizes and explains complex mathematical problems. His efforts has given me an insight into math which theorems explained in text and variables could never do.

However I do see your point that visualizations without understanding can be misleading. Hence the pure, written math is important to read and reason about, but I do believe that some concepts need to be visualized to be fully understood.

[1]: http://www.3blue1brown.com

Re:Ask HN: As a data scientist, what should be in my toolkit... by biswaroop on Tuesday February 20, 2018 @12:41PM (#16422185)
I think what he/she means is poorly designed visualizations. Just because a plot is grayscale and not interactive doesn't mean it's worse than a cluttered poorly-designed super interactive web widget. It's a poor choice of wording, but I think by "overuse" they might mean "unclear but eye-catching". Besides "overuse" is literally the quantity that is excessive.
Re:Ask HN: As a data scientist, what should be in my toolkit... by gesman on Wednesday February 21, 2018 @01:35AM (#16427356)
Good viz is what connects non-ML/AI users to the "magical" results of ML/AI
Re:Ask HN: As a data scientist, what should be in my toolkit... by gaius on Tuesday February 20, 2018 @05:49AM (#16419329)
Indeed. Too many people when asked about their skills or experience just rattle off a list of tools or libraries. Usually the same ones as everyone else!
Re:Ask HN: As a data scientist, what should be in my toolkit... by anewhnaccount2 on Tuesday February 20, 2018 @07:18AM (#16419769)
Out of interest, can you give an example of a problem you've solved using Z3?
Re:Ask HN: As a data scientist, what should be in my toolkit... by ms013 on Tuesday February 20, 2018 @12:34PM (#16422121)
One data problem boiled down to being an instance of the set cover problem (https://en.m.wikipedia.org/wiki/Set_cover_problem). Pretty easy to pose as an integer constraint problem, and Z3 solved it in about 20 minutes for me.
Re:Ask HN: As a data scientist, what should be in my toolkit... by edem on Tuesday February 20, 2018 @08:05AM (#16419999)
I really like to get a degree in Mathematics but I simply don't have the time to throw at it (work, children, etc). What do you suggest I should do to have something on my resume? MOOC maybe?
Re:Ask HN: As a data scientist, what should be in my toolkit... by CoVar on Tuesday February 20, 2018 @09:47AM (#16420660)
Usually MOOC for resume don't help as everyone does them. The advice that I found useful for resume building is working on projects that you can catalog in a portfolio.

With regards to gaining math skills, this upcoming MOOC from Microsoft on EdX looks promising[1].

[1] https://www.edx.org/course/essential-mathematics-for-artific...

Re:Ask HN: As a data scientist, what should be in my toolkit... by petemir on Wednesday February 21, 2018 @07:26AM (#16428921)
Link didn't work for me. This one [1] did.

[1] https://www.edx.org/course/essential-mathematics-for-artific...

Re:Ask HN: As a data scientist, what should be in my toolkit... by edem on Tuesday February 20, 2018 @11:28AM (#16421506)
So you suggest that I should learn from MOOC then go on and work on some projects so that I can prove I really know it.
Re:Ask HN: As a data scientist, what should be in my toolkit... by CoVar on Wednesday February 21, 2018 @12:41AM (#16427145)
Exactly. And to take it one step further, choose one industry you are interested in. That way you will gain invaluable domain experience as you add relevant portfolio projects.

If you don't have an industry in mind, you can use a site like glassdoor.com and search for data scientist positions by city and industry to get a feel for demand.

Re:Ask HN: As a data scientist, what should be in my toolkit... by edem on Wednesday February 21, 2018 @03:29AM (#16427880)
Full disclosure: I'm in the industry for 10+ years as a programmer. I just realized that if I want to move in the AI direction I'll need some math education. I don't want to become a data scientist.
Re:Ask HN: As a data scientist, what should be in my toolkit... by gtani on Thursday February 22, 2018 @01:01PM (#16439686)
You could work thru one of the ML in R/python books e.g. Géron or Raschka/Mirjalili and then dig into the lin alg, prob, stats, calculus/analysis you see there with the books everybody recommends, LA by Axler, Strang or Proabilty by Bertsekas /Tsitsiklis, Real Analysis by Pugh, Abbott, Strichartz etc
Re:Ask HN: As a data scientist, what should be in my toolkit... by maayank on Tuesday February 20, 2018 @06:00PM (#16425086)
I have good background in graph theory (IMHO) but don't know many data science use-cases (I'm amateur at that). Could you point to some good start points?
Re:Ask HN: As a data scientist, what should be in my toolkit... by ms013 on Tuesday February 20, 2018 @08:24PM (#16426041)
Graphs show up all over the place. Social media: who is connected, which people interact. Cybersecurity: which computers/programs/users interact with which other computers/programs/users. Retail analytics: which products are bought with which other products; which products are more important in a graph than others.

Basically, any problem where you can establish relations between elements can be treated as a graph. I've used graphs for image analysis before too: pixels are vertices, edges represent neighborhood relations - especially useful when you make nonlocal connections (e.g., nonlocal means; graph-cut methods for segmentation; etc...)

I've worked with them in three of the above contexts: cybersecurity (my current projects), retail analytics, and image analysis. I've avoided social network stuff - never cared for that area much.

Re:Ask HN: As a data scientist, what should be in my toolkit... by Aardwolf on Tuesday February 20, 2018 @10:08AM (#16420848)
R is not present in your list, did you ever try it and what's your opinion about it?
Re:Ask HN: As a data scientist, what should be in my toolkit... by mojoe on Tuesday February 20, 2018 @11:15AM (#16421396)
R is good for one-off analysis problems, but is really bad for large distributed production systems. My team moved a large internal analysis application from R to Python because Python works well for both statisticians and software engineers.
Re:Ask HN: As a data scientist, what should be in my toolkit... by ms013 on Tuesday February 20, 2018 @12:36PM (#16422133)
I know R and have used it in the past. I just don’t like the language. I keep RStudio around though because on rare occasions I do look around in it to see if it has something I need. So rarely though that I forgot to list it...
Re:Ask HN: As a data scientist, what should be in my toolkit... by asafira on Tuesday February 20, 2018 @10:13AM (#16420906)
I think it is safe to say that they don't think it's too important for them given their main message, but hey, maybe they have an opinion anyway...
Re:Ask HN: As a data scientist, what should be in my toolkit... by throwaway7645 on Tuesday February 20, 2018 @10:33AM (#16421069)
If you have Mathematica, you might not need R as both are like Swiss Army Chainsaws for Data Analytics.
Re:Ask HN: As a data scientist, what should be in my toolkit... by throwaway7645 on Wednesday February 21, 2018 @07:17AM (#16428875)
Not sure y I was downvoted here. I've used both products in this problem area. Mathematica is definitely more than a CAS. Both are great in their own ways.
Re:Ask HN: As a data scientist, what should be in my toolkit... by uptownfunk on Wednesday February 21, 2018 @12:12PM (#16430993)
I think visualization can be a helpful tool to understand the data. I have seen some DS's get caught up in visualization for visualization's sake which I think can be wasteful.

I definitely think a solid mathematical understanding helps to build quantitative and critical thinking skills which are very key in data science.

Re:Ask HN: As a data scientist, what should be in my toolkit... by fhk on Tuesday February 20, 2018 @05:08PM (#16424692)
@ms013 interested to know how you are using the solvers, are you willing to share any further details?
Re:Ask HN: As a data scientist, what should be in my toolkit... by ms013 on Tuesday February 20, 2018 @08:27PM (#16426052)
Responded to someone else earlier about this. Used solvers for problems that end up requiring solutions to problems like minimum set cover or schedule optimization problems. Basically, problems where a naïve or brute force approach will take forever to run and you need to use a real solver to attack it. These usually are data problems that end up looking like what would traditionally be considered under the umbrella of operations research.
Re:Ask HN: As a data scientist, what should be in my toolkit... by Balgair on Tuesday February 20, 2018 @02:44PM (#16423482)
Honestly, in any STEM major, esp. the physics heavy ones, those maths areas should be well understood. Is any (physics heavy) STEM major also a Data Scientist then too?
[dead]
Re:Ask HN: As a data scientist, what should be in my toolkit... by squigs25 on Tuesday February 20, 2018 @12:10PM (#16421899)
Unless you do a masters degree in Data Science, AI, CS, Statistics, or a related field, a masters degree only serves as proof that you are intelligent and work hard. Someone without a masters degree can still have those attributes, but they would just have to prove it some other way.

For example, if you have a bachelors degree from a top engineering school (MIT, Cal Tech, Stanford, Berkeley, etc.) you have proven that you are intelligent and can work hard.

People without a masters degree, but more business experience, bring a different perspective, and are often more business results focused, and potentially work more collaboratively than an individual who just graduated from a masters program.

Source: I am a Data Science hiring manager, and have interviewed 100+ candidates at several companies

Re:Ask HN: As a data scientist, what should be in my toolkit... by dsacco on Tuesday February 20, 2018 @02:11PM (#16423091)
> Unless you do a masters degree in Data Science, AI, CS, Statistics, or a related field, a masters degree only serves as proof that you are intelligent and work hard. Someone without a masters degree can still have those attributes, but they would just have to prove it some other way.

I think the commenter’s point is (implicitly) about specific, directly relevant Master’s degrees. Obviously a general Master’s wouldn’t provide much of an advantage. The difficulty isn’t demonstrating intelligence and work ethic, it’s demonstrating targeted expertise.

> People without a masters degree, but more business experience, bring a different perspective, and are often more business results focused, and potentially work more collaboratively than an individual who just graduated from a masters program.

To be honest with you, this sounds to me like complete speculation. I’m not saying it’s wrong; rather it seems like it’s at best unempirical, and at worst unfalsifiable. The qualifiers you’re using (like “potentially”, or “often”) don’t seem like strong heuristics.

I think it would be helpful to discuss straightforward job descriptions. For most real data science roles, I would not weight any of what you’ve listed (except collaboration) as being remotely as useful as demonstrable expertise in computer science and statistics. For candidates without a Master’s degree, I wouldn’t take business experience or lack thereof as a signal whatsoever - I’d look for a relevant heuristic to replace it.

Re:Ask HN: As a data scientist, what should be in my toolkit... by minimaxir on Tuesday February 20, 2018 @12:38PM (#16422151)
During my recent data science job hunt, I received a lot of resistance due to my lack of a Master's/PhD: https://twitter.com/minimaxir/status/951117788835278848
Re:Ask HN: As a data scientist, what should be in my toolkit... by throwaway713 on Tuesday February 20, 2018 @03:36PM (#16423945)
I read your tweets and totally agree that data science interviewing is really bizarre. I have a PhD and currently work as a principal data scientist at a large company, but I've been interviewing with some other companies and the interviews are outright strange.

I've had questions ranging from reversing strings on a whiteboard to checking for valid email addresses. I had another question about flipping biased coins and calculating probabilities. It's all nonsense and totally unrelated to the skills I developed during my PhD which primarily consisted of performing massive amounts of machine learning on high performance computing systems over large sets of data to extract important insights.

But — if solving these algorithm puzzles quickly and without errors is the key to a $300k+ job, so be it. I'll just practice this nonsense until I've optimized for the skill of "interviewing", and then maybe I can contribute in some kind of meaningful way to the company with actual data science.

Re:Ask HN: As a data scientist, what should be in my toolkit... by mr_overalls on Tuesday February 20, 2018 @11:17AM (#16421416)
I'd call myself more of a "data plumber", with a double-BS in physics and computer science, but I'm considering returning for a MS so I'm more qualified to do interesting work.

Is there a consensus about what kind of Master's would be most useful data-sciency stuff? Computer science? Stats?

Re:Ask HN: As a data scientist, what should be in my toolkit... by sampo on Tuesday February 20, 2018 @11:55AM (#16421767)
> I'd call myself more of a "data plumber"

I think the actual term is Data Engineer.

Re:Ask HN: As a data scientist, what should be in my toolkit... by jebediah on Tuesday February 20, 2018 @11:43AM (#16421662)
Lots of universities have Data Science Master's Programs, which may ultimately fall under the Computer Science, Mathematics or Stats department. So, it depends on the university.
Re:Ask HN: As a data scientist, what should be in my toolkit... by mr_overalls on Wednesday February 21, 2018 @10:54AM (#16430318)
Data Science degrees seem to be such a hodge-podge of topics, with very uneven quality between programs. For long-term benefit, I'll admit I'm more inclined to tackle a traditional MS like CS, math, or stats.
Re:Ask HN: As a data scientist, what should be in my toolkit... by Jeff_Brown on Tuesday February 20, 2018 @12:28PM (#16422073)
I have needed this term for years. Thanks!
Re:Ask HN: As a data scientist, what should be in my toolkit... by booleandilemma on Tuesday February 20, 2018 @09:28AM (#16420503)
I think data scientist, much like software engineer, is something you can call yourself without having any credentials whatsoever.

It’s why technical interviews can be so brutal, unfortunately. There are a lot of frauds out there. Money attracts frauds.

What’s the fizzbuzz test for data scientists anyway?

Re:Ask HN: As a data scientist, what should be in my toolkit... by red_hare on Tuesday February 20, 2018 @10:24AM (#16420978)
I'm a data engineer for a startup that's trying to hire its first data scientist. The range of candidates that apply with this title is massive. Defining our expectations has been challenging.

My phone screen "fizzbuzz" is having them calculate a standard deviation from an array of data w/out with only basic operators (no numpy.std). Then explain why they choose population/sample and explain the difference.

I studied math in undergrad so one of my requirements is "knows more math than me".

Re:Ask HN: As a data scientist, what should be in my toolkit... by dsacco on Tuesday February 20, 2018 @01:00PM (#16422353)
> I studied math in undergrad so one of my requirements is "knows more math than me".

What kind of questions are you asking to ensure that they’re correct when they’re speaking about math you don’t know?

Re:Ask HN: As a data scientist, what should be in my toolkit... by red_hare on Sunday February 25, 2018 @02:16PM (#16460598)
This is a pretty hard problem I haven’t solved just yet. Generally, my in person interview is based on a set of DS problems I’ve been working on and had to do research myself to solve. What I look for is a strong intuition of the underlying math. Like, I can give them a formula and they can intuitively express what that means and explain it to me and then explain the next place they would take the solution. It’s not a perfect measure, but i’ve found the comfort with core concepts to be the most common trend among great data scientists I’ve worked with in the past.
Re:Ask HN: As a data scientist, what should be in my toolkit... by peatmoss on Tuesday February 20, 2018 @01:20PM (#16422535)
I think regurgitation of math formulas is a terrible way to hire for most data science positions. I've seen a breakdown of data scientists into two categories:

1) People who are great at the mathematics behind the statistical tooling

2) People who are great at conceptualizing a relevant question, operationalizing it, and then using a computer to apply appropriate models.

I think in most cases, for businesses needing to solve business problems, the latter kind is probably more useful. There are applications where the former is required, but you probably know if you need this kind of data scientist.

I should also add that these traits aren't mutually exclusive, but that individual data scientists typically are stronger or weaker along approximately those axes.

In general, I still dislike the term "data science" because it obfuscates meaningful distinctions between math nerds, computer science nerds, and research nerds who happen to do some applied stats.

Re:Ask HN: As a data scientist, what should be in my toolkit... by red_hare on Sunday February 25, 2018 @06:09PM (#16461766)
I actually agree with your breakdown. But, as a "data engineer" with a math background who's spent 5 years building analytics tools, I already identify as your type-2. We're working in a field that already has a rich history of established statistics that needs to be interpreted and broken down, so I think we're looking for someone who's a type-1.

I do, however, think anyone with some lick of statistics background should know the formula for a standard deviation. Considering how fundamental the idea of variance is in statistics.

Re:Ask HN: As a data scientist, what should be in my toolkit... by jerednel on Tuesday February 20, 2018 @11:30AM (#16421528)
At least with fizzbuzz you are working through how to logically solve a problem. This is just regurgitating a formula. I don't see how this is helpful.
Re:Ask HN: As a data scientist, what should be in my toolkit... by shepardrtc on Tuesday February 20, 2018 @11:46AM (#16421691)
Its designed to quickly weed out people who don't know the underlying math, just as FizzBuzz is designed to quickly weed out people who don't know programming.
Re:Ask HN: As a data scientist, what should be in my toolkit... by Xcelerate on Tuesday February 20, 2018 @04:04PM (#16424181)
I work as a data scientist, and my graduate research involved harmonic analysis over compact groups, optimization over Riemannian manifolds, and loopy belief propagation. You'd reject me in an interview because I couldn't remember the formula for standard deviation off the top of my head?
Re:Ask HN: As a data scientist, what should be in my toolkit... by red_hare on Sunday February 25, 2018 @06:12PM (#16461775)
It's not a hard weed-out for us. But, if you talk through variance for 10 minutes, you get pretty close to the formula for standard deviation.

Also, for our role, we're specifically hiring someone with extensive stats background since a large part of the role is learning domain-specific statistics of the industry we're targeting and figuring out how we can adopt those models with our data.

Re:Ask HN: As a data scientist, what should be in my toolkit... by shepardrtc on Saturday February 24, 2018 @08:44PM (#16456646)
Personally, I don't ask weed-out questions. Never have, never will. I'm just saying that's what they're doing.
Re:Ask HN: As a data scientist, what should be in my toolkit... by dsacco on Tuesday February 20, 2018 @01:02PM (#16422371)
No, using numpy would be analogous to a simple formula. Doing it without numpy requires actually understanding what’s going on.

It’s a filter that theoretically allows false positives (which is why you continue with other questions), but it really shouldn’t have any false negatives.

Re:Ask HN: As a data scientist, what should be in my toolkit... by Xcelerate on Tuesday February 20, 2018 @03:55PM (#16424120)
> My phone screen "fizzbuzz" is having them calculate a standard deviation from an array of data w/out with only basic operators

This is just my n=1 opinion, but this is a terrible test for data science skills. I've had to calculate standard deviation by hand many times in my life, but my short term memory is such that despite doing that dozens of times over the past two decades, I still can't recall the formula off the top of my head. And then there's the whole n vs (n-1) thing in the denominator which has something to do with degrees of freedom, but I would just Google that as soon as I needed to know (depending on exactly what I was trying to do with the data).

So I don't understand how your question in any way tests someone's skills at analyzing data to extract valuable business insights. At best, it tests someone's ability to memorize formulas and minutiae (although I'll grant you that understanding the difference between a sample and the population is important).

Personally, I think take-home interviews with real data sets are the best way to gauge a candidate's skills. You're actually testing them with a work sample, and they are not under artificial time or memorization constraints.

Re:Ask HN: As a data scientist, what should be in my toolkit... by arca_vorago on Tuesday February 20, 2018 @11:07AM (#16421339)
I have to admit this scares me just a little bit. I'm a senior sysadmin who is trying to lateral transition into data science, but I'm no math whiz, I'm just good at pragmatic use of tech stacks and have a generally analytical mind. If you are a math undergrad how could I ever expect to know more math than you? Of course a standard deviation should be easy, but your comment on math just stuck out to me.
Re:Ask HN: As a data scientist, what should be in my toolkit... by dsacco on Tuesday February 20, 2018 @01:06PM (#16422407)
> If you are a math undergrad how could I ever expect to know more math than you?

Read through, and do all the exercises in, one textbook each for:

1. Calculus

2. Linear Algebra

3. Abstract Algebra

4. Analysis

5. Topology

6. Probability Theory

7. Number Theory

...more or less in that order. Make sure your calculus book covers single variable and multivariable calculus. Supplement with applied mathematical statistics. Do that, and you have the equivalent of a mathematics undergrad (as far as relevant courses are concerned).

You could even do this with something like UIllinois’ NetMath program, or some courses on Coursera. You can swap out Number Theory for Complex Analysis or deeper Probability Theory and it’d be more relevant.

Re:Ask HN: As a data scientist, what should be in my toolkit... by alexbeloi on Tuesday February 20, 2018 @02:12PM (#16423092)
I would amend as follows:

Skip abstract algebra, topology and analysis. If you find yourself in the same room as a number theory book, walk away slowly without making eye contact lest it cast a spell on you.

Re:Ask HN: As a data scientist, what should be in my toolkit... by dsacco on Tuesday February 20, 2018 @02:16PM (#16423156)
If you skip analysis and at least elementary topology your understanding will be limited to discrete probability, at best. If you skip abstract algebra, you’ll miss out on a lot of buildup to advanced linear transformations and operations in vector spaces.

Sure, skip number theory. Like I said, you could swap that out.

Re:Ask HN: As a data scientist, what should be in my toolkit... by alexbeloi on Tuesday February 20, 2018 @03:02PM (#16423639)
You could also spend months of your life learning the intricacies of measure theory, but then every measure in probability is a positive sigma-additive unit measure.

One can learn the necessary topology, analysis (etc.) in the relevant places (and the relevant depths) that they come up.

Re:Ask HN: As a data scientist, what should be in my toolkit... by jeffreyrogers on Tuesday February 20, 2018 @01:44PM (#16422768)
That doesn't seem like a good use of time. I've tried reading through and doing the exercises in an abstract algebra textbook. It's a lot of work and the applicability to real world problems is virtually non-existent. I think a more targeted approach would give you a better return on your time.
Re:Ask HN: As a data scientist, what should be in my toolkit... by dsacco on Tuesday February 20, 2018 @02:01PM (#16422979)
Sure, I agree. Abstract algebra isn’t directly helpful. But:

1. The context is knowing more math than someone who has an undergraduate degree in it,

2. Abstract algebra is part of such a degree, and contributes significantly to overall mathematical maturity, and

3. You can avoid some subjects in the short term, but in the long term you can’t progress further without a reasonable mastery of algebra and analysis.

Probability theory and linear algebra are heavily used in data science. You won’t be as competitive a candidate for a job if you don’t have a firm grasp of both subjects. At a certain point, linear algebra ceases to be distinct from abstract algebra, and those exercises you were doing become applicable to real world results.

Re:Ask HN: As a data scientist, what should be in my toolkit... by arca_vorago on Tuesday February 20, 2018 @02:11PM (#16423088)
Thank you for this list, going on the todo, along with every other relevant comment on this thread. (emacs org mode is my ds notebook and todo app)
Re:Ask HN: As a data scientist, what should be in my toolkit... by red_hare on Sunday February 25, 2018 @06:21PM (#16461810)
Honestly, the market is so oversaturated with PHDs who are switching to DS I don't see how anyone can transition into it from a different role. I'm speaking for myself as well as someone with a math background, most people don't consider a math degree and 4 years of applying math models as a data engineering experience enough to be a "data scientist". They just weed out anyone without a PHD.

But, what you're describing I would consider "data engineering" (at least how I have been hired to do it). Working through the business problems and pragmatically facilitating data, pipelines, databases, and models to solve those problems. It's less established and less "hot" but, IMO, it's a much more valuable job to most businesses.

Re:Ask HN: As a data scientist, what should be in my toolkit... by shepardrtc on Tuesday February 20, 2018 @11:41AM (#16421644)
Maybe consider being a data engineer or a systems engineer? There's a pretty big demand for people that can set up, maintain, and assist the data scientists with the more complex tech stacks out there. In a former job as a systems engineer, I set up Hadoop clusters and helped manage data going into and out of it. And if you do decide to continue learning to become a data scientist, you'll already have a solid footing on the tech they actually use.
Re:Ask HN: As a data scientist, what should be in my toolkit... by arca_vorago on Tuesday February 20, 2018 @02:15PM (#16423128)
That might be an option to learn from, but it's not my end goal. As a senior sysadmin who was working for and reporting to PHD execs, I saw directly how what was needed was someone to do the data science and then bring convincing results and reports to the execs, essentially distilling the knowledge and wisdom of what needed to be done. I really want to fill that disconnect. (eg one of my failures as a sysadmin was me focusing too much on the technical, and now I want to expand and play the business board room politics game, but with data science)
Re:Ask HN: As a data scientist, what should be in my toolkit... by sah2ed on Tuesday February 20, 2018 @11:41AM (#16421648)
I think it is a reasonable expectation to require a certain baseline of expertise, considering the first thing the poster admitted to was: "I'm a data engineer for a startup that's trying to hire its first data scientist."

Much like how the early hires at Twitter were not deeply experienced in high availability work -- segregating the architecture of a predominantly RoR code base to be resilient at scale, which lead to countless "fail whale" outages, before they eventually landed someone who helped them re-think their architecture to use RoR for what it's good at while introducing the JVM and other languages to handle other aspects of their workload.

Re:Ask HN: As a data scientist, what should be in my toolkit... by Balgair on Tuesday February 20, 2018 @02:42PM (#16423458)
> ... of data w/out with only basic operators... (emphasis mine)

I'm having a little trouble trying to parse that sentence. Could you explain it better?

Based on what I think is being asked, the question is essentially: What is a STD? I think this is a very straightforward and fair question.

For less Stat-y HNers: For normally distributed data, the STD is the root of the Variance. The Variance is just the average of the square of the difference between the data points to the mean. Essentially: Take a point, find the distance to the mean, square that, average over all points you've done that to. That's the variance. Root the variance, that's the STD.

Re:Ask HN: As a data scientist, what should be in my toolkit... by charliej2 on Wednesday February 21, 2018 @05:51AM (#16428527)
This was useful info for a noob, thanks. It makes sense to me. If you understand SD in principle, you don't need to memorize the formula for a simple exercise like this, so it's a good filter for candidates who can demonstrate they do understand some basic math.
Re:Ask HN: As a data scientist, what should be in my toolkit... by telchar on Tuesday February 20, 2018 @01:29PM (#16422617)
I'm in that boat. I think a technical bachelor's plus work experience and self-study allows me to get along, but similarly to mr_overalls in the sibling comment I'm going back to school for a master's degree. It seems that getting into a data science career can be done without at least a master's but it seems that it would be hard to advance without one.
Re:Ask HN: As a data scientist, what should be in my toolkit... by cirgue on Tuesday February 20, 2018 @10:39AM (#16421121)
Yes, they do. Having a masters or not is orthogonal to being a quack, especially in an environment as buzzword-laden as data science/ML.
Re:Ask HN: As a data scientist, what should be in my toolkit... by red_hare on Tuesday February 20, 2018 @10:28AM (#16421015)
I agree that a graduate degree goes a long way. Even though the requirements of the role varies greatly from team to team, it almost always involves going out to learn and apply new math to solve a problem. That's one of those things a graduate degree (humanities as well, not just STEM) tends to require of you. And a Phd demonstrates you've done that for years.