> - If you're already employed with your agriculture PhD, there must be a number of opportunities for you apply the techniques that you're currently learning wihout leaving the industry. That's probably the path that I would suggest - it would allow you to expand your skillset without taking big risks and you'll have more options in the future. Use the career capital that you already have and explore your options instead of making a sharp turn in your career direction that might leave you disappointed.

This is real gold. If you have existing knowledge about some area than you can learn and apply those things, thus you'll get the traction that you want earlier, and it would be more rewarding in the end.

If there is a chance, I won't miss that opportunity. Good luck.

In general, domain knowledge is incredibly valuable and will generally trump someone who may have some more technical skills but doesn't know the field. I'd add that a lot of larger companies--heard someone from Shell talking about this a couple months ago--are training up people who have knowledge of the business with "citizen data scientist" skill sets.

Without knowing all the details of your situation, it seems at least a much lower risk path to acquire some data science skills--maybe your company will even pay for it--that you can pair with your existing domain knowledge.

Your comment will forever be under rated due to how spot on it is. It's a fundamental issue in tech that people don't appreciate. While yes, some problems are universal and general tech can solve them without industry experience, I'm pretty sure most people agree that those problems are either solved or there's an army of devs already on it. These days it's not really enough to just be a dev or data sci. You need to know a field and apply your tech knowledge in better ways than the uninitiated would never imagine.

The problem is that real world, physical business data is taken from what one can get, not what one would want. And usually "what the business was collecting for (unrelated purpose)."

This means it has nearly infinite caveats and assumptions. A specifications doc or readout will never sufficiently express all of these. Especially if humans were involved in the data generated.

Consequently, the most useful data products are going to turn on whether of not you (did this small thing) to (correct for this obvious bias or flaw that anyone familiar with the industry knows).

>The problem is that real world, physical business data is taken from what one can get, not what one would want.

Yea, sorry, but part of your job in data sci is to collect the right data. Data doesn't magically exist and we are not stuck with what's out there. A data sci job is to figure this stuff out. Tech has a weird culture of not doing their job. Kind of like the Zip Recruiter ads. "Working as a hiring manager, hiring new people is the worst part of my job." Bitch, that IS your job. If you dont do that, what's the point in keeping you around? Bee keepers collect honey. Yea it's not exactly easy if you're not careful, but they dont bitch about it because they knew what they signed up for.

Data sci/analysis is about collecting and analyzing data, in not straightforward ways. Because if it were easy and didnt require any effort, why are they needed?

You're both right. Sometimes we have to work with the data we have. Other times we have to create or buy the data we need.

Some companies aren't experienced at building infra to collect data and don't know how to do it. Or their environment is too complex or expensive to sample data from. The data scientist's job in such cases is to do their best with what exists, show success and make a business case for investing resources into data collection infrastructure.

In other cases, when the required sensors don't exist and the information is critical to decision making, you can either buy the data or work with with an engineering group or external vendor to integrate and build out the sensors needed. Need foot traffic data? You can buy from a data marketplace like https://datarade.ai, where there exist various vendors (like SafeGraph -- which was recently used in a COVID19 study published in Nature) aggregating foot traffic data from cell phones. There are datasets that can be used as inferential proxies (so called "alternative data") for the actual data one needs.

Need to collect in-store data? I was at the NRF conference (the world's largest retail tech conference) in NYC back in January and there were a boatload of vendors hawking different types of retail analytics sensors.

In certain small scale operations, you can even engage field operations and get the in-store retail staff to help collect data and upload manually. (you'll need a good relationship with the field supervisor of course)

Sometimes the data does exist but is inaccessible, say in the ERP or in some proprietary format -- then you have negotiate with certain business groups or with OEM vendors in order to get the data out.

It all boils down to whether the data has value that exceeds (by a margin) the cost of collecting them. If the answer is yes, there's often a way to do it (albeit sometimes imperfectly).

Is it part of the data scientist's job description to create or participate in creating data collection infrastructure? I guess this depends on the company but for many companies the answer is yes.

I agree with you too. I think it's a mix of exec and management dont fully grasp the job and its implications if you shortcut too much. At the same time, too many data sci are in it for the keyword/sexiness of the job and are not of the personality type to take hardline stands. Inexperience leads to a lack of trust from higher ups. A lack of backbone from the experienced results in performing more incompetence. Which results in more lack of trust. Experienced personnel leave, more inexperienced comes in and do things the cheap, shortcut, buy bad data way, plus no backbone to combat against this when seen... and you see how this can spiral into a shitshow that I've been noticing in some consulting projects I've been in.

But yes, data collection should be part of their job. I'm having a hard time understanding why the person who analyzes the data should have a good word at least in what data is collected.

Have you collected data from and deployed products to a 2000+ store environment?

Okay, how is my argument changed if I answer yes or no? Is what you're talking about a data sci's responsibility or not? If collecting, analyzing and deploying data in reports or db is too difficult for you, data sci isn't for you. I'm not telling them HOW to do their job. I'm clarifying that you have to DO the job if you signed up for it. Dont like it? Get out. We all screwed up by taking jobs we didnt like. Nothing wrong with that. Get out of the kitchen if you dont like the heat or the smell.

I'm curious about the relative experience you're talking from.

You're trying to make a point about the fact that you need to push the business and/or obtain data yourself, but I'm saying that can be a vastly more difficult problem than you think (or just flat impossible) at scale.

Why single data scientist should be responsible for obtaining data from 2000 stores? Data at scale requires people at scale. Data engineers would do this job.

There is a book "Range: Why Generalists Triumph in a Specialized World" which claims there are domain-specific problems that are more likely to be solved by people originally outside that domain.

I don't doubt there are examples where a fresh set of eyes and lack of knowledge about what can and can't be done can break out from "the way we've always done things." But it's probably not the way to bet in the general case.

The book claims that "generalist" are the rule at least if we look at the very top of certain fields e.g., Nobel laureates.

The data science professionals my company has had on staff have been a joke. It’s similar to the wave of inexperienced developers hired in the late 1990s; employers didn’t have the skills to prove that the contractors didn’t know what they were doing, so a ton of money was poured in with big dreams.

Part of that is that they don’t understand what we do and weren’t trained. They other part is that the business was just told they needed data people.

What businesses need today are to understand what they’re interested in first, and then hire people with proven experience and knowledge to accomplish those goals.

Having a PhD in any applicable subject in addition to sufficient practical knowledge of data science, machine learning, and AI would be a substantial plus for a niche position in the industry, a non-profit, or even in finance.

Don’t give up.

> practical knowledge of data science

This is key. What do companies actually want when they hire a data scientist? Actionable products that make their business better.

What does it take to produce actionable products? A data strategy (collection, ingress, normalize, enrich, store, expose), a compute provisioning strategy, data engineering (pull from source system(s), land in target stores in a reliable, available, automated manner), data science, and data application (reporting, integration with target systems, app development).

Which of those components do they typically have? A data scientist. Because they just hired one.

Career-wise, the more unifaceted your skillset is, the more you're limited to employers that already have all the other pieces in place. Which effectively limits you to very large enterprise (~T100).

Start to be able to fill some of the other roles yourself, and you can compete for and succeed in smaller and more interesting opportunities.

>the fundamental skills that you need are mathematics and software engineering

So much this. If I have to interview another junior-level DS who has a MNIST project in their github and still somehow can't manage fizzbuzz or a fibonacci function I'm probably going to take up religious asceticism.

EDIT: I said junior, but I meant Senior. We're talking people with PhD's who claim to have done extensive software engineering in previous roles.

Try filtering them on their specific ability to visualize things. My neighbor was 80+ year old math guy who could see graphs of equations in his head.

My hypothesis is that good stats people probably don't visualize (Aphant) or visualize very specific types of data in a unique way. Without visualization, people tend to fall back to logical thinking - or emotional thinking, depending.

For example, I have a friend who can look at 2D seismic data and see what the underground formation looks like in his mind, in 3D.

The person you are replying to seemed to be complaining about their lack of programming ability, not statistical ability, so how would this help in their situation?

Yup, I think I and that child are talking past each other. :-(

It's good though because they raise some really valid points about the importance of intuition. I joke with my colleagues that all we're doing is encoding data as a hilbert space and slapping an algorithm on it, but that elides the fact that intuition like the child is talking about is important for knowing how to build that hilbert space.

I think there is an issue with focus of the role. If job is more focused on programming then the candidate must be at least proficient in coding. Otherwise if you need a mathematical modelling person then you need to look into relevant training background (undergrad degree etc). A lot of my colleagues in my research institute are from a physics background. Because molecular biology require a lot of statistics. They are ok coders but what they really contribute is the modelling part.

If I have to interview another whose only ML tools are GLMs, random forests and boosted trees (only ever with one hot encoding, of course) I’m going to do the same.

Those and SVR's get me through 99% of the algorithmic part of my job, though!

The rest is some unsupervised stuff like k-means and PCA.

What would you like to see instead? (INB4 CNNs/RNNs other deep learning topics)

As a software engineer who has worked intimately with data scientists in Big Tech companies, this comment sums it up exactly.

One minor thing to add is that not only is the data science field overflowing with junior/mid candidates, but the entire computer science field as a whole.

What happens in seven years when those candidates become experienced? Will they have dropped out? Will there be a glut of experienced software developers?

I sincerely doubt a large portion of them are cut-out for the sector at all. That's the issue.

> There is a big oversupply of people for the junior-mid level data science jobs.

Honest question is there any field where that isn’t true right now? Seems like there are no junior jobs in any sector.

German here. From my perspective it's hard to find

- embedded systems programmers, meaning people who know real-time systems, have good C/C++ knowledge, know their way around the Linux kernel and are also able to do basic things with a scope.

- good(!) C++ programmers in general

- people who know devops and software development infrastructure

C++ might not be sexy, but there's a vast amount of legacy software out there which is not going away anytime soon.

It's funny because I have all these (though it's been a few years since I've been doing C++ development daily), yet if I were laid off tomorrow I know from experience it could take me a whole year to find a new job in any of that domain.

By the way junior C++ developer is an euphemism because no company will hire a junior C++ developer. It's the most difficult programming languages that can take years of practices/studying to be able to use.

That is true. I found that for embedded C++ to be successful, you need to have some good (excellent sometimes was harmful, but that's a different topic) C++ _as well as_ good domain and optimization skills. It's a combination that is more and more scarce. C++ tends to be used in domains where size and speed matter. You need to know what the compiler and machine will do with your code to be fast. And what's more, only the domain knowledge will let you pick the right data structures to be fast and compact in the first place.

And there are a lot of foot guns around... ;-)

> I found that for embedded C++ to be successful, you need to have some good (excellent sometimes was harmful, but that's a different topic)

Why were excellent C++ skills sometimes harmful?

First of all, it leaves the team behind if you use features that are too advanced. Makes collaboration hard if you have wildly diverging skills (in both ways).

Also, it did in our case take you away so far from the bare metal, that the code was elegant, but slow. It did not play well with our static allocators and allocated/deallocated way too much, especially temporaries.

You might i.e. check out talks like [0].

[0] https://m.youtube.com/watch?v=NH1Tta7purM

I agree. Another thing to consider is portability. You might have to port your software to obscure platforms with bad C++ compilers where compiler bugs are not uncommon at all, so it's better to stay clear of more advanced features. Also, you might have a legacy system which does not have modern C++ compilers, so you might have to restrict yourself to C++03. Another thing is code size, so templates should be used judiciously.

These replies both don't seem like they describe someone with "excellent" C++ skills. Sounds more like someone who knows a lot of advanced features of the language, but doesn't make the correct tradeoffs considering portability and/or performance.

Maybe it could be that he didn't know all the requirements up front and his mediocre colleague accidentally wrote code that better suited the company's unstated requirements (e.g. portability to niche systems). But that's hardly the most likely explanation.

Ah, ok. In my head I do have a distinction between excellent C++ and excellent domain skills. So that person had excellent C++ skills but did not properly choose which C++ subset to use for the domain. From your comment I think you would say "excellent" implies "in all relevant aspects", whereas I just meant the pure programming language. What I tried to write in my first comment is that it is more and more difficult to find both combined in one person.

I think for C++ programmers either you have to pay a premium and hire senior people in their 40s (like 60s and 70s for COBOL), or find someone who is willing to learn and have the patience to get him on road not now, but 3-6 months from now.

I feel that it's rare that companies choose the second road so usually they pay a premium.

But all three are by no means junior.

I live in Amsterdam. What I'm seeing here is that the security field is lagging behind the data science / web dev field. It's not easy to find junior jobs, but a lot easier than data science / web dev.

The most fun example that I currently have is that my LinkedIn profile is super wide. I market myself as a generalist software dev, because that's what I am. I'm not a specialist and I've dabbled in almost everything.

Naturally, no recruiter contacts me because of that. But you know who does? A European company looking for a reverse engineer.

I found it quite surprising that they were capable of reading that I have some IDA Pro experience, because it's in the fine print of my profile and not at all readable when you skim it.

> I found it quite surprising that they were capable of reading that I have some IDA Pro experience, because it's in the fine print of my profile and not at all readable when you skim it.

Wouldn't searching profiles for "IDA Pro" find it?

I guess, but I wouldn't be on the top of the list as I am mentioning it once and there's also a lot more mobile and web buzz words on my profile.

They don't read your profile at first. Sometimes they don't even read it at all before contacting you.

They search for keywords, and rely on the search algorithm prioritisation and a good looking one-line summary of your profile to pick you out of pages of search results.

They'll find your "IDA Pro" if that's in the list of terms they are looking for, no matter how deeply it's buried in the description of your past jobs. If they don't read your profile they might not even know that it's buried.

one factor here is that 'senior' doesn't really mean anything anymore other than 3+ years of experience.

"There is a big oversupply of people for the junior-mid level data science jobs."

I feel this is true for most tech jobs. I see tons of senior job postings, but relatively very few entry-level or mid-level postings.

I don't exactly agree on why employers prefer PhDs even if not exactly in the field.

Having PhD shows that you have committed to one field hard enough and long enough to be granted the title even if it is PhD in philosophy.

Also, if you switch from another career try to figure out how you can use your past experience to differentiate yourself from other people.

For example, I have studied theoretical math and I have spent 20 years as software developer working on varied projects from embedded to algorithmic trading to backend to frontend applications.

I am currently switching to electronics and will use my knowledge to work on projects that require high level of both math and software engineering skills. Think in terms of realtime control systems, building reliable and performant stuff (reliability to differentiate from competition), etc.

Never think of moving as rebooting your career. Always start with mindset of building on top of what you know -- always move forward, not backward.

A little change in mindset goes a long way.

Could you please say more about how you find maths-intense software engineering niches and projects? What niches are there?

I too am a pure maths PhD with a background in software, but I am somewhat disappointed with the data science world.

I actually think that we agree on the PhD requirement.

> Having PhD shows that you have committed to one field hard enough and long enough to be granted the title even if it is PhD in philosophy.

Yes, exactly. That's what I meant by "strong initial filter".

This reply should be made into billboard and placed in front of every "Data science bootcamp" out there.

> Depending on your background it might take years of additional studying.

Excellent summary but this one bit is trying to gloss over reality. It will most certainly take multiple years of additional studying when you come in contact with anything related to software engineering, because that's just what it is.

Of course there is no shortage of people who learn one language, one framework, one tool and sit on their ass depending on the people they work with to pick up the slack. If that's you, none of the things being said in this thread matter - just go get your job, make your money and live your life :)

I slightly disagree about your reasoning on the PhD thing, for some roles. Having a PhD in an experimental science is a (weak) indicator that you’ve learned how to design, conduct, and analyze an experiment on your own.

What kind of math? It takes me a while to learn math so any focused area would help cut down the time taken immensely (not that I don't enjoy learning tidbits like supremum/infimum)

Multivariable calculus, differential equations, linear algebra, probability theory, statistics.

Ranked from most to least essential, I'd suggest: probability, statistics, linear algebra, basic calculus, operations research, multivariate and partial and vector calculus, diff eq.

True but order of importance is different from order of study. You should really start with calculus because you can't study probability without it.

>There is a big oversupply of people for the junior-mid level data science jobs.

Which shouldn't be too much of a problem because they have the skills to analyse the market and find a local optimum that suits them.