News for hackers. Stuff that matters.

jeremystan

Re:Deep Learning with Emojis (not Math) on Sunday May 07, 2017 @12:12PM (#14286112)
OP here, happy to answer any questions
Re:3M Instacart orders open sourced on Wednesday May 03, 2017 @05:23PM (#14260248)
OP here, happy to answer any questions
Re:Using Deep Learning to Pick Groceries Faster on Wednesday March 29, 2017 @02:21PM (#13989101)
hey, i'm the OP and would love to discuss / answer any questions

also, another shout out to the OpenAI and YC teams for their great feedback

Re:Data Science Challenges at Instacart on Thursday February 18, 2016 @12:23PM (#11127449)
To persist sparse models to the DB, especially if you use L1 regularization (like Lasso) then many coefficients will be 0, and don't need to be stored or processed. Insofar as you store coefficients and features in a "tall" format (e.g., user, feature_key, feature_value), then space is conserved. Scoring can be done in DB with joins and group by, or in another language with similar operations.

Changing variables frequently can be versioned in the feature and model coefficients tables, but takes care.

I haven't used Postgres JSONB, but if you have problems with JSON in R check out the tidyjson package (I wrote when dealing with Mongo data previously).

Scoring in another language is best avoided if you can. But supporting R "real time" services will also come with many complications. Hence, we use Python when we really want that.

SparkR was completely unreliable when I first tested it over a year ago, but may have improved. Though the Spark Python API has some limitations compared to Scala, so I would guess the latest SparkR is even further behind, but we haven't tested it. Long term I'd love for that to be the answer to these questions.

Re:Data Science Challenges at Instacart on Thursday February 18, 2016 @09:34AM (#11125960)
Agreed, that wasn't worded well - i'll try to explain. By 'employees in the field' we mean people who work in operations and management roles in the cities we operate in. They work with shoppers in stores and respond to shopper and customer feedback. They have ideas about how to improve our logistics and our apps - and while many ideas will be about user interfaces, many others will relate to how the algorithms operate behind the UIs.
Re:Data Science Challenges at Instacart on Thursday February 18, 2016 @09:24AM (#11125893)
We use R in production in two ways:

1. For batch processes that run daily, hourly or minutely, where the models are rebuilt on every run, and outputs (often predictions) are written to a database 2. For computation of coefficients in large sparse regularized models, where the coefficients are written to a database and scoring is done in another language in real-time

For situations where we want real-time predictions, recommendations or optimizations, we tend to setup Python services instead. For batch processes, you can definitely store models in S3 to re-use them, and I've done that at other companies. But in general I've found it better to rebuild models frequently and cache them for short periods of time only if they are cost-prohibitive to rebuild.

Re:Data Science Challenges at Instacart on Wednesday February 17, 2016 @04:55PM (#11121756)
re-reading what I wrote I can see how it comes off as canned - thanks for the feedback
Re:Data Science Challenges at Instacart on Wednesday February 17, 2016 @04:05PM (#11121392)
I find that great data scientists care a tremendous amount about the problems they tackle and the impact they have, and less so the specific methods used. But we will definitely write more in the future on details. Here I wanted to share the range of problems, how we organize (not commonly discussed) and what we look for in candidates. That can be useful to other startups looking to build data science teams.
Re:Data Science Challenges at Instacart on Wednesday February 17, 2016 @04:02PM (#11121366)
We've worked hard to open source projects whenever we think they'll be useful broadly: https://www.instacart.com/opensource. There is definitely more of this we can (and I hope will) do in the future.
Re:Data Science Challenges at Instacart on Wednesday February 17, 2016 @03:58PM (#11121333)
We work hard to get back to candidates quickly - in some cases the same day as they interview, but at least the day after if not. We know the market is very competitive and want candidates to make the best decisions they can - so it's in our best interest to act quickly! We are always working to improve how we screen, interview and respond to candidates in hiring, so will take this feedback to heart. Thank you for providing it.

Regarding the focus on OR, that was definitely the case for our first few years, and while it's still important, we have definitely expanded our focus beyond it.

Re:Data Science Challenges at Instacart on Wednesday February 17, 2016 @02:53PM (#11120717)
Behind the scenes, Instacart is a revolutionary new e-commerce marketplace, an incredibly sophisticated last-mile logistics engine, and a dynamic source of work for thousands of personal shoppers. Each of these aspects of Instacart could be a whole company elsewhere, and data science plays a key role in our success in each endeavor.

In this article, I (our VP Data Science) highlight some challenges the data science team is tackling at Instacart ranging from logistics to personalization. I also go into detail on how we have organized data science to have maximal impact and what we look for when recruiting data scientists.