I'd also be really interested to learn how you generate synthetic data. Is it a 'simple' case of manually generating as much as possible and then using the statistics of this to bulk it out? Or are you using something more complex to augment it?

The general class of model that you can use to create synthetic data is called a generative model [1]. The style of learning that pits one of these generative models against a discriminative one (the class you can't generate data from, including traditional neural networks) is called adversarial learning [2]. It's worth noting that normally you can use generative models for classification on their own as well.

[1] https://en.wikipedia.org/wiki/Generative_model

[2] https://en.wikipedia.org/wiki/Adversarial_machine_learning

One example of synthetic data generation was for our OCR project. We took a corpi of word choices (Project Gutenberg, modern books, the UPC database for receipts, etc.), took several thousand fonts, and combined it with geometric transformations that mimic distortions like shadows, creases, etc. to bootstrap millions of fake OCR like scannable documents.

We aren't using GANs yet, but are definitely keeping an eye on them. Work like InfoGANs which has the GAN learn a ground-truth like label are very promising, but GANs don't yet work at the image sizes necessary to really make this promising. I do think in the next year or two we will see these problems solved and GANs will become an integral part of synthetic data generation.

Ooh, neat! I've used GANs to generate synthetic temporal sequence data for training electrophysiologic (i.e., EEG, EMG) signal decoders. In fact, I wrote up the section of my dissertation on this topic today! In my experience it worked quite a bit better than other generative techniques (I've used convolutional variational autoencoders in the past for this and had so-so results). Looking forward to seeing what you guys do with this!

I am surprised that you are using AWS. It's generally cheaper and faster when you are at massive scale to build your own. Why do you do it this way?

in-house data turking setup

Can you clarify: Are you are turking with customers inputs or with internal team members?

synthetically generating millions of example data points

Are you implementing this via GAN or hand tuned?

You're right about scale, but I think forgetting that the model training is on very different hardware than what production web and storage servers use.

Unless the have the need for dedicated GPU hardware at scale paying by compute unit still makes the best business sense.

Disagree. That's the whole point of me asking. It's cheaper and will be better tailored to build your own GPU stack if you need it at scale than it is to "rent one" from the cloud providers.

I think it comes down to comfort and how critical it is to business processes.

What the heck is data turking?

Using something like Mechanical Turk to manually add ground truth data to something, though we don't use Mechanical Turk but rather other providers.

What are other providers that you use? Since you are using it as input to your training set, it should have fairly high accuracy (even more accurate than manual errors in tagging)

Ditto, are there any providers you are allowed to mention that you have had good experience with?

I think they mean manually labeling data en-masse. Maybe tagging photos with a description of what's in it, as a common example.

"Turking" is probably a reference to Amazon's Mechanical Turk service, where you can to get lots of low-effort work done by people.

I guess crowd sourcing ground truthing/generating more validated data with mechanical turk?

The crowd sourcing aspect is accurate, but I think the turk half is used more generically past the actual Amazon browser GUI experience tool to whatever annotation tool one has available or has invested development in. Stuff like https://www.crowdflower.com/

One other thing worth pointing out is why you can't always lean on the Amazon Turk per se because some of the tasks you need to train from require specialist knowledge that takes years to hone. It is not a task for the common layperson who can spot a cat in an image or anything else that exploits "common knowledge". In the case of the company I'm at we had to pay contract work for these specialists to come in house and operate our annotation tools to build up our ground truth data.

Yep, it's just a historical reference to Mechanical Turk since that was one of the first of such services. You are probably not going to be able to get minimum wage online workers in 3rd world countries to annotate medical images.

What was your data set like?

It was basically a bunch of microscope images of a certain kind of tissue.

Ditto on not being able to rely on some mechanical turk services - although the quality of answers/labels you get back inevitably depends on how well you have laid out the task at hand, we have additionally witnessed poor quality output that was outright rogue - tasks being completed in sub seconds or same answers given by a user no matter the question - both pointing to tasks being completed by bots not humans.

In fact we had resolved to building a turker bot detector and started rejecting tasks completed in suspicious ways. Only once we have built ourselves a trusted turkers population did we start to get quality data back. I suspect most people don't bother to go back and reject poor quality answers and that is why the bots survive.

Wow, I believe it. We didn't have so much a rogue situation, but you really do have to constrain their actions to just what you wanted. I tried to find the source but there was some YouTube video I watched where the guy made this good comment about creating GUIs where you have to put a real emphasis on preventing users from doing things you do not want them to do. You can't always focus on features but also constraints. I really took that message to heart after experiencing some of the human unpredictability found in building up training data. It made for some interesting payment debates to ask them to redo some work that was incomplete. Fun stuff.

I am studying ML this semester at university. I would like to get some ideas on where does a company like DropBox use ML. Can you state some problems that the team is working to solve or use cases?

Two examples of where we use machine learning at Dropbox is corner and edge detection in our mobile docscanner (https://blogs.dropbox.com/tech/2016/08/fast-and-accurate-doc...) and computer vision for OCRing.

Does Dropbox do OCR-ing? I couldn't find how to do that in the mobile app. I love the scan upload feature.

I really hope you are employing world class security folks and not gonna have another password leak. I have found after the scan feature, I'm uploading more of my paper life.

When using the Docscanner, we do OCRing of the images for Dropbox for Business users only right now. The OCRed Docscanned PDF gets copy/pasteable text, and its search tokens go into the search index for later searching. We only offer full text search for Dropbox for Business customers as well right now, not Pro.

> we never look at or train with real user data, unless explicitly donated

What is the rationale behind this? Why not train?

I can answer this. I'm a paying Dropbox customer and I didn't donate my data. The information is mine, and we can discuss using it, but my expectation of privacy includes Dropbox not using it for any purpose that isn't storing and serving it to me unless I've expressly stated otherwise.

While I'm not that interesting of a person, I do have an interest in things like my tax returns, financials, and other PII to not be in the hands of Dropbox employees.

Short answer: Dropbox is doing the right thing, and I really appreciate that.

Couldn't user data make it into the model, and be reverse-engineered from it, if you used it to train the model?

I mean, machine learning models aren't designed as one-way cryptographic hash functions, so you can't be sure that some of the training data can be inferred through the model's behavior.

Yes! there is an interesting paper on this subject: https://arxiv.org/abs/1609.02943

Counterpoint: https://arxiv.org/abs/1701.04082

Source: https://news.ycombinator.com/item?id=13824047

Preserving users' privacy first and foremost is absolutely the right thing to do here. I'm really glad Dropbox is willing to trade away a bit of model accuracy for better user privacy.

Because the ML models might start reproducing snippets of user data.

Perhaps they don't have the metadata for the data?

What does Dropbox use machine learning for?

See my reply to dnt404 for two examples. We have nearly an exabyte of data at Dropbox ( https://blogs.dropbox.com/tech/2016/03/magic-pocket-infrastr... ); there's lots of interesting ML things we can do with that, whether its searching, suggestions, etc.

You have an exabyte of other peoples private data that you're mining and essentially exporting to other files?

Am i misunderstanding or are you admitting to severely abusing your users trust?

See my original parent comment about not touching user data for privacy reasons :)

He says they don't use user data without explicit permission and they generate the training data using "turking," which means paying people to classify things manually.

Are there any public posts discussing how you augment data?

Also, what do you use you spin up multiple jobs in parallel using EC2?

No public posts yet unfortunately. For spinning up jobs we have an internal tool that can spin up machines and provision them, but in my own experiments I just usually end up spinning up machines and rsyncing experiments over.

Ah, fair enough. Too bad, I'd be really interested in hearing how you do that. I work for a consulting firm as a data scientist, and we have a lot of issues with data size. I'm aware of the tricks for image data (rotating & flipping, playing with different gamma levels), but I haven't seen much for standard, tabular style data.

I'm curious, how (if at all) do you verify how well your models are performing on the user's data?