News for hackers. Stuff that matters.
Re:Everyone wants to do the model work, not the data work [pdf] by sanxiyn on Sunday March 28, 2021 @10:48PM (#26617201)
Real story. I worked on nudity detection for mobile realtime video chat app with on-device deep learning model. (It was successful.) Should be easy: label some data, start from a good network pretrained on ImageNet (we used SqueezeNet), fine tune. The problem was that ImageNet is photos, not video frames, and distribution of video frames from mobile video is somewhat different. Incredibly large proportion of frames are either blank (camera is blocked) or ceiling (camera is misdirected). We ended up adding lots of data for blank and ceiling explicitly labeled as blank and ceiling. It became an excellent ceiling detector, detecting ceilings it never have seen before.

Is there a mobile video frames dataset for labeling ceilings? Why not? I can't believe I am the only one who experienced this. Why is not a ceiling dataset worthy of, say, CVPR? It will improve mobile video on-device deep learning more than most of CVPR papers. This is a serious problem.

Edit: I understand why niche datasets remain in industry and not academia. Public datasets are better if they are of general interest. But ceiling dataset is of general interest to anyone who wants to process video frames originating from blocked or misdirected camera aka smartphones, and it's hard to imagine topics of more general interest once you have obvious things like face.

Re:Everyone wants to do the model work, not the data work [pdf] by sillysaurusx on Monday March 29, 2021 @12:34AM (#26617739)
Little crafty trick: grab the flickr dataset (yfcc100m) and snag photos tagged as whatever you want. It's like a janky google image search. I've put together datasets of airplanes, bicycles, dogs, etc this way.

It's not entirely accurate, but it's good enough. Within a few hours you can have a pretty large dataset of whatever you want, really. (Yay for massive dataset plus user tags.)

You can snag a copy from my server, if you want. (Warning: It's a direct link to a 54GB json file.) https://battle.shawwn.com/sdc/f100m/yfcc100m_dataset.json

I have a script called "janky-image-search" that returns random results from searching this dataset. Here are a few random hits for `janky-image-search ceiling`:

https://www.flickr.com/photos/35981213@N00/9374577304

https://farm3.staticflickr.com/2084/2513539756_7a768de44c.jp...

http://www.flickr.com/photos/96453841@N00/7077625757/

https://www.flickr.com/photos/28742299@N04/4683245369/

etc.

The quality is hit or miss, but it seems better than 70%.

EDIT: Here's a gallery of the first 100 hits for "ceiling": https://cdn.gather.town/storage.googleapis.com/gather-town.a...

But as you can see, it's not effortless. Most of those ceilings are old. So it depends what you want. It's why labeled data is worth billions of dollars (scale.ai et al).

Re:Everyone wants to do the model work, not the data work [pdf] by willvarfar on Monday March 29, 2021 @01:03AM (#26617872)
Super neat!

(Small detail: the first 100 hits for ceilings are all _interesting_ ceilings? Nothing like the kind of ceilings usually encountered on a video chat app?)

Re:Everyone wants to do the model work, not the data work [pdf] by caslon on Monday March 29, 2021 @03:53AM (#26618697)
Why would anyone post something boring on a photography site? There's a bar for quality on flickr; it's not twitter.
Re:Everyone wants to do the model work, not the data work [pdf] by mcv on Monday March 29, 2021 @05:26AM (#26619181)
Yeah, flickr might not be a good data set to train for boring images. Maybe we need a high quality repository of bad photos.
Re:Everyone wants to do the model work, not the data work [pdf] by johnmoberg on Monday March 29, 2021 @04:06AM (#26618758)
Of course, the point is that it may not be good data for training a model.
Re:Everyone wants to do the model work, not the data work [pdf] by caslon on Monday March 29, 2021 @04:09AM (#26618779)
I was clarifying, since the above poster was using question marks.
Re:Everyone wants to do the model work, not the data work [pdf] by sanxiyn on Monday March 29, 2021 @12:50AM (#26617803)
I found that people don't upload misdirected ceiling photos and videos to YouTube and Flickr. How unfortunate. Yes, it's better than nothing.
Re:Everyone wants to do the model work, not the data work [pdf] by sillysaurusx on Monday March 29, 2021 @02:13AM (#26618198)
Now you've made me want to create a random-ceilings dataset. I spent about an hour trying to think of some way to get you a bunch of un-curated ceilings, to no avail.

If you ended up using the ceilings uploaded to your app directly from your users, it might be possible that there wasn't any better solution than the one you came up with.

Re:Everyone wants to do the model work, not the data work [pdf] by tdeck on Monday March 29, 2021 @02:54AM (#26618436)
Perhaps if there's a database of those 360 photo globes you could crop out the upward-pointing part?
Re:Everyone wants to do the model work, not the data work [pdf] by IAmEveryone on Monday March 29, 2021 @01:51AM (#26618091)
That dataset will work well for detecting nudity in Zoom calls from the Sistine Chapel :)
Re:Everyone wants to do the model work, not the data work [pdf] by teruakohatu on Monday March 29, 2021 @01:54AM (#26618104)
Before I download 54gb from your server, can you tell me the structure of the json?

What is a good tool to search it without 64gb of free memory?

Re:Everyone wants to do the model work, not the data work [pdf] by sillysaurusx on Monday March 29, 2021 @02:10AM (#26618181)
My janky-image-search script looks like this:

  tag="${1}"
  shift 1
  curl -fsSL https://battle.shawwn.com/sdc/f100m/yfcc100m_dataset.json | jq '{user_tags, machine_tags, description, title, item_download_url, item_url}' -c | egrep -i "\"($tag)\"" "$@"
Basically, it trades bandwidth for memory. And hetzner has free bandwidth.

Here's curl -fsSL https://battle.shawwn.com/sdc/f100m/yfcc100m_dataset.json | head -n 1, which should give you all the info about the structure. Or most of it.

  {"date_taken": "2013-08-10 11:05:54.0", "item_farm_identifier": 3, "item_url": "http://www.flickr.com/photos/9315487@N04/9497940823/", "user_tags": [], "title": "IMG_7216", "item_extension_original": "jpg", "latitude": null, "user_nickname": "meowelk", "date_uploaded": 1376370010, "accuracy": null, "machine_tags": [], "item_server_identifier": 2856, "description": null, "item_secret_original": "15f4257122", "user_nsid": "9315487@N04", "license_name": "Attribution-NonCommercial-ShareAlike License", "capture_device": "Canon EOS 7D", "item_marker": false, "item_id": 9497940823, "longitude": null, "item_download_url": "http://farm3.staticflickr.com/2856/9497940823_0c0d854111.jpg", "license_url": "http://creativecommons.org/licenses/by-nc-sa/2.0/", "item_secret": "0c0d854111"}
I just dump those to disk and then use `jq` to extract the item_download_url.

FWIW, the full ceiling search finished with 9,325 results. Here's the first 1k.

https://cdn.gather.town/storage.googleapis.com/gather-town.a...

Re:Everyone wants to do the model work, not the data work [pdf] by teruakohatu on Monday March 29, 2021 @03:51AM (#26618685)
Thanks for the information. So the 54 gigabytes is just plain text JSON? How big is the full image dataset?
Re:Everyone wants to do the model work, not the data work [pdf] by sillysaurusx on Tuesday March 30, 2021 @12:19AM (#26631226)
Good question. It's 100m images, but I haven't tried to find how large all of it is.

We downloaded a subset of 3 million images, which is apparently 379.77 GiB. So a linear extrapolation would be (100/3*379.77) = ~12,659 GiB for the full 100m images.

12TB really isn't too bad. It's massive, yes, but imagenet 21k is 1.2TB.

Re:Everyone wants to do the model work, not the data work [pdf] by joncrane on Monday March 29, 2021 @10:53AM (#26622608)
Wow I immediately recognized that first one as the ceiling of a Washington DC Metro station.
Re:Everyone wants to do the model work, not the data work [pdf] by nxpnsv on Monday March 29, 2021 @02:35AM (#26618314)
I don’t need it, but that’s really awesome of you!
Re:Everyone wants to do the model work, not the data work [pdf] by ethbr0 on Sunday March 28, 2021 @11:17PM (#26617360)
Because fewer PhD are minted by "Improved on SotA ceiling detection rates by 50%."
Re:Everyone wants to do the model work, not the data work [pdf] by sanxiyn on Sunday March 28, 2021 @11:30PM (#26617434)
Maybe not PhD, but as a research topic, I think doing this well is an interesting task. I learned that ceilings are not featureless. (I know, it's a kind of ridiculous to say I "learned" this, since I could have learned this from, oh well, looking at my ceiling just now, but it really was a surprise when I encountered this in machine learning data context.)

Ceilings often have lightings! Lightings (either on and off) vary between geography and culture! (They do! Extreme variations!) Lightings, if on, will cause flares! We just sampled from actual distribution (privacy policy included we can monitor connection for anti-abuse and we can use data for research and development of anti-abuse), but dataset collected with care will be valuable. I am sure actually doing this will reveal other considerations, and even just with points I mentioned, it should be good enough for a paper, if not for PhD.

Re:Everyone wants to do the model work, not the data work [pdf] by ethbr0 on Sunday March 28, 2021 @11:32PM (#26617441)
That sounds like the problem in a nutshell. "X is simple, so just gin me up a data set of it." (Four months later) "X is not simple, in all these ways..." Which is another way of saying that discovery and communication with SMEs is an important side skill in project success.
Re:Everyone wants to do the model work, not the data work [pdf] by bonoboTP on Monday March 29, 2021 @06:56AM (#26619875)
What would be the novel scientific contribution over any other image classifier? What's the storyline? How do I get excited? You collected a dataset for some boring class and trained a standard sota model on it. This is an undergrad project report, not a research paper.

The main question people ask when considering reading a paper is "So what??"

Re:Everyone wants to do the model work, not the data work [pdf] by notsuoh on Monday March 29, 2021 @01:02AM (#26617868)
Not an ML person, but I generally understand the concept. Why do you care about ceilings here? I get that the camera is pointed weird ways, but if you're trying to detect nudity presumably if trained on a bunch of images of that subject, it would just call ceilings False.
Re:Everyone wants to do the model work, not the data work [pdf] by sanxiyn on Monday March 29, 2021 @01:05AM (#26617880)
No, because training set does not include ceilings.
Re:Everyone wants to do the model work, not the data work [pdf] by notsuoh on Monday March 29, 2021 @01:08AM (#26617890)
I think I understand, so you basically need in your training set anything that the production model would presumably see for it to work well? You can't just say here are a bunch of positives and a bunch of negatives, the negatives have to be actual things the model will see.
Re:Everyone wants to do the model work, not the data work [pdf] by sanxiyn on Monday March 29, 2021 @01:23AM (#26617965)
If you think about it, this can't be otherwise. Positive/negative is misleading, since that distinction is the entire thing we are trying to teach. Let's call two labels A and B. You tell the model:

This long penis is A. This short penis is A. This cat is B. This dog is B. Now, what is this ceiling?

The model, looking at the ceiling, discovers a long fluorescent tube. It is long! Neither cat nor dog is long (The model is yet to discover longcat), and while penis comes in long and short variety, all penises seem long-ish. The ceiling is A.

Re:Everyone wants to do the model work, not the data work [pdf] by jononor on Monday March 29, 2021 @08:04AM (#26620356)
In the typical ML task formulation, closed-set classification, the model is forced to make a choice between the provided categories, even if the output is not really either. And neural networks tend to perform erratically outside of the training data distribution.

Adding common inputs to the training (or at least validation and test) sets is a good solution. Its hard data work, but will pay off. There are some techniques outside of closed-set classification that can help reduce the problems, or make the process of improving it more effective:

- Couple the classifier with a out-of-distribution (novelty/anomaly) detector. Samples that score high are considered "Unknown" and can be flagged for review. - Learn a distance metric for "nudity" instead of a classifier, potentially with unsupervised or self-supervised learning (no labels needed). This has higher chance of doing well on novel examples, but it still needs to be validated/monitored. - Use one-class classifier, trained only on positive samples of nudity. This has the disadvantage that novel nudity is very likely to be classified as "not nudity", which could be an issue.

Re:Everyone wants to do the model work, not the data work [pdf] by sanxiyn on Monday March 29, 2021 @01:10AM (#26617901)
Yes, with emphasis on "well". It does work without ceilings. But your training set should include everything you will frequently encounter in test set. Ceilings are too frequent to miss.
Re:Everyone wants to do the model work, not the data work [pdf] by rubatuga on Monday March 29, 2021 @11:08AM (#26622859)
If your model supports Out of distribution detection, you are very correct.
Re:Everyone wants to do the model work, not the data work [pdf] by amelius on Monday March 29, 2021 @04:24AM (#26618848)
Why didn't you train on random images from the web? If you carefully select your sources, most of them will be non-nudes.
Re:Everyone wants to do the model work, not the data work [pdf] by Cthulhu_ on Monday March 29, 2021 @04:31AM (#26618882)
I'm sure there's ways to extract loads of e.g. tiktok videos, especially the non trending / highly produced ones (I'm making assumptions about the use case here).
Re:Everyone wants to do the model work, not the data work [pdf] by collyw on Monday March 29, 2021 @06:35AM (#26619736)
If you "carefully select your sources" then you aren't training on "random images from the web".

If you are using random images from the web, there is a decent chance that you will get a result like this.

https://www.theguardian.com/commentisfree/2021/feb/03/what-a...

The Guardian labels it sexist. I assumed that it was because the majority of young attractive women on the internet are posting bikini pictures.

Re:Everyone wants to do the model work, not the data work [pdf] by gwd on Monday March 29, 2021 @07:10AM (#26619980)
> The Guardian labels it sexist. I assumed that it was because the majority of young attractive women on the internet are posting bikini pictures.

They're both true at the same time, which is exactly the point. The people in the article in the paper weren't trying to teach their neural net that men usually wear suits and women usually wear bikinis, but they did. They only noticed because the failure mode was really obvious. In some other situation, the exact same kind of sexist bias might exist in the data -- and important decisions, affecting people's lives, may be made -- without anybody noticing first.

Re:Everyone wants to do the model work, not the data work [pdf] by iaml on Monday March 29, 2021 @05:38AM (#26619289)
I'm wondering if you could just take some 3d scenes of say flat interiors and then render them in a good quality with camera looking in random directions to use that as training data. You get actually correct labels because you can calculate it when rendering and label accordingly.