Fully agree - This seems like a project with rather limited in scope. And that's ok. The authors even acknowledge it. It's not an absolute judge of SFW'ness, neither does it claim to be. I can see it as a tool to skim a large set of videos for further human review. Besides, beach volleyball very much is NSFW depending on the situation.

> And that's ok. The authors even acknowledge it

That doesn't always stop corps productizing it, and making it their official NSFW filter. Good enough is good enough, just so long as they can pay less salaries.

Exactly. So many products get sold based on handwaving and being able to check boxes. Anybody releasing open-source software needs to ask themselves, "What uses will this be put to?" Computers can be instruments of freedom or instruments of control.

I don't think people should be wary of "what it could be used for" outside privacy violations or military action. The distinction "used for freedom vs control" is too vague/subjective.

In this case, I think it's best just to hold companies accountable for discrimination that isn't based on some protected attribute. Companies lean to heavily on "right to refuse service (for any reason)" and "we can choose who to do business with".

The fact we distinguish between "things you are allowed to discriminate on" and "thins you aren't" makes for a very convoluted process. Same with at-will employment tbh. We should just flip it from a "you need a reason to thing you were unjustly discriminated against" to "you need a good reason to fire someone".

Subjectivity seems fine to me when it's a decision each person has to make for themselves. If you don't want to think about what you might be helping others to do, you don't have to. But I will.

You give an example of what kind of thing you will consider?

I mean, "not safe for work" doesn't mean "immoral". Beach volleyball isn't something you should be watching at work, at least not in the kind of open office plan where everybody sees your screen. There's a lot of talk about workplace "lad culture" being hostile to women these days -- possibly enough to mean liability; also you probably shouldn't be hostile to women in that specific way.

It's interesting. I've not tested the model on anything too risque, but again, with the well known Baywatch intro as a frame of reference, wide shots of the whole cast in their swimsuits, are fine. But a close up of any single cast member a swimsuit will invariably trigger the model. Male or female.

In the blog, I suggest this could be the result of an uncultured data set when training the CNN. Or perhaps the dataset was fine, and this is pushing the hard limit of what ResNet50 CNN architecture can do (the off the shelf model I use for this is an extension of ResNet50).

Some of the anomalous results are amusing. One day, I uploaded a video of a female violinist in concert, and the model flagged every close up of her as NSFW! Just those closeups. Wide shots, and closeups of other musicians were absolutely fine.

Again some of that might be down to me (clunky code / very low NSFW threshold). And I suspect the model I used was itself a PoC (https://github.com/emiliantolo/pytorch_nsfw_model). But it does make you wonder how the bigger labs with critical products, like Palantir, handle doubts like this.

Why? During the Olympics the beach volleyball plays out across our floors at work just like any other event. In fact they played the olympics on the big screens because it’s easier on the network.

>Beach volleyball isn't something you should be watching at work, at least not in the kind of open office plan where everybody sees your screen.

Watching the olympics at lunchtime is commonplace and accepted in most workplaces. Being a creep about it is not. It's almost always a pretty clear line between the two.

It only becomes sexualized by the behavior of the people watching - professional athletes competing for their country isn't something intrinsically inappropriate.

This is a great example of how categorizing the content isn't a great solution.

If someone is sexualizing something in the workplace, the problem is the person's behavior, not the thing they are sexualizing.

The situation we find ourselves in came from setting boundaries around content instead of behavior. Our society treats nudity itself as if it were the act of sexualization, assuming that people have no control over their own behavior.

I sometimes have mens cycling on while I work. This summer I'll properly have the womens tour de france on. They are almost as 'scantily' clad as beachvolley players. Though their bellies are covered. Is that NSFW or hostile to women? Is it okay to have the woman 100 meters dash on or is it as inappropriate as beach volley (they do wear almost as 'revealing' clothing)? Is mens sport okay? Maybe marathon and other sports with skinny clothed men is SFW, and not diving muscular men in tight panties is not?

I understand it may be subjective, but I find it hard to agree that women cyclists are as ‘scantily’ clad.

I can't speak for your office culture but in mine we don't watch TV at work, period.

The idea that watching scantily-clad women s.a a Beach volleyball tournament is automatically hostile to women is just ingrained at this point. Maybe one day we'll return to normality.

NSFW does not literally mean "not safe for work" anymore, in almost all usages now it refers to explicit sexual material

While I agree in part, at the rate new content is produced and uploaded to the platforms around the globe, could you even fill enough seats with reviewers at an economically-doable rate to keep up?

*and they’re not trying to make a video service for Kids

>depending on the situation.

That's part of the issue.

We've seen plenty of author acknowledged research grade models and datasets get used in production, this is part of how black people get labeled as "monkey" every once in a while

I want to believe it's a matter of adding human labor to the Pool of Knowledge in the AI engine(s); this is, this is not, go do the learning thing.

But when it comes to online video checking, it's another tool in the arsenal; it's the first lines of defense. First check: does the signature match a previously known and confirmed video marked as porn or otherwise unacceptable. Second check: does the more fuzzy AI thing consider it porn with a high probability. Third check, which will be things the AI mark as 'not sure' or incorrectly marks as 'porn', and things humans flag up manually, will be humans checking and judging things.

And of course it'll be flexible, because the definition of porn - or what is 'not acceptable' - will vary by country and culture, and these companies want to be active everywhere.

except Europe, unless they are given tax breaks and data, amirite

Why does everything have to be perfect? Humans get this wrong all the time as you say, and we still do it - right?

Seems like we already acknowledge good enough practices and are okay with them. Lets not demand perfection in the same way we shouldn't assume perfection of ~~AI~~ ML.

It could potentially revolutionize the opposite application, those sites where someone manually finds and catalogs all the time points at which movies contain nudity so people can see their favorite celebs in the buff. False positives are probably worth the automation there.

Even if this can't do a 100% job, isn't it still useful to automatically identify the obvious NSFW content?

yeah but this is true for profanity filters as well, and yet useful profanity filters still exist. Is it 100% accurate? No. But it's probably still useful.

Really? Can you point to a single example of a useful profanity filter?

I'll take your silence as a "no".

You're right that stuff is quite difficult. I write a Firefox addon (https://addons.mozilla.org/en-US/firefox/addon/wingman-jr-fi..., https://github.com/wingman-jr-addon/wingman_jr) and train an associated NSFW model (https://github.com/wingman-jr-addon/model) - I've been at it for a few years now, and have had to plug many specific edge cases.

   - Babies (https://github.com/wingman-jr-addon/wingman_jr/issues/22)
   - Beach volleyball (but this definitely has SFW and NSFW variants, based on a somewhat subjective line)
   - Athletes in general. The model particularly thought some American football players were NSFW for a long time.
   - Swimming
   - Yoga - again, most SFW and some NSFW here but it still struggles
   - Wrestling was a tough one for sure
   - Pokemon

While indeed tough, I've seen definite progress. So it's not just a matter of tech, but also of considering the human element - the state of the art may not be up to the challenge of perfection, but it is definitely up to a point of true utility for some use cases. I'm happy about that.

As a note, it uses an EfficientNet Lite L0 backbone - I'm a bit limited in what type of scanning I can perform in a sufficiently speedy manner.

I also agree on the context for sure - one reason I haven't tried switching to an object detection method (and that I don't rely heavily on truly random crops) is that the focus of the image is highly important for the NSFW-ness in some cases. True, two images may contain the same content ... but one is far worse than the other. The nature of CNN's still has some of this location-invariance baked in, but I don't want to exacerbate it.

One challenge I think the OP may run into here that may also not be immediately obvious is that accuracy on image stills does not translate that well to video. I have basic video support in my addon, and while I knew there would be some differences, I was surprised at how many discrepancies there really are. As two examples:

   - Images in video are often blurrier. In true still images, there is a somewhat higher prior involved with amateur NSFW content and blurriness. This can be a source of false positives.
   - The opposite of the note above about focus. Taking stills of moving images will have many transitory frames that seem inappropriate on their own because it seems as if they are focusing on something when in reality the camera is just panning - obvious to the human, less so to the model trained on stills.

At any rate, given how well your list of edge cases coincided with failures I've grappled with, I'd be interested to see how well you think my addon stacks up for still images when set to stay in "normal" mode. I'd love to hear any feedback you have via GitHub so I can make it better.

The biggest difference between our two projects, is the effort you've put into training your own model, which I think is amazing.

One of the technical issues that you pointed out, is that a model trained on still images, shouldn't be expected to work on video. While I did not train a custom model for this this project, I'm current working on another DNN model for a completely different purpose, where I think feeding frame deltas into the model, will improve the outcome.

As a hobbyist, I would reckon for porn and the like, analysing frozen frames is probably just enough. For violence however, I would agree with you and say that some effort to encode motion would be essential.

Focussing on NSFW content generally, I would guess, depending on the scale of your project, that you will forever run into 'edge cases' for NSFW images, even before you run into the soft wall of subjectivity.

I agree that the tech is improving all the time, and I think something like this can be made to be truly useful one day. Possibly soon. But it would need a large, active development team, a great deal more compute, and a LOT of data. In much the same way that no home/garage coder can hope to put together a model like GPT3 right now, I would think that a foolproof NSFW classifier would need more resources than you or I have access to at this moment.

But things change all the time.

Thinking about what you're doing, one thing I might suggest, if you have time to develop it, is to add some kind of 'recording' mechanism to your plugin, so that the users themselves can add to your dataset... But you have to wonder how many users will allow that! XD

I'm also wondering if a Firefox extension is the best place for your model? To that end, I would suggest putting the app on a server (which is what I originally wanted to do with my hack) which will give you the opportunity to crowdsource data collection. People might be more willing to volunteer data in that way (in a similar way to how people use https://builtwith.com/).

You're also very welcome to take the UX work I've done on this opensource project (because this hack was ultimately just a UX experiment), and plug your model in. If your model and trained weights are available, I'd like to try and create a branch myself, if I have time.

Also, as hobbyist building knowledge, I hadn't heard of `EfficientNet Lite` before. I'd been considering Darknet - https://pjreddie.com/darknet/ for embedded stuff until reading your post.

Thanks for the response dynamite-ready. There's a lot in here, but I'll try to comment on a couple items. Some of your suggestions I've actually thought about extensively, so perhaps you'll find the reasoning interesting?

Regarding the current state of tech: I agree the tech still has quite a ways to go. I think one of the most interesting aspects here is how e.g. NSFW.js can get extremely high accuracy - but not necessarily perform better in the real world. I think it speaks in part to the nature of how CNN's work, the nature of the data, and the difficulty of the problem. Still, having seen how incredibly good "AI" has gotten in the last decade, I have quite a bit of hope here.

Regarding putting it on a server: that is indeed a fair question, but my desire is to keep the scanning on the client side for the user. In fact, it was actually the confluence of Firefox's webRequest response filtering (which is why I didn't make a Chrome version) and Tensorflow.js that allowed me to move from dream to reality as I had been waiting prior to that time. I can't afford server infrastructure if the user base grows, and people don't want to route all their pictures to me. So I guess I see the current way it works as a bonus, not a flaw - but it DOES impact performance, certainly.

Regarding data collection with respect to server - yes, this is something I've contemplated (there's a GitHub issue if you're curious). There are, however, two things that I've long mulled over: privacy and dark psychological patterns. Let me explain a bit. On the privacy front - it is not likely legal for a user to share the image data directly due to copyright, so they need to share by URL. This can have many issues when considering e.g. authenticated services, but one big one also is that the URL may have relatively sensitive user-identifying information buried in its path. I can try to be careful here but this absolutely precludes sharing this type of URL data as an open dataset. On the psychological dark patterns front - while I'm fine with folks wanting to submit false positives, I think there's a very real chance some will want to go flag all the images they can find that are false negatives (e.g. porn). I don't think that type of submission is particularly good for their mental health or mine. So, in general, I think user image feedback is something that would be quite powerful but needs a lot of care in how it would be approached.

Regarding the UX - thanks! And you're welcome to try the model as well - I've tried to include enough detail and data to allow others to integrate as they wish: https://github.com/wingman-jr-addon/model/tree/master/sqrxr_... Also, let us know how things go if you try out Darknet.

Good luck!

Mind elaborating on Pokemon?

For whatever reason, animated Pokemon were a particularly tough case - I remember Charizard in particular being a tough offender for false positives. I cannot find a reference to it now, but I distinctly remember that at one point Yahoo's open_nsfw also had issues with Pokemon (although these two networks do not have a common lineage). Why would these cause problems? Not sure - maybe something to do with gradients of near-skin-tones.