Ask HN: Local Image Classification in the Browser?

3 points · JawsOfALion · 27 days ago

I want to create a chromium extension, one of the main components of the extension is classifying images (think dynamic content filtering, a few different categories, one of which is recognizing inappropriate content).

Originally I wanted to use a multimodal llm to classify images, because they tend to do quite well at classifying images with little dev effort, but it seems like it won't be possible to my knowledge to get a local model working with a Chrome extension, and an api call for each image will be too expensive as my goal is for it to be free to use.

So next I looked into tensorflow mobile net, and tried this specific example:

https://github.com/tensorflow/tfjs-examples/tree/master/chrome-extension

It looked promising and while it technically worked, it seemed to do very poorly on categorizing most things(except tigers, it seemed to consistently recognize them well). Accuracy was far too low.

Anyways I would like to hear opinions of people who are more knowledgeable in this field, what's the best solution to do a rough, but accurate classification of images with the least dev effort and runnable on a browser? Should I invest time experimenting with other tensorflow mobilenet models, or should I expect fairly low accuracy in them too? (I would like to as much as possible avoid investing in the effort of training my own custom model at this stage)