Commons:WMF support for Commons/Upload Wizard Improvements/Logo detection

As part of our work to help with media moderation, we developed a tool to automatically detect logos when uploaded on Commons, in order to facilitate their evaluation by the community. A need for machine detection tools was raised in several discussions and user interviews we had in the past with the community, and logos are the second reason for media deletion.

The integration of the tool will be worked on during July - September 2024 as part OKR work for Fiscal Year 2024-2025 key result WE2.3 (“Guide contributors to add images and references that comply with project guidelines and increase trust in content, for example, by flagging potential issues during their upload/addition”).

Current workflow

Tracked in Phabricator
Task T349641

The logo detection tool will automatically detect if a media uploaded by a user has a high likelihood to be a logo, and will tag it accordingly after its upload, to ease moderation review. The current ideas are either tagging it accordingly or applying automatically a template that categorizes the media as a logo to be verified (with an ability to filter by own or not own work). This is the first step towards an automatic process to estimate the likelihood that an uploaded image will be deleted for any reason.

In addition, we are assessing to potentially alert uploading users that the media they are uploading might be a logo and that, if it doesn't meet the guidelines, it might be deleted. In case of multiple uploads, the warning will be shown close to the affected image. The warning may be limited to certain classes of users (i.e. all those who do not have autoconfirmed status), pending community consensus.

Evaluation

Given as input an image file, we detect whether it's a logo by training an EfficientNetV2 classifier to predict logo and out-of-domain probability scores. The experiment overall showed promising results for eventual integration in the ecosystem.

We report below the model that performed best against a test dataset, and observe that it's accurate enough to reliably fulfill the task.

• source: available Commons images
• # images: 47,976 - half belonging to Category:Logos, half random
• accuracy: 96.9
• AUC precision/recall: 98.8
• AUC ROC: 99
• loss: 10.2
• best training epoch: 8

Metrics

• accuracy
• area under the curve (AUC), computed separately for each class and then averaged across classes, see also en:Receiver operating characteristic#ROC curves beyond binary classification
  • AUC precision/recall
  • AUC ROC
• model's loss function, i.e., categorical cross-entropy

How likely is this a logo?

The gallery below displays 50 images and their logo probability score. Images are randomly sampled from the test dataset described above.

• 
  99.82 %
• 
  99.86 %
• 
  99.70 %
• 
  98.60 %
• 
  99.92 %
• 
  99.87 %
• 99.09 %
  99.09 %
• 
  0.32 %
• 
  0.22 %
• 
  2.53 %
• 
  0.99 %
• 
  0.77 %
• 
  0.32 %
• 
  0.45 %
• 
  1.71 %
• 
  0.04 %
• 
  0.51 %
• 
  95.43 %
• 
  99.98 %
• 
  0.27 %
• 
  1.38 %
• 
  64.14 %
• 
  0.11 %
• 
  3.97 %
• 
  0.28 %
• 
  99.77 %
• 
  0.09 %
• 
  99.99 %
• 10.42 %
  10.42 %
• 
  99.79 %
• 
  99.47 %
• 
  0.24 %
• 
  1.05 %
• 
  99.86 %
• 
  0.54 %
• 
  0.92 %
• 99.92 %
  99.92 %
• 
  0.62 %
• 
  0.53 %
• 
  99.92 %
• 
  99.99 %
• 
  0.23 %
• 
  99.93 %
• 
  98.68 %
• 
  99.97 %
• 
  99.72 %
• 
  0.06 %
• 
  83.41 %
• 99.91 %
  99.91 %
• 
  99.38 %

Reports on logos detected by the tool

This section lists monthly reports created by the Structured Content team about Commons uploads detected as logos. We publish 2 datasets: one of available uploads and one of deleted uploads as of the publication date.

• September 2024, as of 2024-10-24
  • available uploads
  • deleted uploads