Commons:Bots/Requests/GogologoBot

GogologoBot (talk · contribs)

Operator: MFossati (WMF) (talk · contributions · Statistics · Recent activity · block log · User rights log · uploads · Global account information) (assign permissions)

Bot's tasks for which permission is being sought: add the following structured data statement and qualifier to the file page of a new upload that is detected as a logo by this tool.

• statement: instance of (P31) logo (Q1886349)
• qualifier: determination method or standard (P459) machine learning (Q2539)

Automatic or manually assisted: automatic, supervised

Edit type (e.g. Continuous, daily, one time run): continuous

Maximum edit rate (e.g. edits per minute): it depends on the amount of image uploads and on the amount of images detected as a logo. Hard to tell for now 2 edits per minute. Please note that this is an anecdotal estimate based on test edits. See also #c-MFossati_(WMF)-20241015104800-Krd-20241011094700

Bot flag requested: (Y/N): Y

Programming language(s): Python, Pywikibot

Source code: https://gitlab.wikimedia.org/toolforge-repos/gogologo

MFossati (WMF) (talk) 12:19, 24 July 2024 (UTC)

Discussion

• I think it'll much better application for bot it it could detect non-trivial logos or logos already deleted. --EugeneZelenko (talk) 14:41, 24 July 2024 (UTC)
• Wouldn't it be better to add them with a separate property? While I'm in favor of adding more such ways to identify images, I don't think it mixes well with other statements. This was attempted and finally discarded with "depicts" statement a while back. Please make sure these statements can also be searched with Special:Search. Enhancing999 (talk) 14:53, 1 August 2024 (UTC)

  Hey Enhancing999, thanks for your comment. Could you please provide any specific pointers to the previous attempt you mentioned? MFossati (WMF) (talk) 11:29, 12 August 2024 (UTC)

  here Enhancing999 (talk) 11:31, 12 August 2024 (UTC)

• Is this bot going to be used as "act once on new uploads", "act once on all existing files", "potentially act more than once on the same file", or what? Unless it only acts exactly once on any given file, what is to prevent it getting into an edit war if its edit is reverted or otherwise changed? - Jmabel ! talk 18:11, 1 August 2024 (UTC)

  Hi Jmabel, thanks for your question. The bot is expected to act once on new uploads. MFossati (WMF) (talk) 11:31, 12 August 2024 (UTC)

  • Good. Is there any chance that the bot could also look at the wikitext for {{Own work}} and add a maintenance category (call it Category:Own work logo to checked) if it appears to be a logo and is claimed as "own work"? We see that combination a lot, and it is almost never true. And possibly something similar for a logo + any CC license, because that's usually false as well: we very rarely get a license for any logo that is above the threshold of originality. - Jmabel ! talk 15:15, 12 August 2024 (UTC)

    I agree that the ability to search for logos plus own work and/or CC licenses would make a lot of sense. I think this is something we can do by querying structured data. For instance, we can already run a query like this to look for own work files with CC BY-SA 4.0. As soon as the proposed logo statements get added, we can then insert a wdt:P31 wd:Q1886349 constraint in the query. MFossati (WMF) (talk) 09:50, 14 August 2024 (UTC)

•  Comment As requested by the rules, we've test-run the bot on 100 uploads randomly sampled from uploads made between Aug 21 and today, and here are the results:
  • 4 medias were deleted beforehand, so no edit
  • 1 media was skipped (maximum retries attempted due to maxlag without success), so no edit
  • 95 medias were successfully edited

It seems that it successfully worked, but we'll wait for community review. Sannita (WMF) (talk) 15:34, 30 August 2024 (UTC)

It appears each file is edited twice. Is that for technical reason, or can the edits be combined in any way? Krd 17:36, 30 August 2024 (UTC)

Great point, Krd! It made me realize that the current code first adds the claim, then adds the qualifier, thus producing two edits. I've just tried that we can do the other way around. So - yes - we can indeed combine them into a single edit. I've updated the code accordingly. Thanks a lot, this is really helpful. MFossati (WMF) (talk) 14:16, 9 September 2024 (UTC)

Can you use another property than P31 as suggested above? I think we should avoid a re-run of c-a t where WMF mostly ignored community input.
 ∞∞ Enhancing999 (talk) 17:52, 30 August 2024 (UTC)

Hi @Krd and @Enhancing999, thanks for your feedback and sorry for the late reply, for some reason your replies did not appear in my notifications.

While we wait for @MFossati (WMF) to be back in office for answering the first question, we are open to suggestion as to which property to use. @Enhancing999 do you already have one in mind? Sannita (WMF) (talk) 16:11, 5 September 2024 (UTC)

You can create one ad hoc.
 ∞∞ Enhancing999 (talk) 17:16, 5 September 2024 (UTC)

@Enhancing999 Sorry for the long answer, but I felt the need to clarify some things about the request.

We need to start somewhere to see if the experiment is of some value to the moderators. This is an experiment within the first quarter OKR work for FY24/25 (WE2.3.1). We don't think a new property would work, especially because the property proposal request would likely be considered too specific in scope to be accepted by the Wikidata community, not without reasons.

We can quickly and easily use an existing property, and see if it’s valuable. If not, we will rollback as quickly and easily. The property instance of (P31) seems like the best fit, because we think it’s specific and meaningful. More importantly, the property is indexed, thus enabling search queries both in Special:Search and in Special:MediaSearch. Furthermore, qualifiers are also indexed, so it will be possible for moderators to find media classified as a logo by this bot. You can either use a search query (example with Special:Search, example with Special:MediaSearch) or a SPARQL one to achieve it.

If detecting and tagging incoming logos does not help with easier logo moderation, then our plan is to rollback our own edits at the end of the experiment. If it does help, then we’re planning to investigate other ways to store and query such data, as we are considering other experiments in the near future as suggested by the community. Sannita (WMF) (talk) 15:09, 9 September 2024 (UTC)

Wikidata easily creates properties that are just meant to be used for Commons. This shouldn't take much time and compared to working speed of WMF (It's seven weeks since you asked for input), this shouldn't be an issue. Nothing prevents you for indexing this property as well.

If you think a separate property wont work, it means that ultimately this wouldn't work using instance of (P31) either. I think such implementations need more attention than once every month.

Given the massive community backlash WMF got from an ill-prepared, hastily implement, not community feedback driven, likely costly previous experiment mixing machine contribution with our highly valued volunteer contributors, I think it's good to take good care this time, especially as a simple way was suggested already seven weeks ago.
 ∞∞ Enhancing999 (talk) 15:43, 9 September 2024 (UTC)

@Enhancing999: unless there are a lot of false positives (and I don't think there are), the tagging of these as instance of (P31) : logo (Q1886349) seems at worst harmless. What would be the advantage of a distinct property? - Jmabel ! talk 04:45, 10 September 2024 (UTC)

There are likely few false positive in the first test set as it's still followed, but last time, it became problematic when person at WMF developing it moved on to something else.

Based on past experience, I guess you know what happens afterwards: you will have to wait 7 weeks for an acknowledgment, then you will be told to ask for a change in the next wishlist, and, even if everybody agrees with it, you will have to wait for the next annual plan to have it scheduled. Possibly somebody will then throw it out entirely, because they don't know how to fix it.

In any case, the idea is to classify also images where there is a lower confidence in the automatism so review is necessary.

Using two different properties allows users to easily switch between volunteer assessment and machine assessment, focus on volunteer assessment while excluding machine assessment if they happen to agree.
 ∞∞ Enhancing999 (talk) 11:12, 13 September 2024 (UTC)

Is a coat of arms or a military unit insignia or a sports uniform a logo per the definition a "logo"? --Krd 07:29, 13 September 2024 (UTC)

@Krd: we're targeting images similar to Category:Logos, thus making a distinction between other classes such as Category:Coats_of_arms or Category:Sports_kit_templates. MFossati (WMF) (talk) 13:45, 13 September 2024 (UTC)

In my personal opinion there are too many false positives. Krd 13:52, 13 September 2024 (UTC)

Special:Permalink/923690458 has a gallery of images edited by the bot. Personally, I don't think false positives are an issue as such, at least when they are clearly distinguished from manual edits (see separate property above).
 ∞∞ Enhancing999 (talk) 14:08, 13 September 2024 (UTC)

I agree that most of them are some kind of symbols or graphics, but I'd guess a third of them would not be put under Category:Logos, so "instance of logo" doesn't make much sense then. Am I mistaken? Krd 14:16, 13 September 2024 (UTC)

It really depends what the logo people want to do with it. Today it's "logos", but it could be just any image type or topic. The confidence level of the classification can also evolve or be changed.
 ∞∞ Enhancing999 (talk) 14:29, 13 September 2024 (UTC)

Come to think of it, maybe the statement with the new property should be qualified with the confidence level (for the classification of the image) and the program version being used (if not available, the current date).
 ∞∞ Enhancing999 (talk) 09:08, 17 September 2024 (UTC)

That makes sense to me. Krd 11:46, 17 September 2024 (UTC)

You read my mind: this is definitely something I wanted to propose. MFossati (WMF) (talk) 13:53, 17 September 2024 (UTC)

I've just realized that the bot has made accidental edits that weren't meant to be there, sorry for that! I've manually reverted them. Please refer to the test edits. MFossati (WMF) (talk) 13:58, 17 September 2024 (UTC)

Similar to Special:Permalink/923690458 can you do a gallery that shows all images that you consider valid test cases (ideally include the confidence level for the classification as a legend).
 ∞∞ Enhancing999 (talk) 09:21, 21 September 2024 (UTC)

The (determination method or standard (P459), machine learning (Q2539)) qualifier distinguishes the bot's edits from manual ones. The queries mentioned here retrieve the bot's ones. You can compare these two queries: with qualifier (bot's edits) VS no qualifier (non-bot's edits). As a side note, nothing prevent us from trivially looking at the bot's contributions, too. MFossati (WMF) (talk) 13:46, 17 September 2024 (UTC)

You can't distinguish them any more when someone thinks they are correct and also adds a P31. Or would they have to remove the qualifiers? And no, looking at individual files and/or edits is definitely not a solution. Please make sure the results can be view by querying both with search and on SDC portal (hopefully eventually open).
 ∞∞ Enhancing999 (talk) 09:16, 21 September 2024 (UTC)

We could also create a Wikidata item for this bot and use it as the qualifier value, instead of machine learning (Q2539). MFossati (WMF) (talk) 14:01, 17 September 2024 (UTC)

It's better to use a separate property and qualify that with the program version being used. A year or two later, one will otherwise have a hard time which version of the bot considered what by which threshold. I suggest we create to properties:

• "Commons machine image type"
• "Commons machine image subject"

The second for later uses, if you wan't to try to determine a logo topic.
 ∞∞ Enhancing999 (talk) 09:20, 21 September 2024 (UTC)

• I'd like to highlight the main goal of this experimental bot, namely to help moderators find potentially problematic media. MFossati (WMF) (talk) 14:06, 17 September 2024 (UTC)

  What is the commitment of WMF to maintain this going forward? How much time will you spend maintaining it in the next months each week? Or will it be discontinued after a month?
   ∞∞ Enhancing999 (talk) 09:23, 21 September 2024 (UTC)

  We are committed to maintaining this bot for as long as it needs to be. As already mentioned, this is one of our priorities for the year, and definitely won’t be dropped after one month. On the other hand - after careful consideration with the team - we won’t be pursuing the path of creating new Wikidata properties, nor adding the confidence score as structured data, as part of this work of identifying and providing a way for easier moderation of logos.

  While we agree that probabilistic statements supported by confidence scores are a very relevant topic, to the best of our knowledge no available Wikidata property can express so yet, and we see the need for a cross-community broad discussion that is outside of this experiment’s scope. If no consensus is reached on this bot request, the alternative is that we periodically release lists of potential logos to be considered (this time with confidence score), like we recently did. Sannita (WMF) (talk) 13:42, 25 September 2024 (UTC)

  Creating appropriate properties is a fairly straightforward process. As you seem to have some issues with having these created for that data, I think the dataset approach is preferable.

  It also wont leave us with data the community needs to clean up next year, once the experiment has ended, as last time.
   ∞∞ Enhancing999 (talk) 12:20, 29 September 2024 (UTC)

  Do we have moderators who use the output of the bot for anything? I think it hasn't been outlined above, so I'm still trying without offense to understand who is in need of that, or if it may be a solution looking for a problem. Krd 13:36, 26 September 2024 (UTC)

  Hi @Krd, sorry for the late reply, but notifications aren't working on this page for some reason. As of now, as far as we know no one is using the output of the bot, mostly because it has not been approved yet and we are waiting on approval to fully resume our work. But the reactions on the admins' noticeboard to our dataset about potential logos seems to show that our work is effectively useful to identify potentially problematic logos, and can let admins and moderators focus on a narrower set of images, instead of relying on reports on last uploads. Plus, as we already stated, this proposal comes from several discussions and user interviews we had in the past months with the community where the need for machine detection tools was raised, so it is a solution to a problem that the community raised. Sannita (WMF) (talk) 13:54, 3 October 2024 (UTC)

  I think users would prefer categories, but dataset seem to work too. Given the problems with getting the statements right and the closed nature of SDC statements, I think it's preferable to pursue the two other ways (dataset, categories).
   ∞∞ Enhancing999 (talk) 16:56, 9 October 2024 (UTC)

• Looks like a great way to reduce mod/admin maintenance workload and reduce the number of copyvios on WMC. Please extend it or create similar bots to also detect other copyvios as proposed here or similar. Thanks for developing it, it seems very useful! --Prototyperspective (talk) 11:37, 21 September 2024 (UTC)

  Hi @Prototyperspective, thanks for your positive feedback and suggestions (and also sorry for the late reply, but notifications aren't working properly on this page). We do think this work is valuable and we heard good feedback. Before engaging in new bot requests, though, we think it would be better to close the current request with enough consensus to go on. We are going to do another experiment for automated detection in the next months as part of our planned work, but we also don't want to operate without or against community consensus. For now, as the current bot request has not been approved, you can access logos identified by the logo detection through exported datasets. Sannita (WMF) (talk) 13:56, 3 October 2024 (UTC)

@MFossati (WMF): You didn't specify the edit rate in the request. What do you expect the daily edit count to be, and how many edits will be required to classify existing files? Does it make sense to start with one edit per minute for an extended test run? --Krd 09:47, 11 October 2024 (UTC)

What do you expect the daily edit count to be

The test edits indicate 1 edit every 33 seconds on average, with an estimated daily count of 2,600 circa.

how many edits will be required to classify existing files

Please note that the currently requested task only accounts for new uploads. We may consider scaling the bot up to existing files in a subsequent request, if that's useful for the community (broader discussion needed). However, we can't compute the total amount of edits beforehand, because we'll have to run (or dry-run at least) the classifier over existing files first.

Does it make sense to start with one edit per minute for an extended test run?

I think it would be reasonable to stick to the average edit rate to do so, i.e., 2 edits per minute. MFossati (WMF) (talk) 10:48, 15 October 2024 (UTC)

Please put it live as suggested, as extended test run. Krd 10:56, 15 October 2024 (UTC)

 Done The bot is running and will edit 3,833 recent uploads, detected as logos between Aug 21 and today. See also the initial test. MFossati (WMF) (talk) 15:02, 16 October 2024 (UTC)

Please do not manually edit with the bot account. --Krd 07:18, 20 October 2024 (UTC)

I addressed a user request and also noticed incidental duplicate statements, so I took care of them. The affected pages were edited through pywikibot code, so no manual edits happened. Not sure why those edits got the manual revert tag, though. MFossati (WMF) (talk) 09:49, 21 October 2024 (UTC)

Special:Diff/945223918 is a manual edit, isn't it? Krd 10:15, 21 October 2024 (UTC)

That wasn't intended: I forgot to logout from the bot account and login to my user one before signing that edit. I've fixed the signature one minute later, see Special:Diff/945224965. MFossati (WMF) (talk) 13:37, 21 October 2024 (UTC)

• The extended run of the bot has finished, ~3,800 images have been edited and identified as a logo, you can take a look at the bot's contributions to evaluate them. What do you think of the results? Do you think the bot could be now allowed to run? Sannita (WMF) (talk) 16:56, 24 October 2024 (UTC)
  • First thought is that all of the "BSicon" stuff is wrong, none of those are logos. - Jmabel ! talk 19:41, 24 October 2024 (UTC)
  • Picking 10 others at random:
    1. File:MEFA Logo.svg already appropriately categorized as a logo, clearly below TOO.
    2. File:Bridger Aerospace logo.svg Like the previous one. I did add Category:Logos of companies of the United States because uploader hadn't thought to say what country.
    3. File:Club América de Palpa.png might be problematic; claimed as "own work" (which might be true for the particular PNG); roughly in the neighborhood of TOO, I don't know rules for Peru, someone might want to take a look at this. Needs categories, in any case.
    4. File:Avelonia Logo 2024.png similar case to the last, although in this case I can't even quickly tell what country.
    5. File:2gotravellogo.png needs categories, from Philippines, appears to be below TOO.
    6. File:CTree RootNet Bubble blue rev 05 interim 600dpi.png. No idea what this is, needs cats, might be a logo or not, doesn't say what country, below TOO almost anywhere.
    7. File:BR Verkehr Variante.svg appropriately categorized, clearly below TOO, unproblematic.
    8. File:Auliq Records wordmark.png clearly below TOO, heavily but poorly categorized, already nominated for deletion as out of scope (on which I have no opinion).
    9. File:Filebrowser - banner.svg already correctly identified as a free software logo, the claimed license is indeed granted at specified source (but might be below TOO anyway).
    10. File:ALoSeguro.svg not clearly a logo (just text form of a slogan) though categorized as such, clearly below TOO, appropriately categorized.
  • So the usefulness of this is supposed to be …? - Jmabel ! talk 20:04, 24 October 2024 (UTC)

    Thanks for your proactive feedback! I think it really follows the direction of our long-term goal, namely to help moderators find problematic media.

    • Re: BSicon - this was already raised in a previous request and fixed;
    • re: random examples - you can find more information along the lines of your comments in the reports we published. For instance, categories and own work are there. See [1], [2], and [3].

    MFossati (WMF) (talk) 14:15, 25 October 2024 (UTC)

    • @MFossati (WMF): I'm an admin, and based on my sample above, I don't see how this is any more help in finding problematic media than looking at any random collection of newly uploaded files. Can you clarify how this is supposed to be more useful in finding problematic files than just a random selection of recent uploads? - Jmabel ! talk 18:45, 25 October 2024 (UTC)

      @Jmabel Thanks for your feedback. Marco is currently out of office, so I'll try to take a stab at it: we're trying to narrow down the number of potentially problematic files, by highlighting only the potential logos among the quite high number of uploads that everyday are made on Commons.

      We received already some positive feedback about our work in the last six months, and according to our findings our experimental tool can detect ~47% more files that are not correctly categorized as logos or are not correctly pointed out as such. This does not substitute the need for a human eye to evaluate them, but at least should be enough to help said human eye to find what they need to find, without resorting to check a "random collection of newly uploaded files". This is also what we're trying to do with our datasets published on the Admins' noticeboard.

      Anyway, if this bot request is just not enough useful, we'll stick to publishing datasets for you admins to consider. Sannita (WMF) (talk) 19:02, 29 October 2024 (UTC)

      • Is there any "control" for this experiment? E.g. any pre-bot baseline on some task or tasks that we can measure against to see whether the bot is actually helping anything happen any better than before? - Jmabel ! talk 21:18, 29 October 2024 (UTC)

        @MFossati (WMF):  ? --Krd 05:21, 5 December 2024 (UTC)

        Given a monthly dataset such as November 2024, we compute the gain ${\mathcal {G}}$ as follows: ${\frac {{\mathcal {I}}-({\mathcal {D}}+{\mathcal {T}}+{\mathcal {C}})}{\mathcal {I}}}\times 100$, where ${\mathcal {I}}$ is the total amount of input images detected as a logo, ${\mathcal {D}}$ is the amount of input images that were deleted, ${\mathcal {T}}$ is the amount of input images with Template:PD-textlogo, and ${\mathcal {C}}$ is the amount of input images with at least one category that has an occurrence of logo. We consider ${\mathcal {D}}+{\mathcal {T}}+{\mathcal {C}}$ to be the human curation baseline that matches uploads detected as a logo. ${\mathcal {G}}$ represents the percentage of potential logos that hasn't undergone human curation. For instance, November 2024 has ${\mathcal {G}}=48.6\%$. MFossati (WMF) (talk) 15:53, 6 December 2024 (UTC)

Please summarize. Have all objections been resolved? What is the conclusion? --Krd 06:52, 14 December 2024 (UTC)