Commons talk:Structured data/Computer-aided tagging

From Wikimedia Commons, the free media repository
Jump to navigation Jump to search

 Tagging notifications can be turned off in user preferences.

SpBot archives all sections tagged with {{Section resolved|1=~~~~}} after 7 days.

Please let the development team know of any questions, comments, or concerns you might have about any aspect of this project's implementation. The team will see what it can do to address them. Keegan (WMF) (talk) 21:04, 16 September 2019 (UTC)[reply]

Hasta las narices!!!!!!!!!!!!!

[edit]

Me están saliendo campanitas rojas a cascoporro toda la mañana, y estoy con una movida de biiiip en Wikidata con 30000 Qs que he de reducir a 25000 más o menos, y también una movida de papeletas electorales que estoy echando una mano, y otra mano la echo con una clasificación por comarcas... Y estoy harto de la biiiip de las campanitas coloradas.

¿¿¿Podeis pararlas de una biiiip vez??? B25es (talk) 07:48, 24 May 2023 (UTC)[reply]

Normalmente, creo que puedo leer español sin dificultades, pero no puedo darle ningún sentido a esto. ¿Puede algún hispanohablante nativo explicar o responder? - Jmabel ! talk 14:55, 24 May 2023 (UTC)[reply]

I will explain. Yesterday (it applies for today too) I was doing too many things with too little time. And the bell that is on the upper line of my Wikimedia screen (the bell that says if I have an alert) started to turn red once and again. No less than five times in a row, may be seven. Once with a diference of seven (-7-) seconds. And I was very upset. Too much upset. The bell wouldn't stop turning red because there were some files I could add structured data too. I couldn't. Just ONE brain and just TWO hands and just NO time. And that's it. B25es (talk) 05:45, 25 May 2023 (UTC)[reply]

Large numbers of trash tags?

[edit]

Should this experiment be put on hold ASAP? We're just seeing a whole load of tagging going on, but the accuracy rate is awful.

There's no point in generating data if it's just unusable garbage and discredits what was there beforehand first. Andy Dingley (talk) 11:15, 28 May 2023 (UTC)[reply]

Yes. Any pretence that this is really about structured data has long become untenable. Andy Mabbett (Pigsonthewing); Talk to Andy; Andy's edits 16:17, 28 May 2023 (UTC)[reply]
I also think that using the word tag for depicts was the starting point of this. I think the only solution would be to make depicts (P180) officially the tag statement and then get a new property for what depicts was originally intended to be. Moving all current statements would also be possible but then we have the problem that all old tools would still use the old one. GPSLeo (talk) 17:19, 28 May 2023 (UTC)[reply]
I am pretty sure the tags I am adding to my own photographs are correct and appropriate. May be one should create a special flag for this, or restrict this to some flags, but I do not thing shutting the whole thing down is a good idea. Categories are the 20th century technology, and they do not perform the function they are expected to.--Ymblanter (talk) 18:31, 28 May 2023 (UTC)[reply]
The problem is that good depicts are difficult to discern from the garbage. One could have a bot act on depicts added by you or added to your uploads, and similarly for other users who think theirs are OK. However, loads of garbage is added all the time, as this tool gives the impression that those are the ones to add. Perhaps the bot could identify the depicts typically suggested by the tool and ignore them, but as the tool becomes more sophisticated that becomes harder. –LPfi (talk) 19:17, 28 May 2023 (UTC)[reply]

Chiming in to agree that we are constantly getting tags that are useless at best, e.g. [1]. Depicts nature (Q7860)? I imagine that could theoretically be added to a third of our files. - Jmabel ! talk 19:40, 28 May 2023 (UTC)[reply]

I think the worst I've seen is this. The whole of Special:Contributions/Daphpb is much the same. Andy Dingley (talk) 22:27, 29 May 2023 (UTC)[reply]
Maybe the tool should be prevented from suggesting any tags which are currently already applied to more than 250 or 1000 images? This wouldn't prevent editors actively choosing these tags, but might prevent some of the useless suggestions. If a tag genuinely applies to thousands of images then it probably needs to be refined. AlasdairW (talk) 21:36, 29 May 2023 (UTC)[reply]
Edits like these: [2], [3], [4] could make me weep. I've no doubt the individual making them is acting with the best of intentions, but such edits bring nothing but harm to the project, and should be reverted en masse. Would User:Keegan (WMF) like to comment? I gather that User:RIsler (WMF), who promised in February 2020 to answer my questions on these points "soon" (see «Bad tags, nagging, and no tags and Misplaced invitation to "tag" images), yet never did so, is no longer with the WMF. (Also @ User:Lydia Pintscher (WMDE) for info.) Andy Mabbett (Pigsonthewing); Talk to Andy; Andy's edits 16:49, 13 June 2023 (UTC)[reply]

WMF response

[edit]

Hello everyone. Thank you for all your thoughtful comments and concerns about the Computer-Aided Tagging (CAT) tool. I am Runa, Sr. Director in the WMF Product department, and I support the Structured Data team (along with a few other product teams). We understand that the accuracy and utility of the tags generated by this tool have been called into question.

Our team is currently discussing your feedback and considering different ways to respond to the issues you've raised. While we don't have a definitive plan of action yet, we're contemplating various approaches, such as conducting a review of a random sample of edits made via CAT and allowing community members to classify those edits as good or bad. This could help us assess whether the CAT tool is doing more good than harm, and inform our next steps.

We also want to clarify that some of the examples mentioned in the discussion were added via the Android Suggested Edits tool, which are chosen manually by individual editors and are not directly related to the CAT system.

We appreciate your patience as we continue to explore this matter.--Runa Bhattacharjee (WMF) (talk) 10:59, 16 June 2023 (UTC)[reply]

@Runab WMF: You seem to overlook that we've been telling you of these issues and asking for your response to specific points since February 2020. Reviews of tags and plenty of examples of typical bad values were provided back then. There is no reason to suppose that the situation is anything but worse now.
We already know that the CAT tool is doing more harm than good. And yes we know many of the bad values were "added via the Android Suggested Edits tool"; it simply asks users to "tag image" and "add tag" - that too is doing more harm than good.
A user-classification system would just add another layer of work for over-strectched volunteers; even supposing it operated with more clarity and sense about how the depicts property is supposed to be used, than the current lamentable system. Frankly, you (collectively) don't seem to understand what the problem is. It is therefore far too early for you to be proposing a solution. Andy Mabbett (Pigsonthewing); Talk to Andy; Andy's edits 16:07, 16 June 2023 (UTC)[reply]
Hello @Pigsonthewing. Focusing specifically on CAT (and, by extension, ISA), we acknowledge that these tools have generated some inaccurate tags. Thanks to your examples, we're aware of the quality issues. We still need to clearly understand the amount of accurate data these tools generate in comparison.
These tools are widely utilized, indicating a measure of value to our community. Therefore, we can't propose disabling them without thoroughly understanding their overall performance - good and bad.
Regarding your comment, "Frankly, you (collectively) don't seem to understand what the problem is." we're striving to grasp the complete picture. We're relying on the "depicts" guidelines, to evaluate the quality of the annotations. We greatly appreciate your sharing if we're missing additional context or insight that will help us understand the problem better.
For transparency, I want to mention that we have had some transitions internally within the Foundation’s Product and Technology teams, and I have started to support this team only earlier this month. Given the ongoing conversation, it is a priority for me at this time to understand this issue better and guide the team towards a meaningful resolution. Runa Bhattacharjee (WMF) (talk) 16:32, 22 June 2023 (UTC)[reply]
  • This is yet another absolutely typical WMF initiative:
    Involve no-one
    Build something useless / damaging
    Blame the editors for not understanding WP
    Load a pointless additional task onto the same editors to clean it up. Badly.
    Delete the whole thing, pretend it never happened, and give golden handshakes to anyone involved.
What we need here is to delete the whole of the data added recently (this is brutal, but simple enough to achieve it). Then don't do it again. Andy Dingley (talk) 18:39, 22 June 2023 (UTC)[reply]
  • "These tools are widely utilized, indicating a measure of value to our community." That is a non-sequitur. It is widely utilised because it is promoted to volunteers, and they do not see, nor receive proper feedback about, the impact of their actions.
    Regarding Commons:Depicts; please see Commons:Depiction guidelines, and their respective talk pages, especially discussions about their disparity (also here). I also again refer you to the (still unanswered) questions I raised on these points in February 2020.
    I appreciate your personal commitment to this. However, having been trying to get the WMF to address this since that date, my patience, for which you collectively thanked us above is, nigh on exhausted. Andy Mabbett (Pigsonthewing); Talk to Andy; Andy's edits 21:01, 22 June 2023 (UTC)[reply]
Logo VfL Hameln
  • P.S. I just fired up Special:SuggestedTags; the first image it gave me was the VfL logo, above. The suggested values are:
    • graphics
    • signage
    • logo
    • rectangle
    • electric blue
    • brand
    • symbol
  • graphics (Q1027879), for example, is intangible, and not something that can ever be depicted. All bar "logo" are inappropriate.
  • With naive users encouraged to "tag" images with such junk (remember, these get added as "depicts" statements, they are not tags), I fail to see any good reason why you "can't propose disabling" this tool, and doing so immediately. We've been asking how to do so since (I say again) February 2020. Andy Mabbett (Pigsonthewing); Talk to Andy; Andy's edits 21:16, 22 June 2023 (UTC)[reply]
Andy and Jmabel: I have followed this issue with interest, and I have only one question: Why did you ever thought this was going to be any different at all? As said, this is the usual WMF aloof makework, launched with spammy splash and lavishly funded, followed more or less swiftly by the usual trainwreck salvage and «golden handshakes» (sadly apt phrase!). I myself never-ever bothered with "depicts", never will, and I’m not surprised it turned out this way. The only adavantage I see in it is that it keeps gamified drive-by pseudo-curation of Commons files away from categorization. -- Tuválkin ✉ ✇ 19:44, 23 June 2023 (UTC)[reply]
@Tuvalkin: For the record: from the outset, my engagement with SDC has been about mitigation. As you may recall, I gave up on trying to work with the people defining up front how "depicts" would work after a Brazilian contributor who will remain nameless told me I didn't know anything about data modeling (a field in which I spent several years of my working life at a senior technical level) and no one stood up for me at all. After that, I've largely confined myself to asking "is this really what you meant" (in about 20 different ways) when I've encountered obviously inappropriate results. Also for the record, I still think my ideas for (1) integrating SDC with categories, rather than having them as rival systems, and for (2) making SDC editable through the Wikitext interface were entirely workable, but a couple of people at Wikimedia Deutschland were the only ones who actively agreed with me, and at this point I don't think the WM-DE team is at all involved in SDC. And the "suggested tags" thing has been nothing but a liability from Day 1.
Still, I do also want to say: much of SDC other than "depicts" (and maybe "captions", which have had similar but lesser problems) actually works pretty well, and is probably of some value.
I think the biggest problem with "depicts" is that it was a mix of a large, tech-driven project and a "folksonomy" approach, with no clear goals other than leveraging Wikidata for its multilingualism. So we now have a growing pile of easily translated crap. - Jmabel ! talk 02:58, 24 June 2023 (UTC)[reply]

Update on Computer-aided Tagging - We're on it!

[edit]

Hello everyone,

I just wanted to drop in and share some updates about the computer-aided tagging issue. We understand this task's importance and its impact on our community. Our team has been working hard to ensure we approach this task in the most effective and efficient way possible.

We aim to finish the evaluation phase in the next few months. Once we clearly understand the situation from the evaluation, we'll be in a better position to estimate when the solution can be implemented. If the solution is straightforward, we may also be able to implement it within this calendar year. However, we'll only know this once the evaluation is complete.

If you want to monitor how things are going, feel free to subscribe to the Phabricator ticket. We'll be posting updates there as we make progress.

We've been reading all your comments and suggestions - they've been super helpful in guiding us.

Thanks for hanging in there with us. Your patience means a lot. We're confident that together, we'll find a solution that works for our entire community. Udehb-WMF (talk) 10:47, 13 July 2023 (UTC)[reply]

"Your patience means a lot."
We don't believe you. Andy Dingley (talk) 10:53, 13 July 2023 (UTC)[reply]

Update - Evaluation Complete

[edit]

Hello everyone,

I'm writing to share the results of our evaluation. First and foremost, thank you for your patience.

After a thorough review, we have found that the tool has generated more inaccurate or misleading tags than accurate ones. As a result, we will be deactivating the tool on September 20, 2023, after completing the necessary code changes. You can visit the Phabricator ticket for more details on the evaluation.

We understand this tool was implemented to improve the tagging process, and we are also aware of your frustration with how long it has taken to address this issue. We sincerely apologize for any difficulties or disruptions this may have caused. We are committed to learning from this experience and will continue to explore alternative solutions that better serve the needs of our community.

Thank you for your understanding. --Udehb-WMF (talk) 12:19, 14 September 2023 (UTC)[reply]

Is there any way to work out how much of WMF's money was spent on a "thorough review" to determine that what the community here was almost unanimously telling you was, in fact, correct? - Jmabel ! talk 14:49, 14 September 2023 (UTC)[reply]
Thank you for your question, @Jmabel. I understand the concern behind it. The review was conducted internally by the Structured Content Team, so no additional financial costs were incurred. However, it did require a significant time investment. This evaluation was essential for due diligence and was undertaken in response to the community's call for the tool's deactivation. We believe that this time investment is crucial for ensuring that we're aligned with the community's needs and expectations. - Udehb-WMF (talk) 14:23, 19 September 2023 (UTC)[reply]
Aww… Corporate drone speak like this makes wonders for the reputation of WMF bigwigs among the editing community… -- Tuválkin ✉ ✇ 12:39, 23 September 2023 (UTC)[reply]
Thank you for doing this. The headline is that 734 of a sample of 1000 "tags" are classed as "bad". What is the plan for removing all the "bad" tags added since February 2020 (and, indeed, over the next week)? Presumably, removing all values added by the tool? Andy Mabbett (Pigsonthewing); Talk to Andy; Andy's edits 15:53, 14 September 2023 (UTC)[reply]
Thank you for bringing this up, @Andy Mabbett. While the WMF can't directly edit content, including removing tags, we're keen on finding a solution to remove inaccurate tags. We're open to collaborating with you or any other community member willing to help. Should technical assistance be needed, the Structured Content Team is available with some scheduling adjustments in the upcoming quarter to support the community by consulting or reviewing technical solutions that the community develops. We're here to provide the necessary support to ensure a successful resolution. -Udehb-WMF (talk) 14:25, 19 September 2023 (UTC)[reply]

This may come across as an unneeded potshot, and I'm sorry if it does, but I have an impression that's only solidified by this now discontinued WMF initiative (as also described by Andy Dingley further up this page, in June): We keep not getting things the community repeatedly asks for that would be really helpful for our everyday volunteer work on Commons, such as a better and directly integrated support for uploading and maintaining video content (the external volunteer-maintained tool video2commons seems to be broken more often than not, as its main developer is no longer active here), or a simple tool, preferrably integrated directly in the Commons interface as well, for mass uploads (OpenRefine is too complicated, in my opinion, and the other external tools again are often broken), or simply fixing the issues with the thumbnail generator... Instead we keep getting things no one has asked for and that create additional work instead of helping the volunteer community... Gestumblindi (talk) 17:30, 20 September 2023 (UTC)[reply]

There is a German figure of speech that translates to "The opposite of "good" is not "bad" but "good intentions". Sometimes I wish there was a formal on-wiki procedure for new features. Start with a pitch and a discussion phase and only start coding after positive feedback from the community. Require community consensus to move something out of beta ... --El Grafo (talk) 08:23, 21 September 2023 (UTC)[reply]
I don’t think that stunts like this have anything to do with good intentions — unless "good" is definied in a ver specific way. -- Tuválkin ✉ ✇ 19:37, 23 December 2023 (UTC)[reply]

Now what?

[edit]

Majority of tags seems to be junk. Now what? Remove them all? Multichill (talk) 10:57, 23 September 2023 (UTC)[reply]

Already being discussed, at Commons:Village pump/Proposals#Shutdown of Computer-aided tagging: Mass revert? Andy Mabbett (Pigsonthewing); Talk to Andy; Andy's edits 14:54, 23 September 2023 (UTC)[reply]
One year later, it’s still junk. Now’s old news, too. I like the idea of mass revert, but why stop there? Rather dismantle the whole thing, both in terms of code and infrastructure and also of funding and personnel: Fire anyone who ever sat at a WMF meeting and said that replacing Human input with auto-generated content is a good idea. Let’s instead make it known far and wide that Wikmedia projects are the one place on line that’s free from the scourge of so-called “AI”. -- Tuválkin ✉ ✇ 21:38, 23 September 2024 (UTC)[reply]
Please note User:SchlurcherBot/Mass revert computer-aided tagging. About 1.3 Million additions were reverted. --Schlurcher (talk) 22:11, 23 September 2024 (UTC)[reply]
@Schlurcher: Thanks! -- Tuválkin ✉ ✇ 20:42, 26 September 2024 (UTC)[reply]

Failed because it uses (only) machine vision instead of file categories

[edit]

I doubt SDC depicts statements are more than time-sink for contributors or useful in any way instead of being redundant to categories. However, if something like that was to be successful it would need to make use of the file's existing categories and suggest tags at least partly based on that. Machine vision could be used in addition with the file categories as context. For example, many items in some category may not 'depict' that thing but instead are only 'about' that thing and so on. Prototyperspective (talk) 11:40, 21 September 2024 (UTC)[reply]

Retrieved from "https://commons.wikimedia.org/w/index.php?title=Commons_talk:Structured_data/Computer-aided_tagging&oldid=929332353"