Commons:Bots/Requests/Gzen92Bot-6

Gzen92Bot (talk · contribs) 6

Operator: Gzen92 (talk · contributions · Statistics · Recent activity · block log · User rights log · uploads · Global account information) (assign permissions)

Bot's tasks for which permission is being sought: File upload from gallica.bnf.fr

Automatic or manually assisted: Automatic.

Edit type (e.g. Continuous, daily, one time run): Occasionally.

Maximum edit rate (e.g. edits per minute): About 20.

Bot flag requested: (Y/N): Nothing.

Programming language(s): PHP
After Commons:Bots/Requests/Gzen92Bot-4, I continued to upload files from Gallica (Category:Bibliothèque nationale de France), using the parameter "any fayes" = royalty free document (see API), without requesting bot authorization (the infobox is formatted the same).
But there is a problem with categories (see blocking Commons:Village pump#Obtuse bot created categories).
When to create a category? 2, 5, 10 files from the same id?
How to name the category? File name or only the id?
Without being able to answer this question automatically (there are a few million files available at the BnF), I will simply leave them in Category:Images from Gallica and Category:Files from Gallica needing categories (images). See 10 files uploaded.
Gzen92 (talk) 17:20, 1 November 2024 (UTC)

Discussion

Some topical categorization should be determined for every file uploaded, in addition to merely adding a source category.

Before uploading any new file. A cleanup plan for the thousand of categories created before (Commons:Village pump#Obtuse bot created categories) needs to be found and implemented (by the uploader or somebody else).
 ∞∞ Enhancing999 (talk) 08:11, 2 November 2024 (UTC)

No problem for this action, if it is the target. Gzen92 (talk) 10:12, 2 November 2024 (UTC)

Please outline your cleanup plan for the existing categories.
 ∞∞ Enhancing999 (talk) 11:38, 2 November 2024 (UTC)

I don't think this is the place to talk about this but it is easy to take the files out of the subcategories of Category:Files from Gallica needing categories (images) and put them in Category:Images from Gallica and Category:Files from Gallica needing categories (images).

The categories can be deleted easily (I don't have the bot permission to do that).

But I still think it's good to group the files, even if the category name could be improved.

Gzen92 (talk) 22:00, 2 November 2024 (UTC)

As a bot operator, I'd expect you to fix problems you created before potentially creating more of them. If you can't present a plan for your past uploads, I don't think this request should be approved. Commons is not a place to dump uncategorized files.
 ∞∞ Enhancing999 (talk) 10:43, 3 November 2024 (UTC)

Topical categorisation is not required, see Commons:Guide_to_batch_uploading#Categories. Also, before a "cleanup plan" has to be presented, clear arguments have to be presented as to why those existing categories are a problem. ~TheImaCow (talk) 22:04, 2 November 2024 (UTC)

Which part are you referring to? Uploaders are required to add categories. Merely adding a user category isn't sufficient.
 ∞∞ Enhancing999 (talk) 10:43, 3 November 2024 (UTC)

Section "Putting it into action" -> "every file you upload should have: a tracking/source category, Your files can have: topic categories". A source category + a {{Check categories}} (substituted with the "category needed" category here) is sufficient. ~TheImaCow (talk) 13:57, 3 November 2024 (UTC)

If we agree that both categories are enough, should we create categories to group files or not? For example this category includes a whole book but the name is long. Gzen92 (talk) 21:44, 3 November 2024 (UTC)

Anyone have an opinion ? With or without category creation ? I would like to continue the job ! Gzen92 (talk) 07:10, 7 November 2024 (UTC)

I don't understand what exactly is requested here. Please say again. --Krd 14:47, 7 November 2024 (UTC)

I upload files and create categories as soon as there are two files in the Gallica folder. Which creates a lot of categories: Category:Files from Gallica needing categories (images) 65,000 files and 7,200 categories which contain 139,000 files (average 19).

Remarks on Commons:Village pump#Obtuse bot created categories, I could create categories from 10 files. Gzen92 (talk) 07:43, 8 November 2024 (UTC)

I still cannot follow. Can you give an example? Krd 08:12, 8 November 2024 (UTC)

I create a lot of categories with few files (for example Category:Hotel de Roquelaure - Juillet 1906 - photographie - Atget - btv1b10516512m). So a lot of categories in Category:Files from Gallica needing categories (images). Two solutions (I don't know): threshold to create a category (10 files?) or no category at all in Category:Files from Gallica needing categories (images). Gzen92 (talk) 08:22, 8 November 2024 (UTC)

I think a bot request is not the right place to find a decision how to proceed. Perhaps this is better discussed as com:Village pump. Once there is a decision, we can perhaps discuss what is the best way to clean up, if required. (If I'm mistaken, please advise.) Krd 08:59, 8 November 2024 (UTC)

I am not sure if this is the right forum, but I am wondering how exactly the copyright status of the images is ascertained. For instance, this image has been uploaded as CC0 despite the linked source page containing:

Note(s) : Toute reproduction doit faire l'objet d'une autorisation préalable de(s) auteur(s), de ses (leur) ayants-droits ou de la société qui les représente

Reproduction : Numérisation effectuée par l'auteur d'une sélection de photographies argentiques

which to me sounds as if BNF does not own the right to the image in the first place. Could you elaborate? Felix QW (talk) 13:20, 11 November 2024 (UTC)

Hello, there is a parameter in the API Gallica "public domain". But you are right, in some html pages it is indicated "specific conditions of use", I know how to exclude them for the future. Gzen92 (talk) 16:29, 11 November 2024 (UTC)

@Krd: After discussions :
Subcategory by year in Category:Files from Gallica needing categories (images), for example Category:Files from Gallica needing categories (images of 1930).
No category for 2 files because often reverse sides (I will manually browse the categories by year to visually identify the reverse sides and move them to Category:Files from Gallica needing categories (images, reverse side).
Category with 3 or more files, also category by year, for example Category:Files from Gallica needing categories (images of 1870).
See 24 edit.
Gzen92 (talk) 18:27, 22 November 2024 (UTC)

Please comment on the current block of the bot. Krd 05:24, 5 December 2024 (UTC)

After Commons:Bots/Requests/Gzen92Bot-4, I continued to upload files from Gallica (Category:Bibliothèque nationale de France), without requesting bot authorization (the infobox is formatted the same). There are a lot of files (> 100,000). Blocked to think about a classification solution. I will be more vigilant about the classification and will make a request when other cases of mass loading arise. Gzen92 (talk) 10:09, 5 December 2024 (UTC)

I'm not sure if I can follow. Does that mean this request is withdrawn until further notice? Krd 10:23, 5 December 2024 (UTC)

No, this request is valid. There was a discussion about how the bot works. In the future, if I have any doubts about the right way to do this, I will open a request. Please unblock the bot thanks. Gzen92 (talk) 11:11, 5 December 2024 (UTC)

Ok, so please say in simple language what this job is exactly about. What exactly will be performed now if this request is approved? --Krd 11:51, 5 December 2024 (UTC)

Import file from gallica.bnf.fr (API).

Category with BNF number if 3 files or more.

File/category in categories by date (fewer files at the root).

At the end, visual search for "white" back faces to isolate them.

But before starting the import, categorization of existing files/categories in Category:Files from Gallica needing categories (images) by date as indicated above.

(sorry i use a translator) Gzen92 (talk) 12:56, 5 December 2024 (UTC)

I'd say please do the next (small) batch and please advise when done. Perhaps it's easier to see the edits. Krd 13:37, 5 December 2024 (UTC)

Some modifications of files and categories. Gzen92 (talk) 15:37, 5 December 2024 (UTC)

Please continue slowly. I think this can be approve if no objection arises. Krd 16:15, 5 December 2024 (UTC)

Categories are the least of the problems with that bot. Much more problematic is that the bot systematically inserts false claims of CC0 on almost everything it uploads. Sometimes, it falsely tags CC0 files that are public domain for some other reason, which is already bad behaviour but at least it's not too damaging. Often, it falsely tags CC0 files that are not free, which is much worse and potentially much more damaging. To evaluate the size of the problem so far, searching only for photos by the professional photographer Daniel Cande, it can be found that the bot uploaded thousands of non-free photos by that photographer. It can be expected that looking at more uploads of the bot, more copyvios of non-free works by other authors might be found. Mass uploads should not be initiated without diligently checking the status of the works. After being notified by another user, the operator of the bot opened a deletion request for 107 copyvios. That's ok, but that adresses less than 1% of the thousands of copyvios uploaded by the bot. IMHO, any new uploads by that bot should be entirely disallowed until the operator meets at least minimal conditions: 1) Gives convincing assurances that the bot will stop faking CC0 dedications on works that are not CC0. 2) Reviews all past uploads. 3) Takes quick action to ensure that all copyvios get deleted as soon as possible. 4) Corrects the status tags of files that may be kept. The complete cleanup of all past uploads should be done before new uploads are allowed. -- Asclepias (talk) 16:40, 8 December 2024 (UTC)

Those are reasonable conditions.   — 🇺🇦Jeff G. ツ please ping or talk to me🇺🇦 17:04, 8 December 2024 (UTC)

Hello, https://api.bnf.fr/api-gallica-de-recherche, parameter "access", "fayes" = public domain. But yes, sometimes it's not the case. I have already modified the code to use the field "dc.rights" ("restricted use (convention BnF-ADM-xxxx-xxxxxx-xx)") in API returns. I can find all the false positives. I will answer on Commons:Help_desk#What_to_do_with_thousands_of_bot_copyvios_?, Gzen92 (talk) 07:25, 9 December 2024 (UTC)

Bonjour Gzen92, Je place mes quelques commentaires et suggestions dans la boîte ci-dessous parce que c'est un peu long. -- Asclepias (talk) 20:23, 9 December 2024 (UTC)

Comments and suggestions in French

Bonjour Gzen92, Tout d'abord, je veux dire que je ne doute pas de votre bonne foi. Mais malheureusement je dois aussi dire qu'à mon avis il ne devrait pas être envigeable d'autoriser un bot à faire des versements massifs sur Commons à moins que son opérateur ait démontré de façon convaincante qu'il possède un bon niveau de maîtrise des conditions fondamentales des droits d'auteur des pays concernés ainsi que des modèles de statut de Commons, leur signification et leur usage. En ce qui concerne des versements depuis les fonds de Gallica, cela implique donc en particulier la bonne maîtrise des conditions des droits d'auteur de Commons:Droit d'auteur par territoire/France et de Commons:Droit d'auteur par territoire/États-Unis. Un certain nombre d'éléments plus spécifiques sont particulièrement préoccupants. 1. D'autres utilisateurs ont déjà souligné le problème que le bot utilise abusivement le modèle CC0, alors qu'en réalité les fichiers qu'il verse sur Commons ne sont pas sous CC0. Or, vous ne semblez pas avoir donné d'indication claire que vous ayez vraiment saisi la nature de ce problème. Seul le propriétaire des droits d'auteur (donc normalement l'auteur lui-même ou son ayant droit) peut émettre une déclaration CC0, par laquelle il déclare qu'il transfère au domaine public son oeuvre qui n'était pas déjà dans le domaine public. Le bot (ni personne d'autre d'ailleurs) ne peut pas prendre l'initiative d'émettre une déclaration CC0. Étant donné d'une part que le texte de la déclaration CC0 n'existe que depuis 2009 et d'autre part que les items versés par le bot depuis Gallica ont une origine beaucoup plus ancienne, il est évident qu'aucun (ou sinon presque aucun) de ces items n'a fait l'objet d'une déclaration CC0 émise par son auteur ou par un ayant droit à une époque où l'item était sous droits d'auteur. Les pages de description des fichiers versés par le bot doivent être corrigées en conséquence, afin de remplacer la fausse déclaration CC0 par les modèles de statut appropriés dans chaque cas. Dans une grande proportion des cas, lorsque l'item est dans le domaine public en France parce que les droits d'auteur y sont expirés, un modèle approprié pourra être par exemple PD-old-auto ou une autre variante de PD-old-70. Il faut ajouter aussi un modèle décrivant le statut de l'item aux États-Unis. Le tout implique de passer en revue tous les versements qu'a faits le bot et, selon le cas, de corriger les pages de description des fichiers libres et de faire supprimer les fichiers non libres. 2. Vous avez constaté qu'il est impossible de s'appuyer inconditionnellement sur les paramètres techniques ni même sur les notices visibles de Gallica. Les notices sont souvent inadéquates ou sans signification utile. Par exemple, les notices disant «consultable en ligne» n'informent pas du statut des droits d'auteur. Les utilisateurs de Commons ont souvent constaté que des items apparemment notés comme libres sur Gallica étaient en réalité non libres. Et à l'inverse que des items notés comme non libres étaient en réalité libres. Il est nécessaire d'évaluer le statut véritable de chaque fonds et même parfois de chaque item dans certains fonds. Dans le doute, le bot devrait probablement s'abstenir de verser les items. Vous pouvez aussi demander des avis sur des cas spécifiques à Commons:Village pump/Copyright. 3. Le bot a reçu depuis un an plusieurs notifications relatives à des items inclus dans des recueils de coupures de journaux que le bot a versés. Toutefois, bien que ces quelques items notifiés ont été supprimés, vous ne semblez pas avoir réagi pour faire le ménage dans ces recueils. Ces recueils sont particulièrement problématiques puisqu'ils groupent généralement des oeuvres de nombreux auteurs différents et puisque les droits d'auteur sur les items d'un recueil peuvent expirer à des dates différentes, selon les années de décès de leurs auteurs respectifs. Par exemple, une grande part des divers items (textes, illustrations ou photos) inclus dans des recueils datant des années 1950 ou même des années 1940 sont vraisemblablement encore sous droits d'auteur en France puisqu'il est peu probable que tous leurs auteurs soient morts il y a plus de 70 ans, soit avant 1954. Si en plus on souhaite que les oeuvres soient libres aux États-Unis, il faut généralement que les auteurs soient morts avant 1937 ou que l'auteur ait publié l'oeuvre avant 1929. Si vous souhaitez verser sur Commons des fichiers provenant de tels recueils d'items hétéroclites, il est nécessaire de prendre le temps d'examiner les droits d'auteur de chaque item et de séparer les items libres des items non libres. Cela est fastidieux. Il ne serait pas acceptable de reporter ce fardeau sur les autres utilisateurs de Commons. Vous préférerez peut-être plus simplement éviter de verser les items de tels recueils. Mais, d'une façon ou de l'autre, il faut quand même faire le ménage dans les recueils déjà versés par le bot. -- Asclepias (talk) 20:23, 9 December 2024 (UTC) Bonjour, j'accepte tes remarques tout à fait pertinentes. Tu en sais plus que moi sur le sujet des droits d'auteur et cela est effectivement plus compliqué qu'il n'y paraissait de prime abord. Je vais me pencher sur la correction des fichiers déjà chargés afin de pouvoir en tirer une conclusion pour la suite : automatisable ou pas. Cordialement, Gzen92 (talk) 15:49, 11 December 2024 (UTC)

In short: IMO, a bot should not be authorized to mass upload files unless its operator demonstrates sufficient knowledge of copyright notions and rules and diligence to apply them, especially when the items are from a repository such as Gallica with a complex mixture of free and non-free items. -- Asclepias (talk) 20:23, 9 December 2024 (UTC)

I totally agree with what others have already said about the licensing issues with the bots uploads. It looks like the bot has been unblocked while still wrongly uploading files as Creative Commons CC0 1.0 when it shouldn't be. Or at least the bot should be adding the appropriate license for the country of origin along with Creative Commons license. Just because the Bibliothèque nationale de France released the images under a Creative Commons CC0 1.0 license on their end doesn't mean the images aren't or wouldn't still be copyrighted in the country of origin and/or shouldn't have the appropriate license for said country. It's not the job of other users to sift through thousands of images after they are uploaded to make sure the licenses are correct. --Adamant1 (talk) 08:52, 12 December 2024 (UTC)

The bot has not uploaded anything since the unblocking. Gzen92 (talk) 13:56, 12 December 2024 (UTC)

I'd agree that a bot owner should take care of any mess created by their bot. Krd 10:29, 12 December 2024 (UTC)

I can do it, I have all the logs of the files created and the data from the Gallica API to make the necessary adjustments. At the moment, I don't know how to solve the problem, since the rights are complex to manage (Gallica's rights are not precise enough). Gzen92 (talk) 13:58, 12 December 2024 (UTC)

Please consider to provide details somewhere, perhaps in the bot's user space. Krd 14:20, 12 December 2024 (UTC)

@Gzen92: For those files with insufficient licensing info in the Gallica API, please check the Gallica website and either document the surricient licensing info or request deletion.   — 🇺🇦Jeff G. ツ please ping or talk to me🇺🇦 18:52, 12 December 2024 (UTC)

For information, deleting requested files Commons:Deletion requests/restricted use (convention BnF-ADM-xxxx-xxxxxx-xx) - files. I will make requests for the categories too, apparently it will be less problematic. Gzen92 (talk) 06:55, 16 December 2024 (UTC)

There are authors which are in the public domain in this DR. Why adding them? Yann (talk) 08:59, 16 December 2024 (UTC)

All these files are indicated "restricted use (convention BnF-ADM-xxxx-xxxxxx-xx)" by the BnF. I did this automatically. But yes it does work for other people, I thank them. For after, the 2793 categories, I will look at the content one by one. It is often plays, shows or films. It will be easier to know if it is recent or not. Gzen92 (talk) 09:07, 16 December 2024 (UTC)

Thanks, but this is a real bad way to create a DR. With a mix of PD and non-PD files, deletion can't be automated. The right way is to create a DR by author. This will be also make undeletion easier when the time comes. Yann (talk) 09:27, 16 December 2024 (UTC)

You are right, i will do better for the categories. Gzen92 (talk) 09:47, 16 December 2024 (UTC)

 Info Commons:Deletion requests/restricted use (convention BnF-ADM-xxxx-xxxxxx-xx) - files almost finished (thanks everyone).

 Info Commons:Deletion requests/restricted use (convention BnF-ADM-xxxx-xxxxxx-xx) - categories in progress.

Gzen92 (talk) 16:22, 20 December 2024 (UTC)

@Gzen92: Cela fait beaucoup de pages de deletions avec peu de fichiers dans chacune. Pourquoi ne pas faire plutôt une demande de suppression par auteur ? Cordialement, Yann (talk) 18:15, 20 December 2024 (UTC)

I'm not following what is happening. The bot has now performed thousands of edits to list files and categories for deletion. Are we now discussing the cleanup task here, or the new task. --Schlurcher (talk) 21:57, 20 December 2024 (UTC)

I was indicating for information the progress to repair my errors. After I would know better how to load the right files. But you are right, the deletion does not concern this request. Gzen92 (talk) 10:24, 21 December 2024 (UTC)