User talk:Magog the Ogre/cleanup.js/Archive 2

← Archive 1

Archive 2

Archive 3 →

Enhanced support for CommonsHelper2

If the cleanup script is executed for this version of a file transferred using CommonsHelper2

This file was originally uploaded at de.wikipedia as [http://de.wikipedia.org/wiki/File:Hausansicht.jpg Hausansicht.jpg], before it was transferred to Commons.

is changed to

{{original description page|de.wikipedia|Hausansicht.jpg}} Transferred to Commons.

Could you please make the script remove “Transferred to Commons.”, too? --Leyo 17:13, 9 August 2012 (UTC)

 Done Magog the Ogre (talk) (contribs) 18:57, 9 August 2012 (UTC)

Thanks for the quick fix. --Leyo 16:29, 17 August 2012 (UTC)

User name and real name

Is there any chance that if the author is given as [[User:Xyz|Real Name]] (example) the script does not change e.g.

{{GFDL|migration=relicense}}

to

{{GFDL-user-de|Xyz|migration=relicense}}

One solution would be to leave the template as is in such cases. Another would be to change it to

{{self|author=[[User:Xyz|Real Name]]|GFDL|migration=relicense}}

What do you think? --Leyo 20:56, 9 August 2012 (UTC)

Major fix up

I've been going through the code and cleaning up some of the coding problems. It is entirely possible that I have introduced bugs into the code by accident. As I do not use this code as often as some of you guys do, by all means report any new bugs or features I've removed here and I will attempt to fix them again. Magog the Ogre (talk) (contribs) 20:03, 10 August 2012 (UTC)

Enabling for upload form

What about enabling the script for the upload form, too? It would be handy for transfers that have to be done manually. --Leyo 15:02, 14 August 2012 (UTC)

 Done Magog the Ogre (talk) (contribs) 19:37, 17 August 2012 (UTC)

Thank you. I guess you did deliberately not add the buttons own and self-photographed. --Leyo 19:47, 17 August 2012 (UTC)

New Bug?

The script doesn't make changes like this anymore... --Brackenheim (talk) 15:35, 17 August 2012 (UTC)

You deleted the wrong part. The author has chosen a different way of attribution in the de.wikipedia original. --Leyo 16:28, 17 August 2012 (UTC)

Template:Other date

What about enabling more options of Template:Other date? It seems that especially the century is quite often used, at least in transfers from de.wikipedia (e.g. 16. Jahrhundert → {{Other date|century|16}}). --Leyo 11:33, 18 August 2012 (UTC)

My PHP script already does this, but it isn't easy to port over by hand. I wonder if it wouldn't be better for the script to use AJAX to retrieve the information off the toolserver. What do you think? Magog the Ogre (talk) (contribs) 18:13, 22 August 2012 (UTC)

I cannot comment on this, because it's too technical for me, sorry. --Leyo 18:14, 22 August 2012 (UTC)

According to EXIF data

I think the script should be using {{According to EXIF data}} template as well. Is there a reason why it was omitted? --Sreejith K (talk) 21:02, 21 August 2012 (UTC)

Can you give me an example? Magog the Ogre (talk) (contribs) 18:11, 22 August 2012 (UTC)

Cleanup v2!

OK, I've added a new button for "cleanup 2", which instead of using JavaScript, actually jumps over to the toolserver and fetches the text directly from the PHP I've already written and which frankly is both further along and more careful in the changes it makes. This new method for cleanup should address most of your concerns, but I am still leaving the old script because I know many of you rely on it, and because it takes less time than the new one (especially with the Toolserver acting buggy like it is right at this moment).

I will still take feature requests on both scripts (the cleanup v1 and cleanup v2). However, I would prefer to do most of my work on v2, because I do not want to have to write redundant code for both PHP and JavaScript (the differences in the programming languages makes perfect portability essentially undoable).

Let me know if you still have any concerns.

Cheers. Magog the Ogre (talk) (contribs) 06:57, 29 August 2012 (UTC)

Oh and one other thing to add: Keep in mind that cleanup v2 is literally just pulling up the same script as the one which User:OgreBot 2 will automatically perform on every upload on a daily basis. So any feature requests for that side of the tool must be super-duper extra-specially safe to perform. If there is a feature which is not 100% safe to perform, I will add it to cleanup v1, and you (the person making the request) can run both on the page. Magog the Ogre (talk) (contribs) 07:03, 29 August 2012 (UTC)

Why not calling the two versions “cleanup JS” and “cleanup TS”? --Leyo 09:47, 29 August 2012 (UTC)

Interesting version 2 feature: When clicking on "save", I am reminded that I haven't written any edit summary, although the usual "cleanup" message is present. --Stefan4 (talk) 11:41, 29 August 2012 (UTC)

That is not a bug with the bot, as much as it is a bug with Mediawiki, which improperly marks an edit summary as not filled in. Nonetheless, I will implement a fix. @Leyo: Yes, I will implement that fix. Magog the Ogre (talk) (contribs) 05:00, 30 August 2012 (UTC)

Both of these are now  Done Magog the Ogre (talk) (contribs) 18:29, 30 August 2012 (UTC)

author=The original uploader

Something went wrong here. --Leyo 21:09, 4 September 2012 (UTC)

 Fixed, I think. Magog the Ogre (talk) (contribs) 23:15, 8 September 2012 (UTC)

Cleanup category

Is it possible to add Cleanup link on categories as well. When you click on it, the script should run on all the files and auto save it. Or we can modify MediaWiki:VisualFileChange.js to have cleanup as one of the options. Whichever way it is, I think it will be handy if we can run the script on multiple pages at once. --Sreejith K (talk) 20:48, 5 September 2012 (UTC)

Were you hoping for cleanup JS, cleanup TS, or both to be able to run on a category? Magog the Ogre (talk) (contribs) 23:16, 8 September 2012 (UTC)

Either of this is fine. I wish we only had one of these and not two which is not helping at all. Or having a toolserver page where one can input a category name and click "Cleanup" button will also suffice. It's tedious to run the cleanup scripts on individual pages if the category is quite big. --Sreejith K (talk) 09:28, 13 September 2012 (UTC)

I appreciate that you're upset with two scripts, but if I already do most of my development in one language, it's really a huge pain to have to duplicate that development in another language. If you'd like, I can provide access to the source code of the PHP and you can make the changes yourself to the JavaScript. Magog the Ogre (talk) (contribs) 13:31, 13 September 2012 (UTC)

Anyway, I can provide a page on the Toolserver, but it may be difficult to allow the user himself to make the edits rather than the bot. Were you hoping for the bot to make it, or the user so that {{BotMoveToCommons}} can be removed? Magog the Ogre (talk) (contribs) 13:34, 13 September 2012 (UTC)

(EC) I don't think cleanup JS should run without supervision (checking the diff before saving). --Leyo 13:35, 13 September 2012 (UTC)

Removing {{BotMoveToCommons}} unsupervised is not a good idea. I was looking for a way of doing the basic cleanups like internationalizing the headers, adding {{User at project}} template, adding {{Uncategorized}} etc. for large number of images at one shot. The basic idea is to force OgreBot to run on a category. --Sreejith K (talk) 15:03, 13 September 2012 (UTC)

Yeah, I can write that script. For security reasons, I'll have to make it so it doesn't run on subcategories too, unless you can think of a safe way to allow it to run on subcategories. Magog the Ogre (talk) (contribs) 17:04, 13 September 2012 (UTC)

I was under the assumption that OgreBot is safe to be run on any file. I did not understand why you are hesitant to run it on sub-folders. --Sreejith K (talk) 00:29, 14 September 2012 (UTC)

Let me give you a scenario: User:TrollA decides to run the script on Category:Images. TrollA makes the same request, or a similar one 500 times. TrollA has just brought down the Toolserver and committed a borderline DOS attack on the WMF. Or another, more benign scenario: User:AdminA decides to run it on a category, but makes a typo, and now the bot is queued to run over 30,000 images! Do you see a way around either of these problems? Magog the Ogre (talk) (contribs) 05:01, 14 September 2012 (UTC)

May be limit the users to run it only on 500 images at a time, max? If required, we can have a "Next" button if users wish to run it on the next 500 or so. --Sreejith K (talk) 06:04, 17 September 2012 (UTC)

┌─────────────────────────────────┘
OK, it's done. It can be found here: tools:~magog/cleanup_multi.htm. It still doesn't do much in the way of idiot-checking yet, so please don't submit your request multiple times (I hope to make it more idiot proof shortly). It also has a hard-limit at 500 files, maximum, no ifs, ands, or buts. Magog the Ogre (talk) (contribs) 00:01, 20 September 2012 (UTC)

Is the link correct? I got a 404 error. --Sreejith K (talk) 06:45, 20 September 2012 (UTC)

Oops, the file got deleted. But it's back now, try again. Magog the Ogre (talk) (contribs) 13:53, 20 September 2012 (UTC)

Cleanup category sort keys

Hello Magog,

there are very often categories which are are sorted with a space as key article But your script removes the space and add the category name, for example. Should not be handled differently? Best Regards -- πϵρήλιο ℗ 10:49, 17 September 2012 (UTC)

 Done Magog the Ogre (talk) (contribs) 21:00, 29 September 2012 (UTC)

Thank you! Up again. Regards -- πϵρήλιο ℗ 19:20, 5 October 2012 (UTC)

Bot cleanup

See this edit. If the bot can cleanup the description, the JS should also do it IMHO. --Sreejith K (talk) 05:16, 30 September 2012 (UTC)

投稿者撮影

Can you tell the script to convert "投稿者撮影" into {{Self-photographed}}? --Stefan4 (talk) 14:08, 5 November 2012 (UTC)

 Done @Stefan4: added, along with a bit of variations. --whym (talk) 11:56, 29 December 2013 (UTC)

Yesno

Can you modify the script to remove {{Yesno}} template? CommonsHelper is adding this template in the license section which is pretty useless. But if the template has a parameter, the script should probably leave it intact. --Sreejith K (talk) 17:03, 20 December 2012 (UTC)

The TS version of the tool is already removing it; do you have a need to have the JS do it as well? Magog the Ogre (talk) (contribs) 22:43, 21 December 2012 (UTC)

Its a good to have feature, but its OK if you do not add it. I used AWB and removed all existing usages of it. I feel CommonsHelper needs to be modified to exclude this template. --Sreejith K (talk) 03:57, 22 December 2012 (UTC)

@Sreejith K: Good idea. Could you please ask Magnus? Also == {{int:license}} == should be changed to == {{int:license-header}} ==. --Leyo 03:15, 18 September 2013 (UTC)

If I had more time, I would ask Magnus to be a coprogrammer so I could fix up his script. He doesn't respond very well to requests that I've noticed, because he is on even less than I am. Magog the Ogre (talk) (contribs) 22:51, 18 September 2013 (UTC)

BTW: {{Center}} is another such template (example). --Leyo 07:07, 19 September 2013 (UTC)

Problem with ambiguous dates

Try "cleanup TS" at Special:PermanentLink/88939002. The date is listed as 7-9-09 which could be either 2009-09-07 or 2009-07-09. This script makes a guess and thinks that it is 2009-09-07. I don't think that it is safe to do this as the cleanup tool can't know what the uploader meant. In this particular case, it is obvious that the uploader meant 2009-07-09 as the file was uploaded to Wikipedia before 2009-09-07. --Stefan4 (talk) 10:21, 27 January 2013 (UTC)

Good catch. That is now  Done. Magog the Ogre (talk) (contribs) 19:58, 27 January 2013 (UTC)

Malfunctions

Hello Magog, I wondering that nobody reports this, the JS has since some days a general failing. It duplicates headlines and line-endings of categories!? (FF18) Best regards -- ΠЄΡΉΛΙΟ ℗ 15:10, 31 January 2013 (UTC)

Please point me to an old version of a page where it will cause a problem. Magog the Ogre (talk) (contribs) 01:37, 1 February 2013 (UTC)

On all file-pages example, but it seems only FF18 (Windows7, Chrome works fine). :-o -- ΠЄΡΉΛΙΟ ℗ 02:19, 1 February 2013 (UTC)

It's working just fine for me on Chrome and FF18 for Ubuntu. Is anyone on Windows that can confirm this bug? Magog the Ogre (talk) (contribs) 02:22, 1 February 2013 (UTC)

Edit summary only if applicable

If I remember correctly, you had once implemented this request, but it does not work anymore. --Leyo 07:59, 1 February 2013 (UTC)

 Done Magog the Ogre (talk) (contribs) 00:45, 13 February 2013 (UTC)

Thank you. --Leyo 07:33, 13 February 2013 (UTC)

Pipes

Hi Magog. I suggest implementing to the script (and the bot) to fix such small parameter errors. This is a rather frequent case and IMO also safe to fix automatically. Occasionally, there are also cases like ||Source, but automated fixing them might be a more delicate issue (i.e. to avoid false positives). --Leyo 05:05, 13 September 2013 (UTC)

+1 yes, unbelievable how often this occurs. -- Perhelion (talk) 15:41, 13 September 2013 (UTC)

 Done [1] Magog the Ogre (talk) (contribs) 22:48, 16 September 2013 (UTC)

Thank you. Could you please implement it to the .js version, too? And maybe also such and similar cases, if you consider them to be safe enough. --Leyo 07:18, 17 September 2013 (UTC)

I could not program the same functionality into the JavaScript, because the PHP has a complex template parsing system that the JavaScript lacks. The JavaScript doesn't know where the template begins and where it ends. As for the second case, it's already programmed into the bot. If you try to run Cleanup TS on the old version of the file, you'll see that. Magog the Ogre (talk) (contribs) 00:03, 18 September 2013 (UTC)

OK. Thank you for the explanation. --Leyo 03:12, 18 September 2013 (UTC)

"cleanup" removed cats, other info

Seems worse than before running it: diff. Is this a common problem with this bot? --Pete Tillman (talk) 16:35, 15 September 2013 (UTC)

No; this was mostly a change I made by hand. I a) removed the false date, and b) replaced the secondary source with the original. You're correct I didn't restore the authorship properly, but it was a poorly formatted copypaste to begin with. Magog the Ogre (talk) (contribs) 21:27, 16 September 2013 (UTC)

That was a change that I mostly made by hand (check the user name). I'm not convinced that there is any point in listing the text "work of a National Park Service employee, created as part of that person's official duties" in the {{Information}} template as that information appears in the copyright tag. --Stefan4 (talk) 21:58, 16 September 2013 (UTC)

Agree re the "work of a National Park Service employee...." bit, which I think was an old copy-paste by me, long ago (I'm the original uploader). Glad to hear it's not the bot! Cheers, Pete Tillman (talk) 23:26, 16 September 2013 (UTC)

Adding author

Hello Magog the Ogre, is it possible, to make changes like this with your script? The link to the userpage could be copied from "Original upload log". Regards, --Brackenheim (talk) 21:18, 15 February 2014 (UTC)

...changes like this. --Brackenheim (talk) 22:44, 22 March 2014 (UTC)

I would not want to include that automatically in the script due to concerns brought up by Leyo. We should be honoring the way that the original uploader wishes to be identified for best compliance with GFDL/CC/etc. Magog the Ogre (talk) (contribs) 19:55, 23 March 2014 (UTC)

Ok. --Brackenheim (talk) 20:00, 23 March 2014 (UTC)

The question here is just unlinked vs. linked when the Author parameter values is the same as the uploader's user name.

My concern addresses cases when they are different. The Author parameter value is indeed essential for e.g. CC-BY licenses. --Leyo 22:12, 23 March 2014 (UTC)

Out of function since months

The script is unusable since months I thought someone report a bug or something. Maybe it's an incompatibility of another of my scripts (but I don't really think so). It duplicates every line break and every head line and something more or nothing. What sometimes works a bit better is the toolserver version. Also the edit summary get completely cleared (if there is something). I thought maybe this tool is discontinued!? So I've not used it for months (may since v2). It would be a pity therefore. Greetings PS.: Maybe you can join with AutoEd or AutoWikiBrowser script!? -- Perhelion 16:15, 31 May 2014 (UTC)

Which operating system, language, and browser are you using? Magog the Ogre (talk) (contribs) 20:39, 31 May 2014 (UTC)

@Perhelion: Magog the Ogre (talk) (contribs) 13:41, 2 June 2014 (UTC)

Hej, good to see you. :-) I've Win7 x64 Firefox 29 (also Nighty), in Chrome (35) it works! But I've test it short, there are some negative differences in the TS version, the summary get deleted and (for example) double categories get not removed but in the JS version. -- Perhelion 15:55, 2 June 2014 (UTC)

@Perhelion: It's working just fine for me in Firefox 29, albeit for Linux. Let's try some other questions:

• Do you have any plugins installed? My NoScript plugin tried to filter the request.
• What is the Mediawiki skin you're using?
• Which page are you having a problem on?

I hate to pester you with questions, but no one else has reported this problem. If we still can't find it after you're done, I suppose I can try in Win7 x64 Firefox 29 myself (not nightly though).

As for the other things you mention, these are known defects. I essentially abandoned the JS version years ago in favor of the TS version, because I don't want to keep two sets of code! I would be happy to open up my source code to you though and you could improve either version. Magog the Ogre (talk) (contribs) 22:58, 2 June 2014 (UTC)

@Perhelion: - I've located this bug, now! It's the TS version that's not working, not the JS. I totally misunderstood. I am working on a fix; hopefully I can do it elegantly. It may take a bit though. Magog the Ogre (talk) (contribs) 00:12, 16 June 2014 (UTC)

Hej, good to hear, but unfortunately I meant mainly the JS version, I can't say what it is. I don't use FF anymore, but I've tested for you something. Nighty 33 now, no addons installed, most user scripts off. Same happens. I used some scripts with FF too but nothing this strange happens (in compare to Chrome). :-/ If you work on the TS version and you (or anybody) can't reproduce this, then we let to bug as it is. → Perhelion     21:00, 16 June 2014 (UTC)

JS version works fine for me, but the diff view in TS version appears to be broken: The diff shows that the whole content was deleted in the edit. --Leyo 21:16, 16 June 2014 (UTC)

@Leyo: see w:WP:VPT#Posting data a diff from an external site. Magog the Ogre (talk) (contribs) 21:32, 16 June 2014 (UTC)

Oh, thanks. Let's hope someone will come up with a solution. --Leyo 23:07, 16 June 2014 (UTC)

Character set problem

This is what happened when I tried to clean up a page. 200×78× was changed into some error because of a character set problem. Note that I used both TS and JS cleanup in that edit, but I think that the error came from the TS cleanup. --Stefan4 (talk) 20:30, 19 June 2014 (UTC)

@Stefan4: I am not able to duplicate this error. Please do the following:

• Try again by clicking here and resubmitting.
• If that doesn't work, try here.

If one of the above doesn't work, please tell me which one failed, as well as your operating system, browser, and default OS language. Magog the Ogre (talk) (contribs) 22:56, 19 June 2014 (UTC)

When I click on link 1 and use "cleanup TS", my browser posts a request to your website. Your website returns a page in UTF-8 encoding which my browser misinterprets as w:Shift-JIS. Some byte sequences are forbidden in Shift-JIS, see w:Shift-JIS#Shift JIS byte map (the first byte of a double-byte character must be followed by a valid second byte). For example, in Shift-JIS, 0x97 (second byte of UTF-8 "×") can't be followed by 0x37 (UTF-8 7) or 0x20 (UTF-8 space). As my browser detects a forbidden byte sequence, it probably corrupts the data. After corrupting the data, my browser runs document.getElementById('thebutton').click(); and posts a request to the server, which ends up as corrupted text in the edit box.

If I use link 2, which goes directly to the toolserver page, my browser magically thinks that the page is in UTF-8, so the correct result ends up in the edit box on Commons.

Would you be able to change Content-Type: text/html into Content-Type: text/html; charset=utf-8 in the server's response header or alternatively add a character encoding tag to the HTML source code?

By the way, is the script really supposed to send wpDiff twice with different values? --Stefan4 (talk) 00:20, 20 June 2014 (UTC)

Part of the post request, with browser+OS data and two wpDiff values

POST /w/index.php?title=File%3AColonial_Brookwood_Village_logo.png&action=edit&functionName=post_cleanup_ts HTTP/1.1 Host: commons.wikimedia.org User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:30.0) Gecko/20100101 Firefox/30.0 [...] -----------------------------2146827590176785334780215406 Content-Disposition: form-data; name="wpDiff" wpDiff [...] -----------------------------2146827590176785334780215406 Content-Disposition: form-data; name="wpDiff" Click me if you have JavaScript disabled -----------------------------2146827590176785334780215406--

Try it again now. I already made several changes to the file - including fixing the wpDiff. I committed those changes too. Magog the Ogre (talk) (contribs) 03:30, 20 June 2014 (UTC)

Thank you, it works now! --Stefan4 (talk) 12:17, 20 June 2014 (UTC)

"ListFiles/Username"

Special:Diff/145114640 is only "cleanup JS" with no other manual editing. Why does the script change the attribution in the GFDL template from "Username" to "ListFiles/Username"? --Stefan4 (talk) 22:45, 2 January 2015 (UTC)

BTW: There are a few more cases. --Leyo 22:50, 2 January 2015 (UTC)

I am aware of both bugs. I have resisted fixing them because I have not wanted to fix this script, but maybe I'll take a look at that and a few other small bugs before I finally give up on it. Magog the Ogre (talk) (contribs) 02:10, 3 January 2015 (UTC)

@Stefan4, Leyo: fixed, I believe. Magog the Ogre (talk) (contribs) 23:22, 16 January 2015 (UTC)

I took one of the search results above and applied the script to the previous version of the file page. And indeed, it works. :-) Thank you. --Leyo 00:27, 17 January 2015 (UTC)

Major refactoring

I did a major refactor today for code clarity. I am hoping not to see any issues, but please inform me if anyone has any problems. Magog the Ogre (talk) (contribs) 05:17, 3 December 2014 (UTC)

Your script doesn't work anymore (using FF 34.0). Yesterday and the day before the browser asked me to stop the script, today nothing happened at all. Neither the JS Nor the TS choice did work. Can you help? Thanks, I miss the tool a lot, --Emha (talk) 11:01, 9 December 2014 (UTC)

Now the browser speaks again saying
Ein Skript auf dieser Seite ist eventuell beschäftigt oder es antwortet nicht mehr. Sie können das Skript jetzt stoppen, im Debugger öffnen oder weiter ausführen.
Skript: https://commons.wikimedia.org/w/index.php?title=User:Magog_the_Ogre/cleanup.js&action=raw&ctype=text/javascript:127 Perhaps that helps. --Emha (talk) 12:40, 9 December 2014 (UTC)

Sorry, I forgot to ping you, @Magog the Ogre: Both links, cleanup JS and cleanup TS still don't work today. This time no reaction at all. Could you help, please? Thanks, --Emha (talk) 14:08, 15 December 2014 (UTC)

@Emha: for TS: try it now, it should work. If it doesn't, please ensure you don't have any browser add-ons which might cause the script not to submit (e.g., NoScript). If that doesn't work either, let me know, and I will try one last solution. For JS: I don't know what to tell you; it is working for me without a problem. Is it possible that your Firefox profile is causing everything to go slowly? Magog the Ogre (talk) (contribs) 02:01, 16 December 2014 (UTC)

@Magog the Ogre: it works now! Thanks a lot, --Emha (talk) 08:47, 16 December 2014 (UTC)

COAInformation

Hello Magog the Ogre,

I would only mention, that the JS version seems crash always on CoA with {{COAInformation}}, example File:COA family de Landgrafen von Hessen.svg. Eventual the toollabs version could bring the server also in trouble... (I'm using Chrome on Win7 x64) Regards ↔ User: Perhelion (Commons: = crap?)  14:03, 10 February 2015 (UTC)

@Perhelion: it's working just fine for me on Chrome. Magog the Ogre (talk) (contribs) 03:22, 11 February 2015 (UTC)