From DoomWiki.org

(Report on Wikipedia linking?)
("PWAD top-5" statistics)
(47 intermediate revisions by 7 users not shown)
Line 1: Line 1:
 +
 +
 
== Archived discussions ==
 
== Archived discussions ==
 
<div class="dw-div-dl">'''[[User talk:Xymph/2016|2016]]'''</div>
 
<div class="dw-div-dl">'''[[User talk:Xymph/2016|2016]]'''</div>
Line 226: Line 228:
  
 
::: I added the main space data. Okay, yes, it's voluminous. :) --[[User:Xymph|Xymph]] ([[User talk:Xymph|talk]]) 06:51, 27 March 2018 (CDT)
 
::: I added the main space data. Okay, yes, it's voluminous. :) --[[User:Xymph|Xymph]] ([[User talk:Xymph|talk]]) 06:51, 27 March 2018 (CDT)
 +
 +
:::: There's so much progress here, people can't help being inspired to further suggestions!&nbsp; :>&nbsp;&nbsp; I hope you won't interpret it as criticism if I make suggestions before testing any bibliographic edits &mdash; it's a big job and I don't want people to have to revert me later.
 +
 +
:::: Regarding ''mass uploaded images'', you have apparently handled this, by counting links in template space once per template instead of once per transclusion.
 +
 +
:::: You're probably correct to parse wikitext instead of HTML; it provides more fine-grained results.&nbsp; Note that:
 +
::::* {{c|<nowiki>[[wiki:</nowiki>}} also points to Wikipedia.
 +
::::* I'm a regex newbie, but the above doesn't seem to include MediaWiki's case insensitivity on the first character.&nbsp; E.g. {{c|<nowiki>{{wp</nowiki>}} and {{c|<nowiki>{{Wp</nowiki>}} are the same.&nbsp; {{wp|fair use}} and {{wp|Fair use}} are the same.&nbsp; Further, interwiki prefixes are [[wIKIpEDia:Help:Interwiki linking#Project titles and shortcuts|completely case-insensitive]].
 +
::::* Similarly, {{wp|Blake Stone: Aliens of Gold}} and {{wp|Blake_Stone:_Aliens_of_Gold}} are the same.
 +
::::* I endorse Gez's idea of a third column: direct links are far more likely to be unmarked sources, at least on the articles I've seen.
 +
 +
:::: If the consensus is not to filter mainspace links (time for that general discussion&nbsp; :D&nbsp;)&nbsp; then there will be many more ideas and many more possible mass edits to implement them.&nbsp; I personally still intend to focus on the task in the OP, and would heuristically skip entries clearly outside it ({{wp|Blu-Ray}}, {{wp|Lava}}).&nbsp; If no one else made use of the remainder, then anything I missed would simply remand to a more general link rot project in the far future.
 +
 +
:::: Hope that makes some sense, and thanks as always for your patience.&nbsp;&nbsp;&nbsp; [[User talk:Ryan W|Ryan W]] ''([[User:Ryan W|living fossil]])'' 19:56, 27 March 2018 (CDT)
  
 
Hey, a suggestion: would it be possible to have a third column listing the link types used? More generally, a bot capable of looking through the wiki pages to find any and all instance of direct links to domains that have interwiki entries would be useful for maintenance. --[[User:Gez|Gez]] ([[User talk:Gez|talk]]) 06:25, 27 March 2018 (CDT)
 
Hey, a suggestion: would it be possible to have a third column listing the link types used? More generally, a bot capable of looking through the wiki pages to find any and all instance of direct links to domains that have interwiki entries would be useful for maintenance. --[[User:Gez|Gez]] ([[User talk:Gez|talk]]) 06:25, 27 March 2018 (CDT)
Line 231: Line 247:
 
: Sure, you mean whether a DW page uses wp template, interwiki or https link for a WP link? How should the third column look like, for example? And what if more than one method is used for the same WP link on a page?
 
: Sure, you mean whether a DW page uses wp template, interwiki or https link for a WP link? How should the third column look like, for example? And what if more than one method is used for the same WP link on a page?
 
: The general suggestion would be a separate bot script. --[[User:Xymph|Xymph]] ([[User talk:Xymph|talk]]) 06:42, 27 March 2018 (CDT)
 
: The general suggestion would be a separate bot script. --[[User:Xymph|Xymph]] ([[User talk:Xymph|talk]]) 06:42, 27 March 2018 (CDT)
 +
::I figure the third column could just contain a simple marker, like D for direct link, I for interwiki link, and T and template link. If more than one method is used, then you get more than one marker. If there's a direct link and a template link, then it'd show DT for example. --[[User:Gez|Gez]] ([[User talk:Gez|talk]]) 07:03, 27 March 2018 (CDT)
 +
 +
&larr;&larr;&larr;
 +
 +
First, to clear up a misconception, bot scripts aren't crawling the site's HTML pages. They communicate via the [[mediawikiwiki:API:Main page|MW API]] and thus, when working with page content, they indeed retrieve and store wikitext, just like a human using the edit form. And in the Template namespace, fetching pages therefore gets the template wikitext; nothing transcluded. I wouldn't even ''want'' to process HTML for page content, that would be error-prone and unnecessarily complicated.
 +
 +
Thanks for the feedback, the interwiki pattern was updated to '{{c|<nowiki>\[\[Wiki(?:pedia)?:(.+?)(?:\|.+?)?\]\]</nowiki>}}'. All patterns are matched using the [http://www.php.net/manual/en/reference.pcre.pattern.modifiers.php ''i'' modifier], so case-insensitively. But the WP paths ''found'' are case-sensitive, and stored that way in the data structure. The script doesn't know which one of those is the canonical WP page. To collate WP paths to some degree, the script now converts all underscores to spaces, to deal with Blake_Stone-type situations. But I'm not sure no WP path legitimately includes an underscore, and whether that would result in a broken WP link in the first column of the tables.
 +
 +
Also, paths starting with a letter always use a capital initial in WP's URL, so the first letter is now uppercased to collate [Ff]air use-type situations. But I can't safely collate "Alice in Chains" and "Alice In Chains", so in those situations there are still multiple entries. Further, in File space, pages ending in " map.png" were skipped, saving about half an hour of processing time.
 +
 +
Lastly, the third column is present, showing very few direct links, most of which I fixed already. No DW pages mixed link types for the same WP path, either. After the following log:
 +
 +
Attempting to log in... Success.
 +
collecting Wikipedia namespaces:  34
 +
 +
processing 'main'... 100 200 300 400 500
 +
processing 'main' from main:Counterattack... 100 200 300 400 500
 +
processing 'main' from main:E1M3:_Toxin_Refinery_(Phobos_Revisited)... 100 200 300 400 500
 +
processing 'main' from main:E4M9:_Bean_Oversight_(Doom_the_Way_id_Did:_The_Lost_Episodes)... 100 200 300 400 500
 +
processing 'main' from main:IRC... 100 200 300 400 500
 +
processing 'main' from main:MAP02:_Security_(H2H-Xmas)... 100 200 300 400 500
 +
processing 'main' from main:MAP06:_Colossus_(The_Rebirth)... 100 200 300 400 500
 +
processing 'main' from main:MAP10:_Eureka_Signs_(BTSX-E2)... 100 200 300 400 500
 +
processing 'main' from main:MAP15:_Descent_to_Core_(Speed_of_Doom)... 100 200 300 400 500
 +
processing 'main' from main:MAP21:_Aberinkula_(MAYhem_2048)... 100 200 300 400 500
 +
processing 'main' from main:MAP27:_Emerald_Spire_(Sunlust)... 100 200 300 400 500
 +
processing 'main' from main:MAP34:_Abstraction_(Claustrophobia_1024_2)... 100 200 300 400 500
 +
processing 'main' from main:Psychophobia... 100 200 300 400 500
 +
processing 'main' from main:The_Doom_Movement_Bible... 100 200 300
 +
processed:  6874 pages
 +
 +
processing 'Doom Wiki'...
 +
processed:    43 pages
 +
 +
processing 'File'... 100 200 300
 +
processing 'File' from File:32in24_MAP09_hilltop.png... 100 200 300
 +
processing 'File' from File:AnimFace.gif... 100 200 300
 +
processing 'File' from File:Cchest4_MAP13.png... 100 200 300
 +
processing 'File' from File:CommunityChest-map12-mud.png... 100 200 300 400 500
 +
processing 'File' from File:Congestion1024-map19-end.png... 100 200 300
 +
processing 'File' from File:DTWID-LE_E2M8_map.png... 100 200
 +
processing 'File' from File:Doom3_weapons_07.jpg... 100 200 300
 +
processing 'File' from File:Doom_v0.3_thestore.png... 100 200 300 400
 +
processing 'File' from File:E3M4_map.png... 100 200 300
 +
processing 'File' from File:Eternity-e2m1-s.png... 100 200 300
 +
processing 'File' from File:H2H-Xmas_MAP31_map.png... 100 200
 +
processing 'File' from File:Heretic-e3m4-snail.png... 100 200 300
 +
processing 'File' from File:Interception_MAP04_map.png... 100 200 300
 +
processing 'File' from File:Kansams_Trial_E2M7_map.png... 100 200
 +
processing 'File' from File:MM_MAP02_Secret.png... 100 200 300 400
 +
processing 'File' from File:MementoMori2-map04-down.png... 100 200 300 400
 +
processing 'File' from File:NDCP-map23-torches.png... 100 200
 +
processing 'File' from File:Origwad_2.png... 100 200 300
 +
processing 'File' from File:Psx-command-control-start.png... 100 200 300 400
 +
processing 'File' from File:Requiem-map27-end.png... 100 200 300
 +
processing 'File' from File:Scythe2_MAP03_map.png... 100 200 300
 +
processing 'File' from File:Stardate20X6.png... 100 200
 +
processing 'File' from File:TheEvilUnleashed-e3m5-cage.png... 100 200 300
 +
processing 'File' from File:Vile_Flesh_MAP11_map.png... 100
 +
processed: 12260 pages
 +
skipped:    3465 map pages
 +
 +
processing 'Template'... 100 200 300 400 500
 +
processing 'Template' from Template:MAYhem_1500_21-30... 100 200 300 400
 +
processed:  940 pages
 +
 +
processing 'Help'...
 +
processed:    8 pages
 +
 +
processing 'Category'... 100 200 300 400 500
 +
processing 'Category' from Category:Easter_eggs... 100 200 300 400 500
 +
processing 'Category' from Category:MrGuyTodd_levels... 100 200 300 400
 +
processed:  1491 pages
 +
 +
the new tables for all DW namespaces are online. --[[User:Xymph|Xymph]] ([[User talk:Xymph|talk]]) 12:58, 28 March 2018 (CDT)
 +
 +
: Thank you!&nbsp; I'm slightly out of breath, watching it grow.&nbsp; :>&nbsp;&nbsp; Maybe I spend too much time thinking about the global topology of the wiki, but I sense this could lead in several useful directions, not all to do with modifying links.
 +
 +
: ''They communicate via the [[mediawikiwiki:API:Main page|MW API]]''&nbsp;&nbsp; &mdash;&nbsp;I did know that, and in that very documentation I kept seeing mentions of [[mw:API:Parsing wikitext|"parsed" text]].&nbsp; I hoped there was a feature which could make your task easier by reducing the number of different cases to search for, i.e. [https://doomwiki.org/w/index.php?title=Wikipedia&action=render converting all links into direct links].&nbsp; Of course I accept your word that wikitext is easier; as I said, in hindsight that should be easier for everyone else also.
 +
 +
: ''But I'm not sure no WP path legitimately includes an underscore''&nbsp;&nbsp; &mdash;&nbsp;I'm reasonably sure, based on [[Category talk:D_GARG levels|this experiment]], that equivalence between spaces and underscores is implemented at a fairly low level.&nbsp; I can even type the spaces into my URL bar and still get where I intended: {{c|<nowiki>https://en.wikipedia.org/wiki/Death of Michael Jackson</nowiki>}}&nbsp;&nbsp; Does that work in general?&nbsp; Nope: {{c|<nowiki>https://www.youtube.com/watch?v=VxqaI c9j g</nowiki>}}
 +
 +
: ''But I can't safely collate "Alice in Chains" and "Alice In Chains"''&nbsp;&nbsp; &mdash;&nbsp;True enough.&nbsp; So far none of these points is enormously important&nbsp; :D&nbsp; however:
 +
 +
: ''Thanks for the feedback''&nbsp;&nbsp; &mdash;&nbsp;It's the least I can do.&nbsp; Something is still wrong, because the table omits the link that started my train of thought: [[E1M4: Command Control (Doom)#Sources|this one]].
 +
 +
: [[User talk:Ryan W|Ryan W]] ''([[User:Ryan W|living fossil]])'' 18:12, 28 March 2018 (CDT)
 +
 +
:: Ow, that's because [[Template:Wikipedia]] wasn't mentioned to far, and I didn't realize it either. It's now covered by regex '{{c|<nowiki>{{wikipedia(?:\|(?:title=)?(.+?)(?:\|.+?)?)?}}</nowiki>}}' and type S (Source). Besides with a positional or named parameter, it can be used parameterless which presented a small problem, but that's now solved. See the updated list. --[[User:Xymph|Xymph]] ([[User talk:Xymph|talk]]) 10:58, 29 March 2018 (CDT)
 +
 +
::: D'oh!&nbsp; I saw "exception list" above, and somehow assumed it was part of the regex even though I couldn't see it.&nbsp; (I did page through templates at some point to check if there were any more.&nbsp; There are not.)
 +
 +
::: I think that's all of my questions, so we'll see what others have to say.&nbsp; :>&nbsp;&nbsp;&nbsp; [[User talk:Ryan W|Ryan W]] ''([[User:Ryan W|living fossil]])'' 16:36, 29 March 2018 (CDT)
 +
 +
== Interwiki links ==
 +
''Copied from Wikipedia topic'':<br/>
 +
More generally, a bot capable of looking through the wiki pages to find any and all instance of direct links to domains that have interwiki entries would be useful for maintenance. --[[User:Gez|Gez]] ([[User talk:Gez|talk]]) 06:25, 27 March 2018 (CDT)
 +
 +
: To verify, this script needs to search pages for the URLs in the [[Special:Interwiki|Interwiki table]] (up to the $1 parameter), and show which interwiki prefix should be used instead? --[[User:Xymph|Xymph]] ([[User talk:Xymph|talk]]) 12:26, 27 March 2018 (CDT)
 +
::Yeah, I figure that should be enough. The only potential subtlety would be to watch out for http/https, i.e. there could be an http direct link to a site that has https in the interwiki table (or vice-versa). --[[User:Gez|Gez]] ([[User talk:Gez|talk]]) 13:42, 27 March 2018 (CDT)
 +
 +
: If we're talking about maintenance, the reverse of this (an analogue of [[Special:LinkSearch]] for interwikis only) would occasionally be useful as well, e.g. when a site goes down permanently.&nbsp; Again a low priority, unless someone finds a live example.&nbsp;&nbsp;&nbsp; [[User talk:Ryan W|Ryan W]] ''([[User:Ryan W|living fossil]])'' 20:00, 27 March 2018 (CDT)
 +
 +
:: Not sure what info a "reverse" script (or addition to the current script) should generate &ndash; if you'd like something done, please elaborate.
 +
:: The current script was pretty easy to derive from the previous one, and produces the [[User:Xymph/List of Interwiki links|latest table]] after this log:
 +
 +
Attempting to log in... Success.
 +
collecting unique Interwiki list: 135
 +
 +
processing 'main'... 100 200 300 400 500
 +
processing 'main' from main:Counterattack... 100 200 300 400 500
 +
processing 'main' from main:E1M3:_Toxin_Refinery_(Phobos_Revisited)... 100 200 300 400 500
 +
processing 'main' from main:E4M9:_Bean_Oversight_(Doom_the_Way_id_Did:_The_Lost_Episodes)... 100 200 300 400 500
 +
processing 'main' from main:IRC... 100 200 300 400 500
 +
processing 'main' from main:MAP02:_Security_(H2H-Xmas)... 100 200 300 400 500
 +
processing 'main' from main:MAP06:_Colossus_(The_Rebirth)... 100 200 300 400 500
 +
processing 'main' from main:MAP10:_Eureka_Signs_(BTSX-E2)... 100 200 300 400 500
 +
processing 'main' from main:MAP15:_Descent_to_Core_(Speed_of_Doom)... 100 200 300 400 500
 +
processing 'main' from main:MAP21:_Aberinkula_(MAYhem_2048)... 100 200 300 400 500
 +
processing 'main' from main:MAP27:_Emerald_Spire_(Sunlust)... 100 200 300 400 500
 +
processing 'main' from main:MAP34:_Abstraction_(Claustrophobia_1024_2)... 100 200 300 400 500
 +
processing 'main' from main:Psychophobia... 100 200 300 400 500
 +
processing 'main' from main:The_Doom_Movement_Bible... 100 200 300
 +
processed:  6874 pages
 +
 +
processing 'Doom Wiki'...
 +
processed:    43 pages
 +
 +
processing 'File'... 100 200 300 400 500
 +
processing 'File' from File:32in24_MAP09_hilltop.png... 100 200 300 400 500
 +
processing 'File' from File:AnimFace.gif... 100 200 300 400 500
 +
processing 'File' from File:Cchest4_MAP13.png... 100 200 300 400 500
 +
processing 'File' from File:CommunityChest-map12-mud.png... 100 200 300 400 500
 +
processing 'File' from File:Congestion1024-map19-end.png... 100 200 300 400 500
 +
processing 'File' from File:DTWID-LE_E2M8_map.png... 100 200 300 400 500
 +
processing 'File' from File:Doom3_weapons_07.jpg... 100 200 300 400 500
 +
processing 'File' from File:Doom_v0.3_thestore.png... 100 200 300 400 500
 +
processing 'File' from File:E3M4_map.png... 100 200 300 400 500
 +
processing 'File' from File:Eternity-e2m1-s.png... 100 200 300 400 500
 +
processing 'File' from File:H2H-Xmas_MAP31_map.png... 100 200 300 400 500
 +
processing 'File' from File:Heretic-e3m4-secret.png... 100 200 300 400 500
 +
processing 'File' from File:Interception_MAP03_map.png... 100 200 300 400 500
 +
processing 'File' from File:Kansams_Trial_E2M6_map.png... 100 200 300 400 500
 +
processing 'File' from File:MM_MAP01_map.png... 100 200 300 400 500
 +
processing 'File' from File:MementoMori2-map03-plus.png... 100 200 300 400 500
 +
processing 'File' from File:NDCP-map19-water.png... 100 200 300 400 500
 +
processing 'File' from File:Odessa_3.png... 100 200 300 400 500
 +
processing 'File' from File:Psx-central-processing-02.png... 100 200 300 400 500
 +
processing 'File' from File:Requiem-map22-cacos.png... 100 200 300 400 500
 +
processing 'File' from File:Scythe2-map28-afrit.png... 100 200 300 400 500
 +
processing 'File' from File:Speed_of_Doom_MAP24_map.png... 100 200 300 400 500
 +
processing 'File' from File:TheEvilUnleashed-e3m1-chasm.png... 100 200 300 400 500
 +
processing 'File' from File:Vanguard_MAP06_map.png... 100 200
 +
processed: 12292 pages
 +
 +
processing 'Template'... 100 200 300 400 500
 +
processing 'Template' from Template:MAYhem_1500_21-30... 100 200 300 400
 +
processed:  940 pages
 +
 +
processing 'Help'...
 +
processed:    8 pages
 +
 +
processing 'Category'... 100 200 300 400 500
 +
processing 'Category' from Category:Easter_eggs... 100 200 300 400 500
 +
processing 'Category' from Category:MrGuyTodd_levels... 100 200 300 400
 +
processed:  1491 pages
 +
 +
:: Some info about what listInterLinks.php does:
 +
::* init namespace lists and patterns; DW paths in the exception list are the previous five, plus '{{c|Doom Wiki:Departure from Wikia}}', '{{c|Doom Wiki:Source material}}' and '{{c|Help:Special pages}}'
 +
::* the interwiki list is [https://doomwiki.org/w/api.php?action=query&meta=siteinfo&siprop=interwikimap fetched] and the URLs are converted to regexes, e.g. the first one becomes "{{c|<nowiki>https?://3dfxdev\.net/edgewiki/index\.php\?title=[\w#%/:.=-]+</nowiki>}}"; also, some interwikis/URLs occur in the list twice, so it is deduped from 157 to 135
 +
::* pages are fetched from namespaces analog to listWikilinks and matches for each regex are stored in an array (keyed by interwiki prefix) of arrays (keyed by DW path)
 +
::* the arrays are then sorted by those keys &ndash; but the direct links remain stored in the order they were encountered in that page &ndash; and the table is generated
 +
:: On the first full pass, file pages ending in " map.png" weren't skipped yet but that's enabled now to save time in future runs. If more/different features are needed, let me know. For one thing, the {{c|<nowiki>$1</nowiki>}} regex was a bit limited, I've already expanded it to "{{c|<nowiki>[\w#@%&/?!:;,.~=+-]+</nowiki>}}". --[[User:Xymph|Xymph]] ([[User talk:Xymph|talk]]) 04:50, 3 April 2018 (CDT)
 +
 +
== Templatable links ==
 +
''Moved from Wikipedia topic'':<br/>
 +
And while I'm on that general topic, a third script could search for direct links that could be templated -- I'm thinking about stuff like direct links to forums or archives, [[:Category:External link templates|notably]]. There's no hurry, though. --[[User:Gez|Gez]] ([[User talk:Gez|talk]]) 07:03, 27 March 2018 (CDT)
 +
 +
: Okay, one thing are a time. :) --[[User:Xymph|Xymph]] ([[User talk:Xymph|talk]]) 12:26, 27 March 2018 (CDT)
 +
 +
: This one should be uncontroversial, but "no hurry" is an understatement.&nbsp; :>&nbsp;&nbsp; IMO a bot performing style edits would have to sweep at regular intervals, and I don't assume Xymph or anyone else can commit to that (wishful thinking solution: someone in the Doom community figures out how to host bots as autonomous processes, like the WMF does).&nbsp;&nbsp;&nbsp; [[User talk:Ryan W|Ryan W]] ''([[User:Ryan W|living fossil]])'' 17:04, 27 March 2018 (CDT)
 +
 +
:: Actually, running scripts regularly and posting their output to list pages automatically would be trivial for me, and less so for anyone else. ;-) But multiple admins/editors are monitoring Recent Changes already and when incorrect styles and links are fixed right away, regular runs won't even be necessary. An incidental run every 4-6 months or so would be enough to catch anything falling through those cracks.
 +
 +
:: The new script was really easy to derive from the previous one, and produces the [[User:Xymph/List of templatable links|latest table]] after this log:
 +
 +
Attempting to log in... Success.
 +
collecting external templates list: 20
 +
 +
processing 'main'... 100 200 300 400 500
 +
processing 'main' from main:Cory_Scott_(NiTROACTiVE)... 100 200 300 400 500
 +
processing 'main' from main:E1M3:_The_Gatehouse_(Heretic)... 100 200 300 400 500
 +
processing 'main' from main:E4M5:_The_Witching_Hour_(Crusades)... 100 200 300 400 500
 +
processing 'main' from main:Hub_4:_Desolate_Garden... 100 200 300 400 500
 +
processing 'main' from main:MAP02:_Nullth_Precinct_(Community_Chest)... 100 200 300 400 500
 +
processing 'main' from main:MAP05:_Wasted!_(NDCP2)... 100 200 300 400 500
 +
processing 'main' from main:MAP09_(2_Sectors)... 100 200 300 400 500
 +
processing 'main' from main:MAP14:_Vile_Gear_(Swift_Death)... 100 200 300 400 500
 +
processing 'main' from main:MAP20:_Speedtraps_for_the_Bee_Kingdom_(BTSX-E2)... 100 200 300 400 500
 +
processing 'main' from main:MAP26:_Teleport_Control_(Daedalus:_Alien_Defense)... 100 200 300 400 500
 +
processing 'main' from main:MAP32:_Ultimative_Geheimnis_(Whispers_of_Satan)... 100 200 300 400 500
 +
processing 'main' from main:Porkalator... 100 200 300 400 500
 +
processing 'main' from main:Terminatrix... 100 200 300 400
 +
processed:  6916 pages
 +
 +
processing 'Doom Wiki'...
 +
processed:    43 pages
 +
 +
processing 'File'... 100 200 300 400 500
 +
processing 'File' from File:32in24_MAP09_hilltop.png... 100 200 300 400 500
 +
processing 'File' from File:AnimFace.gif... 100 200 300 400 500
 +
processing 'File' from File:Cchest4_MAP12_Deathmatch.png... 100 200 300 400 500
 +
processing 'File' from File:CommunityChest-map12-end.png... 100 200 300 400 500
 +
processing 'File' from File:Congestion1024-map17-switches.png... 100 200 300 400 500
 +
processing 'File' from File:DTWID-LE_E2M1_map.png... 100 200 300 400 500
 +
processing 'File' from File:Doom3_backpack.png... 100 200 300 400 500
 +
processing 'File' from File:Doom_pinky_2.jpg... 100 200 300 400 500
 +
processing 'File' from File:E3M1_heretic.png... 100 200 300 400 500
 +
processing 'File' from File:Eternally_Yours_MAP12_map.png... 100 200 300 400 500
 +
processing 'File' from File:H2H-Xmas_MAP24_map.png... 100 200 300 400 500
 +
processing 'File' from File:Heretic-e3m1-storage.png... 100 200 300 400 500
 +
processing 'File' from File:Infinity_E1M8_Map.png... 100 200 300 400 500
 +
processing 'File' from File:KansamsTrial-E2M7-start.png... 100 200 300 400 500
 +
processing 'File' from File:MM8BDM_Gameplay_1.png... 100 200 300 400 500
 +
processing 'File' from File:MementoMori2-map01-start.png... 100 200 300 400 500
 +
processing 'File' from File:NDCP-map17-hex.png... 100 200 300 400 500
 +
processing 'File' from File:Obituary_MAP04_map.png... 100 200 300 400 500
 +
processing 'File' from File:Programador1_Strife.png... 100 200 300 400 500
 +
processing 'File' from File:Requiem-map16-elemental.png... 100 200 300 400 500
 +
processing 'File' from File:Scythe2-map13-center.png... 100 200 300 400 500
 +
processing 'File' from File:SpeedOfDoom-map28-redskullkey.png... 100 200 300 400 500
 +
processing 'File' from File:Talosian_Incident_MAP15_map.png... 100 200 300 400 500
 +
processing 'File' from File:Vae_Victus_2_MAP02_map.png... 100 200 300
 +
processed: 12341 pages
 +
 +
processing 'Help'...
 +
processed:    8 pages
 +
 +
processing 'Category'... 100 200 300 400 500
 +
processing 'Category' from Category:Earth_(WAD)_levels... 100 200 300 400 500
 +
processing 'Category' from Category:MrGlide_levels... 100 200 300 400
 +
processed:  1493 pages
 +
 +
:: Some info about what listExternLinks.php does:
 +
::* init namespace lists and patterns; pages in the exception list are just '{{c|Doom Wiki:Central Processing.*}}' and '{{c|Help:Spam/Archive}}'
 +
::* from [[:Category:External link templates|External link templates]] the following were manually added to the patterns list (the others don't contain direct links):
 +
 +
$externlist['Competnftp'] = 'https?://(?:www\.)?doom\.com\.hr/temp/compet-n/';
 +
$externlist['Competnmap'] = 'https?://(?:www\.)?doom\.com\.hr/index.php\?page=compet-n_database&cndb=';
 +
$externlist['Competnwad'] = 'https?://(?:www\.)?doom\.com\.hr/index.php\?page=compet-n_database&wad_id=';
 +
$externlist['Deb-pkg']    = 'https?://packages\.debian\.org/';
 +
$externlist['Drdforums']  = 'https?://forum\.drdteam\.org/viewtopic.php\?';
 +
$externlist['Dsda']      = 'https?://doomedsda\.us/wad';
 +
$externlist['Dsdaftp']    = 'https?://doomedsda\.us/lmps/';
 +
$externlist['Dwforums']  = 'https?://(?:www\.)?doomworld\.com/vb/thread/';
 +
$externlist['Dwforumst']  = 'https?://(?:www\.)?doomworld\.com/vb/post/';
 +
$externlist['Edgeeg']    = 'https?://edge\.sourceforge\.net/edit_guide/';
 +
$externlist['Idftp']      = 'https?://mirrors\.syringanetworks\.net/idgames/';
 +
$externlist['Idgames']    = 'https?://(?:www\.)?doomworld\.com/idgames/';
 +
$externlist['Idgamesmm']  = '<nowiki>ftp://ftp\.fu-berlin\.de/pc/msdos/games/idgames/</nowiki>';
 +
$externlist['Lc']        = 'https?://lccn\.loc\.gov/';
 +
$externlist['Legacywiki'] = 'https?://legacywiki\.net/index\.php/';
 +
$externlist['Realm667']  = 'https?://(?:www\.)?realm667\.com/index\.php\?option=com_content&view=';
 +
$externlist['Vvwiki']    = 'https?://(?:www\.)?vavoom-engine\.com/wiki/index.php\?title=';
 +
$externlist['Wip']        = 'https?://(?:www\.)?wadsinprogress\.info/\?a=listwads&wad=';
 +
$externlist['Zanforums']  = 'https?://(?:www\.)?zandronum\.com/forum/showthread\.php\?tid=';
 +
$externlist['Zdforums']  = 'https?://forum\.zdoom\.org/viewtopic\.php\?';
 +
 +
::* pages are fetched from namespaces analog to listInterlinks and matches for each regex are stored in an array (keyed by external template) of arrays (keyed by DW path); sorting and outputting remained the same
 +
 +
:: On the initial pass, templates being matched by their own template's link were skipped, but because some direct links occur in multiple templates a handful of them were still included as false positives. So for the full run over all namespaces, Template space was skipped.
 +
:: For future runs, '{{c|Idgames archive}}' is added to the exception list as direct links do belong on that page. As usual, file pages ending in " map.png" will also be skipped.
 +
:: If I missed any external link templates in the patterns list, please let me know. --[[User:Xymph|Xymph]] ([[User talk:Xymph|talk]]) 14:29, 12 April 2018 (CDT)
 +
 +
== Level has serious issues with popular mod loaded - put in "bugs", "trivia" or not at all? ==
 +
 +
Hey Xymph, I'm currently working on the walkthrough for {{maplinkgen|MAP17|Metastasize|Community Chest 4}}. While playing through the level, I found that mods where boxes of rockets could be shot and blown up (e.g. Brutal Doom v21 beta) caused serious problems in this level, as one would telefrag the box (as part of the level progression) and die in the explosion. Hence, I'm planning to add the following line in into the page:
 +
 +
"This level should not be played with mods that allow [[box of rockets|boxes of rockets]] to be shot and blown up as if they were [[barrel]]s (or where possible, have the setting turned off in the mod), as in the course of getting to each of the keys, the player will teleport onto a spot with a box of rockets already there. With such mods loaded, the player will [[telefrag]] the box of rockets and be hit by a point-blank explosion, which will likely kill them."
 +
 +
The question is - should this line go into "bugs", "trivia", or should it not be on the page at all? --[[User:15FiftySeven|15FiftySeven]] ([[User talk:15FiftySeven|talk]]) 09:51, 7 May 2018 (CDT)
 +
:Personally I wouldn't put it there. I would mention the issue instead on the [[Brutal Doom]] article. --[[User:Gez|Gez]] ([[User talk:Gez|talk]]) 12:03, 7 May 2018 (CDT)
 +
 +
: The PWAD was designed for Boom-compatible ports, so when an unrelated mod has an unexpected gameplay effect, that's certainly not a map bug, and possibly not even something that needs to be mentioned on the PWAD page or the specific map page. Gez's suggestion works for me too. --[[User:Xymph|Xymph]] ([[User talk:Xymph|talk]]) 12:56, 7 May 2018 (CDT)
 +
 +
==Shedding data from map articles?==
 +
Please double-check the recent changes:
 +
* https://doomwiki.org/w/index.php?title=MAP04:_Water_control_(Doomworld_Mega_Project_2013)&curid=33083&diff=167921&oldid=167880
 +
* https://doomwiki.org/w/index.php?title=MAP11:_ffffabolous_secret_level_(Doomworld_Mega_Project_2013)&curid=33089&diff=167933&oldid=167887
 +
* https://doomwiki.org/w/index.php?title=MAP15:_NUKAGE_PROCESSING_(Doomworld_Mega_Project_2013)&curid=33093&diff=167941&oldid=167891
 +
* https://doomwiki.org/w/index.php?title=MAP18:_SUBBSE_(Doomworld_Mega_Project_2013)&curid=33096&diff=167944&oldid=167894
 +
* https://doomwiki.org/w/index.php?title=MAP19:_Trial_And_Error_(Doomworld_Mega_Project_2013)&diff=prev&oldid=167945
 +
* https://doomwiki.org/w/index.php?title=MAP02:_Twilight_of_the_Gods_(Doomworld_Mega_Project_2013)&diff=next&oldid=167878
 +
These are all tagged with the revision text "use secrets & map data from textmapStats.php" and have atypically large negative deltas. --[[User:Quasar|Quasar]] ([[User talk:Quasar|talk]]) 17:22, 18 June 2018 (CDT)
 +
 +
: I expected that would trigger someone's curiosity. :) As you may remember, [[DMMPST]] does not support [[UDMF]]. [[Doomworld_Mega_Project_2013#Levels|DMP2013Z.wad]] mixes seven UDMF maps with traditional ones, and [[User:XymphBot#dmmpstBot.php|dmmpstBot.php]] is the easiest way to generate a series of new pages. So in loading the pwad on top of the iwad (doom2.wad) as DMMPST requires, and requesting e.g. MAP02 statistics, DMMPST actually returns the iwad's data because it ignores the pwad UDMF data. Compare [https://doomwiki.org/w/index.php?title=MAP02:_Twilight_of_the_Gods_(Doomworld_Mega_Project_2013)&oldid=167676#Statistics MAP02:_Twilight_of_the_Gods_(Doomworld_Mega_Project_2013)#Statistics] with [[MAP02:_Underhalls_(Doom_II)#Statistics]]. I could have manually created the six skeleton pages without the things data, but that would be a bit more cumbersome than just letting dmmpstBot.php rip and fixing the secret sectors and map data afterwards, and shedding the things tables.
 +
: Note that the page for the seventh map article already existed with a prior things table, and I left that one alone even though it's not styled entirely like DMMPST would do it. I assume Joe-ilya entered that correctly [https://doomwiki.org/w/index.php?title=MAP09:_Language_Barrier_(Doomworld_Mega_Project_2013)&oldid=126676#Things at the time].
 +
: I suppose the proper way out of this conundrum would be to make DMMPST handle UDMF, but that's pretty tough and &ndash; given the relatively low number of UDMF maps &ndash; may not be worth the effort. --[[User:Xymph|Xymph]] ([[User talk:Xymph|talk]]) 03:12, 19 June 2018 (CDT)
 +
 +
:: OK thanks for the explanation. I just wanted to be sure something accidental didn't happen :) --[[User:Quasar|Quasar]] ([[User talk:Quasar|talk]]) 03:38, 19 June 2018 (CDT)
 +
 +
== Knee deep in KDiZD and Chocorenderlimits ==
 +
 +
I saw that you just fixed "Knee deep in KDiZD" into KDiZD in the Chocorenderlimits page. There is a project named [https://www.doomworld.com/forum/topic/42197-knee-deep-in-kdizd-whoring-out-screenshots-and-looking-for-mappers Knee deep in KDiZD] and [https://www.doomworld.com/forum/topic/42197-knee-deep-in-kdizd-whoring-out-screenshots-and-looking-for-mappers/?page=11&tab=comments#comment-726841 it seems more likely to be the one that first used Chocorenderlimts], given that it is a vanila project and all, while KDiZD is a ZDoom project. So, were you mistaking? Or did i miss something? --[[User:BuildCat|BuildCat]] ([[User talk:BuildCat|talk]]) 12:09, 27 June 2018 (CDT)
 +
: I ran [https://www.doomworld.com/search/?q=%22Knee%20Deep%20In%20Knee%20Deep%20In%20ZDoom%22&sortby=relevancy some] [https://www.doomworld.com/search/?q=%22Knee%20Deep%20In%20KDIZD%22&sortby=relevancy searches] which came up empty, hence I concluded the red link was an oversight -- but with a nagging thought in the back of my mind that a forum regular might correct me. Thanks. ;) --[[User:Xymph|Xymph]] ([[User talk:Xymph|talk]]) 12:21, 27 June 2018 (CDT)
 +
 +
==PSX maps==
 +
When you're around in IRC again, I'd like to send you the PSX maps, which I've converted to a vanilla-compatible format, for you to make map graphics for and run stats against - we can discuss adding support to DMMPST for thing tables etc. if you want as well - there are some special considerations with regards to how spectres work that would need handling, unless I change the extractor to modify the thing types of special spectres. --[[User:Quasar|Quasar]] ([[User talk:Quasar|talk]]) 14:18, 21 July 2018 (CDT)
 +
 +
: In passing, I'll note that SLADE 3 can generate map images from PSX Doom and Final Doom maps, but you first need to select the VERTEXES lump and type the console command "vertexpsx". --[[User:Gez|Gez]] ([[User talk:Gez|talk]]) 16:27, 21 July 2018 (CDT)
 +
:: Cool; I wasn't sure that ever got implemented. --[[User:Quasar|Quasar]] ([[User talk:Quasar|talk]]) 16:53, 21 July 2018 (CDT)
 +
 +
: If you could update your extractor to change the thing type of regular spectres back to 58 (dec) and of nightmare spectres to some unused type (3007?), then it would require very little additional support in DMMPST to generate correct thing tables for them. --[[User:Xymph|Xymph]] ([[User talk:Xymph|talk]]) 06:20, 22 July 2018 (CDT)
 +
 +
Looking at the new Club Doom map, I think that edit is best reverted. Map images typically prune the meaningless sectors (holding cells, control sectors, etc.) to focus on the actual map part, and the presence of a "signature area" in Club Doom far away from the playable area makes the map thumbnail totally unreadable. --[[User:Gez|Gez]] ([[User talk:Gez|talk]]) 08:16, 27 July 2018 (CDT)
 +
: Not reverted but remedied the usual way. I included the area because it's described in Trivia, but this is fine too. --[[User:Xymph|Xymph]] ([[User talk:Xymph|talk]]) 09:23, 27 July 2018 (CDT)
 +
 +
== Pre-1.9 map uploads ==
 +
 +
Hi Xymph.&nbsp; Wow, the bot's lump parsing improves by the day, it seems.&nbsp; :D
 +
 +
Please see e.g. [[:File:Doom v1.2 E3M9 map.png#File usage|here]]: "The following file is a duplicate of this file"&nbsp; &mdash;&nbsp;that means a pixel-for-pixel duplicate, so no fair use justification will ever be possible.&nbsp; Sorry.&nbsp;&nbsp;&nbsp; [[User talk:Ryan W|Ryan W]] ''([[User:Ryan W|living fossil]])'' 04:10, 28 July 2018 (CDT)
 +
 +
: Sorry but I have no idea what "bot's lump parsing" refers to.
 +
: The pre-v1.9 map views are planned to be used in console map comparisons, because the console levels were based on those older versions. Quasar and I discussed this on IRC yesterday, and I simply uploaded the two whole batches for convenience and naming consistency. Then the comparisons can always use v1.2/v1.666 filenames, and I may write a bot script to do that (including creation of the transcluded gallery, if we get the tabs template working), so naming consistency would help there. If duplicate files are problematic, would it be possible to replace them with redirects to their v1.9 counterparts? --[[User:Xymph|Xymph]] ([[User talk:Xymph|talk]]) 05:08, 28 July 2018 (CDT)
 +
 +
:: Based on [[File talk:Screenshot Doom 20180603 143620.png|this]] testing, I would say yes, replacing with redirects should work, provided nothing is written above the redirect command.&nbsp;&nbsp;&nbsp; [[User talk:Ryan W|Ryan W]] ''([[User:Ryan W|living fossil]])'' 06:53, 28 July 2018 (CDT)
 +
 +
==Ask and ye shall receive==
 +
https://www.doomworld.com/forum/topic/101943-doom-wiki-month-in-review-issue-4-late-edition/ --[[User:Quasar|Quasar]] ([[User talk:Quasar|talk]]) 08:41, 14 August 2018 (CDT)
 +
: Heh, nice. --[[User:Xymph|Xymph]] ([[User talk:Xymph|talk]]) 11:10, 14 August 2018 (CDT)
 +
 +
== Template:Deleted ==
 +
Oh wow.  If I hadn’t seen it in print, I wouldn’t have believed it.  Thank you!  Ironically that happens *less* often in larger batches, because there isn’t all that thinking in between to distract me (but IMO each really has to be judged case by case as I said on DWF).  Glad someone was paying attention.  [[User:MuffinProvensen|MuffinProvensen]] ([[User talk:MuffinProvensen|talk]]) 12:47, 20 August 2018 (CDT)
 +
: I was just about to move this entry into chronological order myself, so... thanks. ;) Also, I had some difficulty understanding what this is about, coming from a little used account -- until I found you're RyanW. :) Still, is this about my adding the deleted template around those two Template:Code instances, or the further series I continued in your footsteps? What does "it" mean? Also, I don't follow DWF apart from a few specific topics, so what does that pointer refer to?
 +
: Really, I appreciate the enthusiastic response, but some of it is lacking sufficient context to follow. --[[User:Xymph|Xymph]] ([[User talk:Xymph|talk]]) 13:02, 20 August 2018 (CDT)
 +
:: Yes, you are all good.&nbsp; "Thank you" referred to both your corrections of me and the additional edits (though I am working from a disorganized offline list and probably would have found the latter eventually).&nbsp; Sorry for the confusion; I try not to put any credentials into my phone that I really care about.&nbsp; :D&nbsp;&nbsp; [[User talk:Ryan W|Ryan W]] ''([[User:Ryan W|living fossil]])'' 17:07, 20 August 2018 (CDT)
 +
 +
== Mapper articles ==
 +
 +
So, we are creating articles for every mapper mentioned in Doom wiki through script now? It does lighten the load for people adding more info thanks to existing skeletons so thanks :) [[User:BuildCat|BuildCat]] ([[User talk:BuildCat|talk]]) 6:13, 11 September 2018 (CDT)
 +
: Not every perhaps, but yes, [[Doom Wiki:Central Processing#Script-generated mapper stubs|see here]]. --[[User:Xymph|Xymph]] ([[User talk:Xymph|talk]]) 06:19, 11 September 2018 (CDT)
 +
 +
==Crazy idea==
 +
Why don't we make some of the marked maps a bit larger on the page when and where it won't disrupt article layout? I don't mean like 2x as big or anything so dramatic, but maybe just large enough that the markers don't require a magnifying glass to check out ;) Usual disclaimer about my poor eyesight applies, of course. --[[User:Quasar|Quasar]] ([[User talk:Quasar|talk]]) 11:58, 8 October 2018 (CDT)
 +
 +
: That's fine for large maps, though in your example case (and with my eyesight) still not enough to distinguish spawn 18 from spawn 19. ;-) Feel free to apply it to the other two SVE maps as well. But I think it won't avoid the need to open the spawn map in another tab to really distinguish all points.
 +
: For [[Id_Map01#Analysis|more]] [[Hank's_Lair#Analysis|typically]] sized DM maps, the standard scaling should be adequate to discern the spawn numbers, or are you proposing to increase their scaling too? --[[User:Xymph|Xymph]] ([[User talk:Xymph|talk]]) 12:57, 8 October 2018 (CDT)
 +
 +
:: True regarding the numbers, but, for me at least, I can more easily tell the blue team spawns from the green DM spots with the size increase. The other examples you give look good already to me. --[[User:Quasar|Quasar]] ([[User talk:Quasar|talk]]) 14:18, 8 October 2018 (CDT)
 +
 +
== "PWAD top-5" statistics ==
 +
 +
Hahaha, Fredrik would be proud [https://doomwiki.org/w/index.php?title=Doom_Wiki:Policies_and_guidelines&oldid=6263#Things_that_may_have_articles].&nbsp; This is exactly the sort of enhancement which only a bot can make &mdash; even if we imagined that we could have a strict rule about double-checking every new map article, people would forget to do so.
 +
 +
One small suggestion: include a date stamp, like the speedrun tables have?&nbsp; That would protect against any future change which created lag between updates (e.g. MediaWiki upgrade disabling Wikimate indefinitely, or more optimistically, an increase in active editors who insist on documenting each megawad on day zero).&nbsp; Hope that makes sense, thanks.&nbsp;&nbsp;&nbsp; [[User talk:Ryan W|Ryan W]] ''([[User:Ryan W|living fossil]])'' 09:05, 21 December 2018 (CST)
 +
 +
: Sure, done. Actually, it's not even a bot script, just a Bash script that greps/sorts through results from earlier DMMPST runs, bot-wise and manually. :) --[[User:Xymph|Xymph]] ([[User talk:Xymph|talk]]) 11:06, 21 December 2018 (CST)
 +
 +
: For your Fredrik reference, I assumed you specifically meant [[Special:Diff/20/37|this change]], and interpreted vanilla maps as stock levels (doing stats for all vanilla maps covered on the wiki seems rather pointless, and then, why not for maps requiring a source port?). So this is now [[List of stock level statistics‎|completed]]. :) --[[User:Xymph|Xymph]] ([[User talk:Xymph|talk]]) 07:09, 26 December 2018 (CST)

Revision as of 08:09, 26 December 2018


Archived discussions

External link editing

Hi Xymph.  I'm not sure about deleting items from Sources sections [1].  That probably means it was the source of some content in that article specifically (e.g. the readme, or some trivia in the DW reviews).    Ryan W (usually gone) 13:31, 24 February 2017 (CST)

Well, you should know, as you created the initial revision which already contained the link. :) Upon further reading, it is related to the "Inspiration and development" mention about "Nathrath's Fab Four". That just wasn't clear, as other Sources entries often have more info like this or use a reference to link it to the place where it's used. With neither, I got the impression the idgames link was used just like here, and that seemed redundant as indicated by my removal reason.
I'll fix them up, including the mix-ups where M5 (in the other collection) didn't get a sources link – nor did M2 – while M6 (not in it) did. ;) --Xymph (talk) 15:00, 24 February 2017 (CST)
Thank you.  FTR I was prepared to hear reasoning for not listing certain sources, which has happened in the past.  And I should know, but I don't, because it was such a disorganized period that I'm thankful to have remembered the difference between square and curly brackets each time.  :>
The general point about footnotes is a good one (but a completely separate project, I think!).  Long story short, the situation you describe does confuse people, but it is a huge and probably non-bottable task to fix it.  I personally would be OK with uniform use of inline references if everyone had put some thought into the question and decided they were good.  I don't believe that is currently the case.    Ryan W (usually gone) 15:28, 24 February 2017 (CST)

Fixes

Thank you for all your fixes. Ducon (talk) 07:38, 21 May 2017 (CDT)

No problem, we all build on previous contributions. If you're going to do more map series, I can help creating the map skeletons bot-wise. It improves consistency and leaves you more time to write information and descriptions, and add screenshots (which are still needed for a gazillion map pages). --Xymph (talk) 07:44, 21 May 2017 (CDT)
I’m taking screenshots for the top 100 WADs and for the cacowarded maps, thus, I create pages for such maps. Ducon (talk) 07:49, 21 May 2017 (CDT)
I'm working on the same, so we can collaborate, like we did implicitly yesterday :) and previously on IC2005. After you create an overview page like Vae Victus 2, you could leave it to me to create the individual map pages with all the basic info that the bot scripts can add. Then you resume with the screens and other info. Of course I'm not always available to do that immediately, so it takes perhaps a little planning ahead (and occasionally patience) for this to work for you too. --Xymph (talk) 08:12, 21 May 2017 (CDT)
Great. I’m going to play Classic episode 2 (https://www.doomworld.com/idgames/levels/doom/Ports/a-c/class_e2). Ducon (talk) 13:59, 21 May 2017 (CDT)
Fine, so create the WAD overview page with sufficient info, then I'll follow with the map pages. And for future WADs you gonna play, creating the overview page will again be my cue – no further posts here necessary. Have fun! --Xymph (talk) 14:25, 21 May 2017 (CDT)

Woodexial's Secrets

I'm using GZDoom's automap instead of an editor and it shows only 7 secrets instead of 8. Was Sector 188 is the one with a Ccacodemon and a backpack in a pit? --ZeroTheEro (talk) 19:20, 22 May 2017 (CDT)

Yes, 188 is the sector just before that pit. I did only a fly-thru, clipping on/off as needed to get to places and trigger the secrets, but the automap in ZDoom does show x/8 secrets too. Not sure how to make it reveal the sector number, and why you got most of them differently than they are stored in the map. --Xymph (talk) 02:31, 23 May 2017 (CDT)
To put it simply, the sector numbers didn't match the secret because I have no idea how to check it. I'm just having guesswork on the sector number since I list the secrets by gradual progression. Please correct me if I am wrong. --ZeroTheEro (talk) 04:28, 23 May 2017 (CDT)
I see. But in skeleton pages the secret sectors are simply in numeric order rather than play progression. I'd rather not be correcting them all the time though. ;) To match the numbers to sectors in the map, I use DeePsea: switch to sector mode with the mouse scroll-wheel, use the 'j' key to bring up the jump dialog, and enter the number to be taken to that highlighted sector. There may be other/better ways in other editors. --Xymph (talk) 04:44, 23 May 2017 (CDT)
Thanks, yo. --ZeroTheEro (talk) 13:23, 23 May 2017 (CDT)
Hi Xymph.  Not to open another can of worms here  ;>  but you mean that bot-generated skeletons use numeric order.  IME human editors adding descriptions tend to reorder them along the base walkthrough route, in hopes that's clearer for the newbies.    Ryan W (usually gone) 11:30, 26 May 2017 (CDT)
That wording is indeed more accurate, and what I meant in general. Although it has also happened that a contributor added the sector numbers manually without descriptions, e.g. from DeePsea's search results for sector type, and those would also be in numeric order. I'd still call that skeleton lists, but once a contributor has added descriptions I'd no longer consider that section a skeleton anymore. And of course the order can then be any that is best suited for the pertaining map. --Xymph (talk) 13:11, 26 May 2017 (CDT)

XymphBot and skeletons (again)

Hi Xymph,

I'm just now beginning to examine some bot edits in more detail.  It's still amazing how many features you managed to add in a short time.  Anyone who has started a few dozen pages from scratch will appreciate the massive efficiency gain (with respect to both actual writing and interlarded decompression :D

I notice that you sometimes return to address an issue manually, e.g.:

  1. erasing a footnote
  2. removing a category
  3. adding a template that was already present
  4. transposing a heading with a paragraph

You've forgotten more about programming than I'll ever know  :>  so perhaps you added validations to the code as needed, or the bot itself alerted you when existing text didn't match the standard form.  But if that wasn't possible (because string parsing is very hard), I would like to help by finding any remaining similar issues.  Is there a way to narrow down relevant edits of each type, e.g. by time stamps or canned edit summaries?

Yes, I've complained about gameplay articles attracting hoi polloi revisions because of the low barrier to entry, but those still deserve to be evaluated in context by a human.

Anyway, I hope that makes some sense, and thanks again for caring about our project.    Ryan W (living fossil) 01:33, 12 July 2017 (CDT)

Thanks for the com(pli)ments. Also, I'm glad to see you becoming more active again. I don't follow "interlarded decompression" though, is that one of your Dr Who terms? ;)
Regarding manual touch-ups: yes sometimes the pattern matching is too tricky, but usually it's a trade-off between effort and gain.
1. Footnotes between/below Things tables are very rare so I simply didn't bother trying to code for them, as it would be harder than manually restoring them.
2. On most map pages the Statistics section is followed by other (stub-)sections. The MW API – and thus dmmpstBot.php – allows updating a single (sub)section, but category lines don't form their own section and are simply included in the last one. So in case of those Heretic pages the cats got stripped off by the bot edit because I didn't code for the possibility of any being in the Statistics section, as pages usually have External links headers and such in between. Additionally, conventionsBot.php warns me of category lines embedded somewhere else than at the end of pages. Lastly, the bot removals of the unnecessary "Levels by name" cats were actually convenient, as those could be dropped from all Heretic pages, while the firemace cat only needed to be re-added on 20 of them.
3. That happened because mapviewBot.php didn't take the empty line before the existing mapspots template into account. It's too long ago to remember exactly, but glancing over the code it could be that mapviewBot.php handles it correctly now. I must admit that sometimes I reviewed the interactive diffs too hastily and submitted edits needing manual clean-up which I then addressed in the pertaining script.
4. This page stood out because of the non-standard walkthrough formatting, but I didn't bother cleaning it up beyond the header move so that skeletonBot.php and mapviewBot.php wouldn't complain.
For cases 1 and 2, since dmmpstBot.php shows me the diff before actually submitting the edit, I know where to return to and manually restore things, and I also manually review all pages in a series afterwards. Thus I'm pretty sure I didn't leave (m)any loose ends of issues caused by bot edits, but I may have slipped up on occasion. I can't think of a pattern by which to locate them though.
For me the fun is both in programming and thus helping myself/others avoid tedious jobs (that were evidently hard to execute consistently and completely), and improving DW's content and presentation of factual information. After a year and a half, I do consider it my project too. :)
Some statistics: in my archive are 166 .ini files (including three for all single-level Doom, Doom II and Heretic PWADs) covering 3499 map pages.
So, much thanks for the offer of help and sorry for not being able to point you directly in the desired, uhh, direction. It you're still undeterred after all the above, here's how you (or anyone) can help. However, currently I'm immersed in INFO.c tools programming and subsequently transferring their output to the wiki, so combined with real-life interference it may take a while before I can return to bot-based map page updates. --Xymph (talk) 07:32, 12 July 2017 (CDT)
Thank you; that is quite informative.  There's certainly no rush, and I hope the bold text didn't imply otherwise.  (I did reread the discussions first, so I'm glad I hadn't overlooked something obvious in your updates.)
I'm learning that "active" is a floating-point quantity  :>  so e.g. I will hesitate to join a project systematically playtesting megawads.  That said, I already had your list bookmarked because some of those articles appeared hastily during an RFC, which I suspected would cause trouble for somebody.  Now that I know it's XymphBot, its previously successful PWADs can serve as examples to follow.
Re "decompression" — AFAICT it's a common experience that long, tedious jobs may be followed by an unrelated mindless task to clear one's head, before one can return to a different demanding activity.  It's hard to plan for, especially with real-life concerns present, so it's another reason bots are great.  :D    Ryan W (living fossil) 16:44, 12 July 2017 (CDT)

←←←

Hi again.  I've changed (hopefully improved) Template:Map skel slightly; does that affect XymphBot at all?  I dimly remembered that it did, but now I can't find the original discussion (oops...), only several cousins:

If I'm barking up the wrong tree again, just say so.  :>   I did not have to change any headings to match our newest articles, which presumably means you and the other "map regulars" had already applied your usual thoroughness.  In theory there might be additional tricks available, e.g. reading the slot automatically from the page title, but I would like to keep the markup simple because non-regulars sometimes create map pages (especially after Cacowards).    Ryan W (living fossil) 00:31, 17 July 2017 (CDT)

You may be remembering the remark at the end of Template:Things. But DMMPST includes its own copies of the templates it uses, so there is no direct consequence. However for consistency, I similarly dropped the ftp: stub and added the Adding custom music wikilink; and adopted the dummy NavboxTemplateGoesHere line for which I had to update navboxBot.php as well. Thanks for the – indeed – improvements. --Xymph (talk) 08:28, 17 July 2017 (CDT)

"Revert misguided spots formatting"

Hi Xymph.  This function appears not to detect multiple letters within the same parentheses.  For example, here the text ''(J, K)'' was unchanged.    Ryan W (living fossil) 06:52, 16 August 2017 (CDT)

Good catch, thanks. Fixed for all Ultimate Doom maps, found none in the Doom2 map pages. Let me know if you find some in another map series. --Xymph (talk) 08:20, 16 August 2017 (CDT)

November featured article thread

Which "both articles", RyanW?   — Hmm, probably just being sloppy with verb tenses again.  "Both articles will have been featured when all is said and done, so there's no harm done to the original goal of showcasing the good work in both".  Does that make sense?    Ryan W (living fossil) 17:49, 10 November 2017 (CST)

I don't dare to say 'no' anymore at this point. :-P Yes --Xymph (talk) 02:38, 11 November 2017 (CST)

Report on Wikipedia linking?

Hi Xymph,

Ideas for future scripts, anyone?   — okay, what about a list of all links to Wikipedia meeting certain criteria?

There was a brief IRC discussion about knock-on effects of Wikipedia deletions.  It can be argued that any game-related article aside from the main AAA releases is at risk.  I would like to remediate this somehow, and as Quasar pointed out, that is much easier before the article is deleted.  :7

It is of course your decision whether this sounds interesting (and whether you have time!).  I have vague ideas about implementation details, which can and should be finalized through general discussion.  I am asking you first, however, because if we don't have a bot then my initial proposal will change.  :D

Thanks for reading, as always.    Ryan W (living fossil) 17:03, 18 March 2018 (CDT)

Time, indeed, is of the essence :) and now that my .exe/iwad project is done, I wanted to turn to other things on my to-do list, on DoomWiki and elsewhere. But this sounds like a multi-day project rather than a multi-month one, so it's doable. Can you elaborate on what the script should do, across which pages against what criteria, producing what kind of output? --Xymph (talk) 06:51, 19 March 2018 (CDT)
I can respond, but it creates no obligation on your part.  :>
Output would be a one-to-many mapping like this:
Wikipedia page Linked from
GeForce 6 series Aspect ratio
Category:Node builders
Christoph Oelckers (Graf Zahl)
Doom 3
PTV (Family Guy) Aspect ratio
File:Familyguy.jpg
Shareware
On any page where "encyclopedic" content is expected, sources might be added, so the territory would be namespaces 0, 4, 6, 10, 12, 14 except
  • Doom Wiki:Central Processing*
  • Doom Wiki:RFC*
  • Help:Spam/Archive
Criteria provide the most room for debate.  If I did this manually, I would want two things:
  1. The ability to identify every link, even if added indirectly (interwiki, transclusion, etc).
  2. A conservative first pass to filter out less likely candidate links.  Total volume is a bit too large to digest otherwise.
I assume #1 can be addressed by requesting pages in HTML format.  #2 is far more heuristic; options include
  • For each Wikipedia article, read Wikipedia's category system to identify related areas (gaming, computing, scripture, ...).
  • List only links formatted locally as references (citation tags, specific templates, ...).
  • List only links under whitelisted section titles.
I would choose the last approach.  The second would be very incomplete (indeed one motivation is to clean up sloppy citations from a decade ago).  The first would be beating the bot's head against the scale and nonlinearity of Wikipedia's categories, at least until it got blocked.  I did manage to compile promising titles already:
  • Additional information
  • Bibliography
  • Credits
  • Discography
  • Encyclopedia
  • External link
  • External links and references
  • External links:
  • External links
  • Footnotes
  • General info
  • General source material
  • Info
  • Information
  • Misc.
  • Misc
  • Miscellaneous
  • More information
  • Note
  • Notes
  • Other links
  • Other notes
  • Other references
  • Other
  • Publications
  • Published works
  • Reference/glossary section
  • References
  • Related articles
  • Related links
  • Related
  • Resources
  • See also
  • Source
  • Sources
  • Useful pages
In this context, a "section" ideally comprises all text until the next header at the same level.  The next header at any level is likely 99% accurate however.
This post represents only my personal hypotheses, subject to later consensus-building and any needed compromises regarding what is or is not feasible to code.    Ryan W (living fossil) 17:34, 19 March 2018 (CDT)
Further idea: this probably needs a throttle to avoid mass image uploads, which link pages such as Fair use through templates, and in practice have section structures varying by user.  If the search returns more than 25 results (say) for the same external link in the same namespace, just write [26 pages in namespace 6] or something.  If it's clearly an unusual case, I'll deal with it manually.    Ryan W (living fossil) 23:18, 22 March 2018 (CDT)
The past two days of work on the script resulted in a fair amount of progress. The last run went over 5 of the 6 namespaces, excluding main – I don't have an idea of the link volume in the latter yet. Because runs over that many pages take a long time, the script logs its progress like this:
Attempting to log in... Success.
collecting Wikipedia namespaces:  34

processing 'Doom Wiki'...
processed:    43 pages

processing 'File'... 100 200 300 400 500
processing 'File' from File:32in24_MAP09_hilltop.png... 100 200 300 400 500
processing 'File' from File:AnimFace.gif... 100 200 300 400 500
processing 'File' from File:Cchest4_MAP13.png... 100 200 300 400 500
processing 'File' from File:CommunityChest-map12-mud.png... 100 200 300 400 500
processing 'File' from File:Congestion1024-map19-end.png... 100 200 300 400 500
processing 'File' from File:DTWID-LE_E2M8_map.png... 100 200 300 400 500
processing 'File' from File:Doom3_weapons_07.jpg... 100 200 300 400 500
processing 'File' from File:Doom_v0.3_thestore.png... 100 200 300 400 500
processing 'File' from File:E3M4_map.png... 100 200 300 400 500
processing 'File' from File:Eternity-e2m1-s.png... 100 200 300 400 500
processing 'File' from File:H2H-Xmas_MAP31_map.png... 100 200 300 400 500
processing 'File' from File:Heretic-e3m4-snail.png... 100 200 300 400 500
processing 'File' from File:Interception_MAP04_map.png... 100 200 300 400 500
processing 'File' from File:Kansams_Trial_E2M7_map.png... 100 200 300 400 500
processing 'File' from File:MM_MAP02_Secret.png... 100 200 300 400 500
processing 'File' from File:MementoMori2-map04-down.png... 100 200 300 400 500
processing 'File' from File:NDCP-map23-torches.png... 100 200 300 400 500
processing 'File' from File:Origwad_2.png... 100 200 300 400 500
processing 'File' from File:Psx-command-control-start.png... 100 200 300 400 500
processing 'File' from File:Requiem-map27-end.png... 100 200 300 400 500
processing 'File' from File:Scythe2_MAP03_map.png... 100 200 300 400 500
processing 'File' from File:Stardate20X6.png... 100 200 300 400 500
processing 'File' from File:TheEvilUnleashed-e3m5-cage.png... 100 200 300 400 500
processing 'File' from File:Vile_Flesh_MAP11_map.png... 100 200
processed: 12260 pages

processing 'Template'... 100 200 300 400 500
processing 'Template' from Template:MAYhem_1500_21-30... 100 200 300 400
processed:   940 pages

processing 'Help'...
processed:     8 pages

processing 'Category'... 100 200 300 400 500
processing 'Category' from Category:Easter_eggs... 100 200 300 400 500
processing 'Category' from Category:MrGuyTodd_levels... 100 200 300 400
processed:  1491 pages
Some info about what listWikiLinks.php does:
  • init namespace lists and patterns; at first I used an API call to get Wikipedia's namespace list, then hard-coded it to save a call; in the exception list, Template:Wikipedia and Template:Wp were also needed
  • the regex patterns are: '{{wp\|(.+?)(?:\|.+?)?}}', '\[\[Wikipedia:(.+?)(?:\|.+?)?\]\]', and '\[https?://en.wikipedia.org/wiki/(.+?)(?: .+?)?\]'
  • from all selected namespaces, lists of pages are queried at 500 per batch and each page is then fetched and searched for the link patterns; this is the most time-consuming part, so a progress marker is logged every 100 pages
  • all data is collected in an array (keyed by Wikipedia path) of arrays (with DoomWiki paths), which are all sorted with flags SORT_NATURAL and SORT_FLAG_CASE
  • the tables are generated by traversing the data array for each Wikipedia namespace; for entries with more than 20 DW links, a more value is appended
Filtering of WP links is not implemented yet, I'm not sure that's wise. For one thing, quite a few WP links occur in the intro (section 0) of a page, so that should be in your list too. For another, perhaps the link volume isn't overwhelming, I am now running the script on main space only to find out.
About avoiding mass uploaded images, the script could skip filenames ending in "_map.png" and "_title.png" to save processing time indeed. Or what (else) did you have in mind?
Do you now want to launch a general discussion before I continue? --Xymph (talk) 05:41, 27 March 2018 (CDT)
I added the main space data. Okay, yes, it's voluminous. :) --Xymph (talk) 06:51, 27 March 2018 (CDT)
There's so much progress here, people can't help being inspired to further suggestions!  :>   I hope you won't interpret it as criticism if I make suggestions before testing any bibliographic edits — it's a big job and I don't want people to have to revert me later.
Regarding mass uploaded images, you have apparently handled this, by counting links in template space once per template instead of once per transclusion.
You're probably correct to parse wikitext instead of HTML; it provides more fine-grained results.  Note that:
  • [[wiki: also points to Wikipedia.
  • I'm a regex newbie, but the above doesn't seem to include MediaWiki's case insensitivity on the first character.  E.g. {{wp and {{Wp are the same.  fair use and Fair use are the same.  Further, interwiki prefixes are completely case-insensitive.
  • Similarly, Blake Stone: Aliens of Gold and Blake_Stone:_Aliens_of_Gold are the same.
  • I endorse Gez's idea of a third column: direct links are far more likely to be unmarked sources, at least on the articles I've seen.
If the consensus is not to filter mainspace links (time for that general discussion  :D )  then there will be many more ideas and many more possible mass edits to implement them.  I personally still intend to focus on the task in the OP, and would heuristically skip entries clearly outside it (Blu-Ray, Lava).  If no one else made use of the remainder, then anything I missed would simply remand to a more general link rot project in the far future.
Hope that makes some sense, and thanks as always for your patience.    Ryan W (living fossil) 19:56, 27 March 2018 (CDT)

Hey, a suggestion: would it be possible to have a third column listing the link types used? More generally, a bot capable of looking through the wiki pages to find any and all instance of direct links to domains that have interwiki entries would be useful for maintenance. --Gez (talk) 06:25, 27 March 2018 (CDT)

Sure, you mean whether a DW page uses wp template, interwiki or https link for a WP link? How should the third column look like, for example? And what if more than one method is used for the same WP link on a page?
The general suggestion would be a separate bot script. --Xymph (talk) 06:42, 27 March 2018 (CDT)
I figure the third column could just contain a simple marker, like D for direct link, I for interwiki link, and T and template link. If more than one method is used, then you get more than one marker. If there's a direct link and a template link, then it'd show DT for example. --Gez (talk) 07:03, 27 March 2018 (CDT)

←←←

First, to clear up a misconception, bot scripts aren't crawling the site's HTML pages. They communicate via the MW API and thus, when working with page content, they indeed retrieve and store wikitext, just like a human using the edit form. And in the Template namespace, fetching pages therefore gets the template wikitext; nothing transcluded. I wouldn't even want to process HTML for page content, that would be error-prone and unnecessarily complicated.

Thanks for the feedback, the interwiki pattern was updated to '\[\[Wiki(?:pedia)?:(.+?)(?:\|.+?)?\]\]'. All patterns are matched using the i modifier, so case-insensitively. But the WP paths found are case-sensitive, and stored that way in the data structure. The script doesn't know which one of those is the canonical WP page. To collate WP paths to some degree, the script now converts all underscores to spaces, to deal with Blake_Stone-type situations. But I'm not sure no WP path legitimately includes an underscore, and whether that would result in a broken WP link in the first column of the tables.

Also, paths starting with a letter always use a capital initial in WP's URL, so the first letter is now uppercased to collate [Ff]air use-type situations. But I can't safely collate "Alice in Chains" and "Alice In Chains", so in those situations there are still multiple entries. Further, in File space, pages ending in " map.png" were skipped, saving about half an hour of processing time.

Lastly, the third column is present, showing very few direct links, most of which I fixed already. No DW pages mixed link types for the same WP path, either. After the following log:

Attempting to log in... Success.
collecting Wikipedia namespaces:  34

processing 'main'... 100 200 300 400 500
processing 'main' from main:Counterattack... 100 200 300 400 500
processing 'main' from main:E1M3:_Toxin_Refinery_(Phobos_Revisited)... 100 200 300 400 500
processing 'main' from main:E4M9:_Bean_Oversight_(Doom_the_Way_id_Did:_The_Lost_Episodes)... 100 200 300 400 500
processing 'main' from main:IRC... 100 200 300 400 500
processing 'main' from main:MAP02:_Security_(H2H-Xmas)... 100 200 300 400 500
processing 'main' from main:MAP06:_Colossus_(The_Rebirth)... 100 200 300 400 500
processing 'main' from main:MAP10:_Eureka_Signs_(BTSX-E2)... 100 200 300 400 500
processing 'main' from main:MAP15:_Descent_to_Core_(Speed_of_Doom)... 100 200 300 400 500
processing 'main' from main:MAP21:_Aberinkula_(MAYhem_2048)... 100 200 300 400 500
processing 'main' from main:MAP27:_Emerald_Spire_(Sunlust)... 100 200 300 400 500
processing 'main' from main:MAP34:_Abstraction_(Claustrophobia_1024_2)... 100 200 300 400 500
processing 'main' from main:Psychophobia... 100 200 300 400 500
processing 'main' from main:The_Doom_Movement_Bible... 100 200 300
processed:  6874 pages

processing 'Doom Wiki'...
processed:    43 pages

processing 'File'... 100 200 300
processing 'File' from File:32in24_MAP09_hilltop.png... 100 200 300
processing 'File' from File:AnimFace.gif... 100 200 300
processing 'File' from File:Cchest4_MAP13.png... 100 200 300
processing 'File' from File:CommunityChest-map12-mud.png... 100 200 300 400 500
processing 'File' from File:Congestion1024-map19-end.png... 100 200 300
processing 'File' from File:DTWID-LE_E2M8_map.png... 100 200
processing 'File' from File:Doom3_weapons_07.jpg... 100 200 300
processing 'File' from File:Doom_v0.3_thestore.png... 100 200 300 400
processing 'File' from File:E3M4_map.png... 100 200 300
processing 'File' from File:Eternity-e2m1-s.png... 100 200 300
processing 'File' from File:H2H-Xmas_MAP31_map.png... 100 200
processing 'File' from File:Heretic-e3m4-snail.png... 100 200 300
processing 'File' from File:Interception_MAP04_map.png... 100 200 300
processing 'File' from File:Kansams_Trial_E2M7_map.png... 100 200
processing 'File' from File:MM_MAP02_Secret.png... 100 200 300 400
processing 'File' from File:MementoMori2-map04-down.png... 100 200 300 400
processing 'File' from File:NDCP-map23-torches.png... 100 200
processing 'File' from File:Origwad_2.png... 100 200 300
processing 'File' from File:Psx-command-control-start.png... 100 200 300 400
processing 'File' from File:Requiem-map27-end.png... 100 200 300
processing 'File' from File:Scythe2_MAP03_map.png... 100 200 300
processing 'File' from File:Stardate20X6.png... 100 200
processing 'File' from File:TheEvilUnleashed-e3m5-cage.png... 100 200 300
processing 'File' from File:Vile_Flesh_MAP11_map.png... 100
processed: 12260 pages
skipped:    3465 map pages

processing 'Template'... 100 200 300 400 500
processing 'Template' from Template:MAYhem_1500_21-30... 100 200 300 400
processed:   940 pages

processing 'Help'...
processed:     8 pages

processing 'Category'... 100 200 300 400 500
processing 'Category' from Category:Easter_eggs... 100 200 300 400 500
processing 'Category' from Category:MrGuyTodd_levels... 100 200 300 400
processed:  1491 pages

the new tables for all DW namespaces are online. --Xymph (talk) 12:58, 28 March 2018 (CDT)

Thank you!  I'm slightly out of breath, watching it grow.  :>   Maybe I spend too much time thinking about the global topology of the wiki, but I sense this could lead in several useful directions, not all to do with modifying links.
They communicate via the MW API   — I did know that, and in that very documentation I kept seeing mentions of "parsed" text.  I hoped there was a feature which could make your task easier by reducing the number of different cases to search for, i.e. converting all links into direct links.  Of course I accept your word that wikitext is easier; as I said, in hindsight that should be easier for everyone else also.
But I'm not sure no WP path legitimately includes an underscore   — I'm reasonably sure, based on this experiment, that equivalence between spaces and underscores is implemented at a fairly low level.  I can even type the spaces into my URL bar and still get where I intended: https://en.wikipedia.org/wiki/Death of Michael Jackson   Does that work in general?  Nope: https://www.youtube.com/watch?v=VxqaI c9j g
But I can't safely collate "Alice in Chains" and "Alice In Chains"   — True enough.  So far none of these points is enormously important  :D  however:
Thanks for the feedback   — It's the least I can do.  Something is still wrong, because the table omits the link that started my train of thought: this one.
Ryan W (living fossil) 18:12, 28 March 2018 (CDT)
Ow, that's because Template:Wikipedia wasn't mentioned to far, and I didn't realize it either. It's now covered by regex '{{wikipedia(?:\|(?:title=)?(.+?)(?:\|.+?)?)?}}' and type S (Source). Besides with a positional or named parameter, it can be used parameterless which presented a small problem, but that's now solved. See the updated list. --Xymph (talk) 10:58, 29 March 2018 (CDT)
D'oh!  I saw "exception list" above, and somehow assumed it was part of the regex even though I couldn't see it.  (I did page through templates at some point to check if there were any more.  There are not.)
I think that's all of my questions, so we'll see what others have to say.  :>    Ryan W (living fossil) 16:36, 29 March 2018 (CDT)

Interwiki links

Copied from Wikipedia topic:
More generally, a bot capable of looking through the wiki pages to find any and all instance of direct links to domains that have interwiki entries would be useful for maintenance. --Gez (talk) 06:25, 27 March 2018 (CDT)

To verify, this script needs to search pages for the URLs in the Interwiki table (up to the $1 parameter), and show which interwiki prefix should be used instead? --Xymph (talk) 12:26, 27 March 2018 (CDT)
Yeah, I figure that should be enough. The only potential subtlety would be to watch out for http/https, i.e. there could be an http direct link to a site that has https in the interwiki table (or vice-versa). --Gez (talk) 13:42, 27 March 2018 (CDT)
If we're talking about maintenance, the reverse of this (an analogue of Special:LinkSearch for interwikis only) would occasionally be useful as well, e.g. when a site goes down permanently.  Again a low priority, unless someone finds a live example.    Ryan W (living fossil) 20:00, 27 March 2018 (CDT)
Not sure what info a "reverse" script (or addition to the current script) should generate – if you'd like something done, please elaborate.
The current script was pretty easy to derive from the previous one, and produces the latest table after this log:
Attempting to log in... Success.
collecting unique Interwiki list: 135

processing 'main'... 100 200 300 400 500
processing 'main' from main:Counterattack... 100 200 300 400 500
processing 'main' from main:E1M3:_Toxin_Refinery_(Phobos_Revisited)... 100 200 300 400 500
processing 'main' from main:E4M9:_Bean_Oversight_(Doom_the_Way_id_Did:_The_Lost_Episodes)... 100 200 300 400 500
processing 'main' from main:IRC... 100 200 300 400 500
processing 'main' from main:MAP02:_Security_(H2H-Xmas)... 100 200 300 400 500
processing 'main' from main:MAP06:_Colossus_(The_Rebirth)... 100 200 300 400 500
processing 'main' from main:MAP10:_Eureka_Signs_(BTSX-E2)... 100 200 300 400 500
processing 'main' from main:MAP15:_Descent_to_Core_(Speed_of_Doom)... 100 200 300 400 500
processing 'main' from main:MAP21:_Aberinkula_(MAYhem_2048)... 100 200 300 400 500
processing 'main' from main:MAP27:_Emerald_Spire_(Sunlust)... 100 200 300 400 500
processing 'main' from main:MAP34:_Abstraction_(Claustrophobia_1024_2)... 100 200 300 400 500
processing 'main' from main:Psychophobia... 100 200 300 400 500
processing 'main' from main:The_Doom_Movement_Bible... 100 200 300
processed:  6874 pages

processing 'Doom Wiki'...
processed:    43 pages

processing 'File'... 100 200 300 400 500
processing 'File' from File:32in24_MAP09_hilltop.png... 100 200 300 400 500
processing 'File' from File:AnimFace.gif... 100 200 300 400 500
processing 'File' from File:Cchest4_MAP13.png... 100 200 300 400 500
processing 'File' from File:CommunityChest-map12-mud.png... 100 200 300 400 500
processing 'File' from File:Congestion1024-map19-end.png... 100 200 300 400 500
processing 'File' from File:DTWID-LE_E2M8_map.png... 100 200 300 400 500
processing 'File' from File:Doom3_weapons_07.jpg... 100 200 300 400 500
processing 'File' from File:Doom_v0.3_thestore.png... 100 200 300 400 500
processing 'File' from File:E3M4_map.png... 100 200 300 400 500
processing 'File' from File:Eternity-e2m1-s.png... 100 200 300 400 500
processing 'File' from File:H2H-Xmas_MAP31_map.png... 100 200 300 400 500
processing 'File' from File:Heretic-e3m4-secret.png... 100 200 300 400 500
processing 'File' from File:Interception_MAP03_map.png... 100 200 300 400 500
processing 'File' from File:Kansams_Trial_E2M6_map.png... 100 200 300 400 500
processing 'File' from File:MM_MAP01_map.png... 100 200 300 400 500
processing 'File' from File:MementoMori2-map03-plus.png... 100 200 300 400 500
processing 'File' from File:NDCP-map19-water.png... 100 200 300 400 500
processing 'File' from File:Odessa_3.png... 100 200 300 400 500
processing 'File' from File:Psx-central-processing-02.png... 100 200 300 400 500
processing 'File' from File:Requiem-map22-cacos.png... 100 200 300 400 500
processing 'File' from File:Scythe2-map28-afrit.png... 100 200 300 400 500
processing 'File' from File:Speed_of_Doom_MAP24_map.png... 100 200 300 400 500
processing 'File' from File:TheEvilUnleashed-e3m1-chasm.png... 100 200 300 400 500
processing 'File' from File:Vanguard_MAP06_map.png... 100 200
processed: 12292 pages

processing 'Template'... 100 200 300 400 500
processing 'Template' from Template:MAYhem_1500_21-30... 100 200 300 400
processed:   940 pages

processing 'Help'...
processed:     8 pages

processing 'Category'... 100 200 300 400 500
processing 'Category' from Category:Easter_eggs... 100 200 300 400 500
processing 'Category' from Category:MrGuyTodd_levels... 100 200 300 400
processed:  1491 pages
Some info about what listInterLinks.php does:
  • init namespace lists and patterns; DW paths in the exception list are the previous five, plus 'Doom Wiki:Departure from Wikia', 'Doom Wiki:Source material' and 'Help:Special pages'
  • the interwiki list is fetched and the URLs are converted to regexes, e.g. the first one becomes "https?://3dfxdev\.net/edgewiki/index\.php\?title=[\w#%/:.=-]+"; also, some interwikis/URLs occur in the list twice, so it is deduped from 157 to 135
  • pages are fetched from namespaces analog to listWikilinks and matches for each regex are stored in an array (keyed by interwiki prefix) of arrays (keyed by DW path)
  • the arrays are then sorted by those keys – but the direct links remain stored in the order they were encountered in that page – and the table is generated
On the first full pass, file pages ending in " map.png" weren't skipped yet but that's enabled now to save time in future runs. If more/different features are needed, let me know. For one thing, the $1 regex was a bit limited, I've already expanded it to "[\w#@%&/?!:;,.~=+-]+". --Xymph (talk) 04:50, 3 April 2018 (CDT)

Templatable links

Moved from Wikipedia topic:
And while I'm on that general topic, a third script could search for direct links that could be templated -- I'm thinking about stuff like direct links to forums or archives, notably. There's no hurry, though. --Gez (talk) 07:03, 27 March 2018 (CDT)

Okay, one thing are a time. :) --Xymph (talk) 12:26, 27 March 2018 (CDT)
This one should be uncontroversial, but "no hurry" is an understatement.  :>   IMO a bot performing style edits would have to sweep at regular intervals, and I don't assume Xymph or anyone else can commit to that (wishful thinking solution: someone in the Doom community figures out how to host bots as autonomous processes, like the WMF does).    Ryan W (living fossil) 17:04, 27 March 2018 (CDT)
Actually, running scripts regularly and posting their output to list pages automatically would be trivial for me, and less so for anyone else. ;-) But multiple admins/editors are monitoring Recent Changes already and when incorrect styles and links are fixed right away, regular runs won't even be necessary. An incidental run every 4-6 months or so would be enough to catch anything falling through those cracks.
The new script was really easy to derive from the previous one, and produces the latest table after this log:
Attempting to log in... Success.
collecting external templates list: 20

processing 'main'... 100 200 300 400 500
processing 'main' from main:Cory_Scott_(NiTROACTiVE)... 100 200 300 400 500
processing 'main' from main:E1M3:_The_Gatehouse_(Heretic)... 100 200 300 400 500
processing 'main' from main:E4M5:_The_Witching_Hour_(Crusades)... 100 200 300 400 500
processing 'main' from main:Hub_4:_Desolate_Garden... 100 200 300 400 500
processing 'main' from main:MAP02:_Nullth_Precinct_(Community_Chest)... 100 200 300 400 500
processing 'main' from main:MAP05:_Wasted!_(NDCP2)... 100 200 300 400 500
processing 'main' from main:MAP09_(2_Sectors)... 100 200 300 400 500
processing 'main' from main:MAP14:_Vile_Gear_(Swift_Death)... 100 200 300 400 500
processing 'main' from main:MAP20:_Speedtraps_for_the_Bee_Kingdom_(BTSX-E2)... 100 200 300 400 500
processing 'main' from main:MAP26:_Teleport_Control_(Daedalus:_Alien_Defense)... 100 200 300 400 500
processing 'main' from main:MAP32:_Ultimative_Geheimnis_(Whispers_of_Satan)... 100 200 300 400 500
processing 'main' from main:Porkalator... 100 200 300 400 500
processing 'main' from main:Terminatrix... 100 200 300 400
processed:  6916 pages

processing 'Doom Wiki'...
processed:    43 pages

processing 'File'... 100 200 300 400 500
processing 'File' from File:32in24_MAP09_hilltop.png... 100 200 300 400 500
processing 'File' from File:AnimFace.gif... 100 200 300 400 500
processing 'File' from File:Cchest4_MAP12_Deathmatch.png... 100 200 300 400 500
processing 'File' from File:CommunityChest-map12-end.png... 100 200 300 400 500
processing 'File' from File:Congestion1024-map17-switches.png... 100 200 300 400 500
processing 'File' from File:DTWID-LE_E2M1_map.png... 100 200 300 400 500
processing 'File' from File:Doom3_backpack.png... 100 200 300 400 500
processing 'File' from File:Doom_pinky_2.jpg... 100 200 300 400 500
processing 'File' from File:E3M1_heretic.png... 100 200 300 400 500
processing 'File' from File:Eternally_Yours_MAP12_map.png... 100 200 300 400 500
processing 'File' from File:H2H-Xmas_MAP24_map.png... 100 200 300 400 500
processing 'File' from File:Heretic-e3m1-storage.png... 100 200 300 400 500
processing 'File' from File:Infinity_E1M8_Map.png... 100 200 300 400 500
processing 'File' from File:KansamsTrial-E2M7-start.png... 100 200 300 400 500
processing 'File' from File:MM8BDM_Gameplay_1.png... 100 200 300 400 500
processing 'File' from File:MementoMori2-map01-start.png... 100 200 300 400 500
processing 'File' from File:NDCP-map17-hex.png... 100 200 300 400 500
processing 'File' from File:Obituary_MAP04_map.png... 100 200 300 400 500
processing 'File' from File:Programador1_Strife.png... 100 200 300 400 500
processing 'File' from File:Requiem-map16-elemental.png... 100 200 300 400 500
processing 'File' from File:Scythe2-map13-center.png... 100 200 300 400 500
processing 'File' from File:SpeedOfDoom-map28-redskullkey.png... 100 200 300 400 500
processing 'File' from File:Talosian_Incident_MAP15_map.png... 100 200 300 400 500
processing 'File' from File:Vae_Victus_2_MAP02_map.png... 100 200 300
processed: 12341 pages

processing 'Help'...
processed:     8 pages

processing 'Category'... 100 200 300 400 500
processing 'Category' from Category:Earth_(WAD)_levels... 100 200 300 400 500
processing 'Category' from Category:MrGlide_levels... 100 200 300 400
processed:  1493 pages
Some info about what listExternLinks.php does:
  • init namespace lists and patterns; pages in the exception list are just 'Doom Wiki:Central Processing.*' and 'Help:Spam/Archive'
  • from External link templates the following were manually added to the patterns list (the others don't contain direct links):
$externlist['Competnftp'] = 'https?://(?:www\.)?doom\.com\.hr/temp/compet-n/';
$externlist['Competnmap'] = 'https?://(?:www\.)?doom\.com\.hr/index.php\?page=compet-n_database&cndb=';
$externlist['Competnwad'] = 'https?://(?:www\.)?doom\.com\.hr/index.php\?page=compet-n_database&wad_id=';
$externlist['Deb-pkg']    = 'https?://packages\.debian\.org/';
$externlist['Drdforums']  = 'https?://forum\.drdteam\.org/viewtopic.php\?';
$externlist['Dsda']       = 'https?://doomedsda\.us/wad';
$externlist['Dsdaftp']    = 'https?://doomedsda\.us/lmps/';
$externlist['Dwforums']   = 'https?://(?:www\.)?doomworld\.com/vb/thread/';
$externlist['Dwforumst']  = 'https?://(?:www\.)?doomworld\.com/vb/post/';
$externlist['Edgeeg']     = 'https?://edge\.sourceforge\.net/edit_guide/';
$externlist['Idftp']      = 'https?://mirrors\.syringanetworks\.net/idgames/';
$externlist['Idgames']    = 'https?://(?:www\.)?doomworld\.com/idgames/';
$externlist['Idgamesmm']  = 'ftp://ftp\.fu-berlin\.de/pc/msdos/games/idgames/';
$externlist['Lc']         = 'https?://lccn\.loc\.gov/';
$externlist['Legacywiki'] = 'https?://legacywiki\.net/index\.php/';
$externlist['Realm667']   = 'https?://(?:www\.)?realm667\.com/index\.php\?option=com_content&view=';
$externlist['Vvwiki']     = 'https?://(?:www\.)?vavoom-engine\.com/wiki/index.php\?title=';
$externlist['Wip']        = 'https?://(?:www\.)?wadsinprogress\.info/\?a=listwads&wad=';
$externlist['Zanforums']  = 'https?://(?:www\.)?zandronum\.com/forum/showthread\.php\?tid=';
$externlist['Zdforums']   = 'https?://forum\.zdoom\.org/viewtopic\.php\?';
  • pages are fetched from namespaces analog to listInterlinks and matches for each regex are stored in an array (keyed by external template) of arrays (keyed by DW path); sorting and outputting remained the same
On the initial pass, templates being matched by their own template's link were skipped, but because some direct links occur in multiple templates a handful of them were still included as false positives. So for the full run over all namespaces, Template space was skipped.
For future runs, 'Idgames archive' is added to the exception list as direct links do belong on that page. As usual, file pages ending in " map.png" will also be skipped.
If I missed any external link templates in the patterns list, please let me know. --Xymph (talk) 14:29, 12 April 2018 (CDT)

Level has serious issues with popular mod loaded - put in "bugs", "trivia" or not at all?

Hey Xymph, I'm currently working on the walkthrough for MAP17: Metastasize. While playing through the level, I found that mods where boxes of rockets could be shot and blown up (e.g. Brutal Doom v21 beta) caused serious problems in this level, as one would telefrag the box (as part of the level progression) and die in the explosion. Hence, I'm planning to add the following line in into the page:

"This level should not be played with mods that allow boxes of rockets to be shot and blown up as if they were barrels (or where possible, have the setting turned off in the mod), as in the course of getting to each of the keys, the player will teleport onto a spot with a box of rockets already there. With such mods loaded, the player will telefrag the box of rockets and be hit by a point-blank explosion, which will likely kill them."

The question is - should this line go into "bugs", "trivia", or should it not be on the page at all? --15FiftySeven (talk) 09:51, 7 May 2018 (CDT)

Personally I wouldn't put it there. I would mention the issue instead on the Brutal Doom article. --Gez (talk) 12:03, 7 May 2018 (CDT)
The PWAD was designed for Boom-compatible ports, so when an unrelated mod has an unexpected gameplay effect, that's certainly not a map bug, and possibly not even something that needs to be mentioned on the PWAD page or the specific map page. Gez's suggestion works for me too. --Xymph (talk) 12:56, 7 May 2018 (CDT)

Shedding data from map articles?

Please double-check the recent changes:

These are all tagged with the revision text "use secrets & map data from textmapStats.php" and have atypically large negative deltas. --Quasar (talk) 17:22, 18 June 2018 (CDT)

I expected that would trigger someone's curiosity. :) As you may remember, DMMPST does not support UDMF. DMP2013Z.wad mixes seven UDMF maps with traditional ones, and dmmpstBot.php is the easiest way to generate a series of new pages. So in loading the pwad on top of the iwad (doom2.wad) as DMMPST requires, and requesting e.g. MAP02 statistics, DMMPST actually returns the iwad's data because it ignores the pwad UDMF data. Compare MAP02:_Twilight_of_the_Gods_(Doomworld_Mega_Project_2013)#Statistics with MAP02:_Underhalls_(Doom_II)#Statistics. I could have manually created the six skeleton pages without the things data, but that would be a bit more cumbersome than just letting dmmpstBot.php rip and fixing the secret sectors and map data afterwards, and shedding the things tables.
Note that the page for the seventh map article already existed with a prior things table, and I left that one alone even though it's not styled entirely like DMMPST would do it. I assume Joe-ilya entered that correctly at the time.
I suppose the proper way out of this conundrum would be to make DMMPST handle UDMF, but that's pretty tough and – given the relatively low number of UDMF maps – may not be worth the effort. --Xymph (talk) 03:12, 19 June 2018 (CDT)
OK thanks for the explanation. I just wanted to be sure something accidental didn't happen :) --Quasar (talk) 03:38, 19 June 2018 (CDT)

Knee deep in KDiZD and Chocorenderlimits

I saw that you just fixed "Knee deep in KDiZD" into KDiZD in the Chocorenderlimits page. There is a project named Knee deep in KDiZD and it seems more likely to be the one that first used Chocorenderlimts, given that it is a vanila project and all, while KDiZD is a ZDoom project. So, were you mistaking? Or did i miss something? --BuildCat (talk) 12:09, 27 June 2018 (CDT)

I ran some searches which came up empty, hence I concluded the red link was an oversight -- but with a nagging thought in the back of my mind that a forum regular might correct me. Thanks. ;) --Xymph (talk) 12:21, 27 June 2018 (CDT)

PSX maps

When you're around in IRC again, I'd like to send you the PSX maps, which I've converted to a vanilla-compatible format, for you to make map graphics for and run stats against - we can discuss adding support to DMMPST for thing tables etc. if you want as well - there are some special considerations with regards to how spectres work that would need handling, unless I change the extractor to modify the thing types of special spectres. --Quasar (talk) 14:18, 21 July 2018 (CDT)

In passing, I'll note that SLADE 3 can generate map images from PSX Doom and Final Doom maps, but you first need to select the VERTEXES lump and type the console command "vertexpsx". --Gez (talk) 16:27, 21 July 2018 (CDT)
Cool; I wasn't sure that ever got implemented. --Quasar (talk) 16:53, 21 July 2018 (CDT)
If you could update your extractor to change the thing type of regular spectres back to 58 (dec) and of nightmare spectres to some unused type (3007?), then it would require very little additional support in DMMPST to generate correct thing tables for them. --Xymph (talk) 06:20, 22 July 2018 (CDT)

Looking at the new Club Doom map, I think that edit is best reverted. Map images typically prune the meaningless sectors (holding cells, control sectors, etc.) to focus on the actual map part, and the presence of a "signature area" in Club Doom far away from the playable area makes the map thumbnail totally unreadable. --Gez (talk) 08:16, 27 July 2018 (CDT)

Not reverted but remedied the usual way. I included the area because it's described in Trivia, but this is fine too. --Xymph (talk) 09:23, 27 July 2018 (CDT)

Pre-1.9 map uploads

Hi Xymph.  Wow, the bot's lump parsing improves by the day, it seems.  :D

Please see e.g. here: "The following file is a duplicate of this file"  — that means a pixel-for-pixel duplicate, so no fair use justification will ever be possible.  Sorry.    Ryan W (living fossil) 04:10, 28 July 2018 (CDT)

Sorry but I have no idea what "bot's lump parsing" refers to.
The pre-v1.9 map views are planned to be used in console map comparisons, because the console levels were based on those older versions. Quasar and I discussed this on IRC yesterday, and I simply uploaded the two whole batches for convenience and naming consistency. Then the comparisons can always use v1.2/v1.666 filenames, and I may write a bot script to do that (including creation of the transcluded gallery, if we get the tabs template working), so naming consistency would help there. If duplicate files are problematic, would it be possible to replace them with redirects to their v1.9 counterparts? --Xymph (talk) 05:08, 28 July 2018 (CDT)
Based on this testing, I would say yes, replacing with redirects should work, provided nothing is written above the redirect command.    Ryan W (living fossil) 06:53, 28 July 2018 (CDT)

Ask and ye shall receive

https://www.doomworld.com/forum/topic/101943-doom-wiki-month-in-review-issue-4-late-edition/ --Quasar (talk) 08:41, 14 August 2018 (CDT)

Heh, nice. --Xymph (talk) 11:10, 14 August 2018 (CDT)

Template:Deleted

Oh wow. If I hadn’t seen it in print, I wouldn’t have believed it. Thank you! Ironically that happens *less* often in larger batches, because there isn’t all that thinking in between to distract me (but IMO each really has to be judged case by case as I said on DWF). Glad someone was paying attention. MuffinProvensen (talk) 12:47, 20 August 2018 (CDT)

I was just about to move this entry into chronological order myself, so... thanks. ;) Also, I had some difficulty understanding what this is about, coming from a little used account -- until I found you're RyanW. :) Still, is this about my adding the deleted template around those two Template:Code instances, or the further series I continued in your footsteps? What does "it" mean? Also, I don't follow DWF apart from a few specific topics, so what does that pointer refer to?
Really, I appreciate the enthusiastic response, but some of it is lacking sufficient context to follow. --Xymph (talk) 13:02, 20 August 2018 (CDT)
Yes, you are all good.  "Thank you" referred to both your corrections of me and the additional edits (though I am working from a disorganized offline list and probably would have found the latter eventually).  Sorry for the confusion; I try not to put any credentials into my phone that I really care about.  :D   Ryan W (living fossil) 17:07, 20 August 2018 (CDT)

Mapper articles

So, we are creating articles for every mapper mentioned in Doom wiki through script now? It does lighten the load for people adding more info thanks to existing skeletons so thanks :) BuildCat (talk) 6:13, 11 September 2018 (CDT)

Not every perhaps, but yes, see here. --Xymph (talk) 06:19, 11 September 2018 (CDT)

Crazy idea

Why don't we make some of the marked maps a bit larger on the page when and where it won't disrupt article layout? I don't mean like 2x as big or anything so dramatic, but maybe just large enough that the markers don't require a magnifying glass to check out ;) Usual disclaimer about my poor eyesight applies, of course. --Quasar (talk) 11:58, 8 October 2018 (CDT)

That's fine for large maps, though in your example case (and with my eyesight) still not enough to distinguish spawn 18 from spawn 19. ;-) Feel free to apply it to the other two SVE maps as well. But I think it won't avoid the need to open the spawn map in another tab to really distinguish all points.
For more typically sized DM maps, the standard scaling should be adequate to discern the spawn numbers, or are you proposing to increase their scaling too? --Xymph (talk) 12:57, 8 October 2018 (CDT)
True regarding the numbers, but, for me at least, I can more easily tell the blue team spawns from the green DM spots with the size increase. The other examples you give look good already to me. --Quasar (talk) 14:18, 8 October 2018 (CDT)

"PWAD top-5" statistics

Hahaha, Fredrik would be proud [2].  This is exactly the sort of enhancement which only a bot can make — even if we imagined that we could have a strict rule about double-checking every new map article, people would forget to do so.

One small suggestion: include a date stamp, like the speedrun tables have?  That would protect against any future change which created lag between updates (e.g. MediaWiki upgrade disabling Wikimate indefinitely, or more optimistically, an increase in active editors who insist on documenting each megawad on day zero).  Hope that makes sense, thanks.    Ryan W (living fossil) 09:05, 21 December 2018 (CST)

Sure, done. Actually, it's not even a bot script, just a Bash script that greps/sorts through results from earlier DMMPST runs, bot-wise and manually. :) --Xymph (talk) 11:06, 21 December 2018 (CST)
For your Fredrik reference, I assumed you specifically meant this change, and interpreted vanilla maps as stock levels (doing stats for all vanilla maps covered on the wiki seems rather pointless, and then, why not for maps requiring a source port?). So this is now completed. :) --Xymph (talk) 07:09, 26 December 2018 (CST)