User talk:Xymph

Archived discussions
2016

External link editing
Hi Xymph. I'm not sure about deleting items from Sources sections. That probably means it was the source of some content in that article specifically (e.g. the readme, or some trivia in the DW reviews). Ryan W (usually gone) 13:31, 24 February 2017 (CST)


 * Well, you should know, as you created the initial revision which already contained the link. :) Upon further reading, it is related to the "Inspiration and development" mention about "Nathrath's Fab Four". That just wasn't clear, as other Sources entries often have more info like this or use a reference to link it to the place where it's used. With neither, I got the impression the idgames link was used just like here, and that seemed redundant as indicated by my removal reason.
 * I'll fix them up, including the mix-ups where M5 (in the other collection) didn't get a sources link – nor did M2 – while M6 (not in it) did. ;) --Xymph (talk) 15:00, 24 February 2017 (CST)


 * Thank you. FTR I was prepared to hear reasoning for not listing certain sources, which has happened in the past.  And I should know, but I don't, because it was such a disorganized period that I'm thankful to have remembered the difference between square and curly brackets each time.  :>


 * The general point about is a good one (but a completely separate project, I think!).  Long story short, the situation you describe does confuse people, but it is a huge and probably non-bottable task to fix it.  I personally would be OK with uniform use of inline references if everyone had put some thought into the question and decided they were good.  I don't believe that is currently the case.    Ryan W (usually gone) 15:28, 24 February 2017 (CST)

Fixes
Thank you for all your fixes. Ducon (talk) 07:38, 21 May 2017 (CDT)
 * No problem, we all build on previous contributions. If you're going to do more map series, I can help creating the map skeletons bot-wise. It improves consistency and leaves you more time to write information and descriptions, and add screenshots (which are still needed for a gazillion map pages). --Xymph (talk) 07:44, 21 May 2017 (CDT)
 * I’m taking screenshots for the top 100 WADs and for the cacowarded maps, thus, I create pages for such maps. Ducon (talk) 07:49, 21 May 2017 (CDT)
 * I'm working on the same, so we can collaborate, like we did implicitly yesterday :) and previously on IC2005. After you create an overview page like Vae Victus 2, you could leave it to me to create the individual map pages with all the basic info that the bot scripts can add. Then you resume with the screens and other info. Of course I'm not always available to do that immediately, so it takes perhaps a little planning ahead (and occasionally patience) for this to work for you too. --Xymph (talk) 08:12, 21 May 2017 (CDT)
 * Great. I’m going to play Classic episode 2 (https://www.doomworld.com/idgames/levels/doom/Ports/a-c/class_e2). Ducon (talk) 13:59, 21 May 2017 (CDT)
 * Fine, so create the WAD overview page with sufficient info, then I'll follow with the map pages. And for future WADs you gonna play, creating the overview page will again be my cue – no further posts here necessary. Have fun! --Xymph (talk) 14:25, 21 May 2017 (CDT)

Woodexial's Secrets
I'm using GZDoom's automap instead of an editor and it shows only 7 secrets instead of 8. Was Sector 188 is the one with a C cacodemon and a backpack in a pit? --ZeroTheEro (talk) 19:20, 22 May 2017 (CDT)
 * Yes, 188 is the sector just before that pit. I did only a fly-thru, clipping on/off as needed to get to places and trigger the secrets, but the automap in ZDoom does show x/8 secrets too. Not sure how to make it reveal the sector number, and why you got most of them differently than they are stored in the map. --Xymph (talk) 02:31, 23 May 2017 (CDT)
 * To put it simply, the sector numbers didn't match the secret because I have no idea how to check it. I'm just having guesswork on the sector number since I list the secrets by gradual progression. Please correct me if I am wrong. --ZeroTheEro (talk) 04:28, 23 May 2017 (CDT)


 * I see. But in skeleton pages the secret sectors are simply in numeric order rather than play progression. I'd rather not be correcting them all the time though. ;) To match the numbers to sectors in the map, I use DeePsea: switch to sector mode with the mouse scroll-wheel, use the 'j' key to bring up the jump dialog, and enter the number to be taken to that highlighted sector. There may be other/better ways in other editors. --Xymph (talk) 04:44, 23 May 2017 (CDT)


 * Thanks, yo. --ZeroTheEro (talk) 13:23, 23 May 2017 (CDT)


 * Hi Xymph. Not to open another can of worms here  ;>  but you mean that bot-generated skeletons use numeric order.  IME human editors adding descriptions tend to reorder them along the base walkthrough route, in hopes that's clearer for the newbies.    Ryan W (usually gone) 11:30, 26 May 2017 (CDT)


 * That wording is indeed more accurate, and what I meant in general. Although it has also happened that a contributor added the sector numbers manually without descriptions, e.g. from DeePsea's search results for sector type, and those would also be in numeric order. I'd still call that skeleton lists, but once a contributor has added descriptions I'd no longer consider that section a skeleton anymore. And of course the order can then be any that is best suited for the pertaining map. --Xymph (talk) 13:11, 26 May 2017 (CDT)

XymphBot and skeletons (again)
Hi Xymph,

I'm just now beginning to examine some bot edits in more detail. It's still amazing how many features you managed to add in a short time. Anyone who has started a few dozen pages from scratch will appreciate the massive efficiency gain (with respect to both actual writing and interlarded decompression :D

I notice that you sometimes return to address an issue manually, e.g.:
 * 1) erasing a footnote
 * 2) removing a category
 * 3) adding a template that was already present
 * 4) transposing a heading with a paragraph

You've forgotten more about programming than I'll ever know :>  so perhaps you added validations to the code as needed, or the bot itself alerted you when existing text didn't match the standard form. But if that wasn't possible (because string parsing is very hard), I would like to help by finding any remaining similar issues. Is there a way to narrow down relevant edits of each type, e.g. by time stamps or canned edit summaries?

Yes, I've complained about gameplay articles attracting hoi polloi revisions because of the low barrier to entry, but those still deserve to be evaluated in context by a human.

Anyway, I hope that makes some sense, and thanks again for caring about our project. Ryan W (living fossil) 01:33, 12 July 2017 (CDT)


 * Thanks for the com(pli)ments. Also, I'm glad to see you becoming more active again. I don't follow "interlarded decompression" though, is that one of your Dr Who terms? ;)
 * Regarding manual touch-ups: yes sometimes the pattern matching is too tricky, but usually it's a trade-off between effort and gain.
 * 1. Footnotes between/below Things tables are very rare so I simply didn't bother trying to code for them, as it would be harder than manually restoring them. 2. On most map pages the Statistics section is followed by other (stub-)sections. The MW API – and thus dmmpstBot.php – allows updating a single (sub)section, but category lines don't form their own section and are simply included in the last one. So in case of those Heretic pages the cats got stripped off by the bot edit because I didn't code for the possibility of any being in the Statistics section, as pages usually have External links headers and such in between. Additionally, conventionsBot.php warns me of category lines embedded somewhere else than at the end of pages. Lastly, the bot removals of the unnecessary "Levels by name" cats were actually convenient, as those could be dropped from all Heretic pages, while the firemace cat only needed to be re-added on 20 of them. 3. That happened because mapviewBot.php didn't take the empty line before the existing mapspots template into account. It's too long ago to remember exactly, but glancing over the code it could be that mapviewBot.php handles it correctly now. I must admit that sometimes I reviewed the interactive diffs too hastily and submitted edits needing manual clean-up which I then addressed in the pertaining script. 4. This page stood out because of the non-standard walkthrough formatting, but I didn't bother cleaning it up beyond the header move so that skeletonBot.php and mapviewBot.php wouldn't complain.
 * For cases 1 and 2, since dmmpstBot.php shows me the diff before actually submitting the edit, I know where to return to and manually restore things, and I also manually review all pages in a series afterwards. Thus I'm pretty sure I didn't leave (m)any loose ends of issues caused by bot edits, but I may have slipped up on occasion. I can't think of a pattern by which to locate them though.
 * For me the fun is both in programming and thus helping myself/others avoid tedious jobs (that were evidently hard to execute consistently and completely), and improving DW's content and presentation of factual information. After a year and a half, I do consider it my project too. :)
 * Some statistics: in my archive are 166 .ini files (including three for all single-level Doom, Doom II and Heretic PWADs) covering 3499 map pages.
 * So, much thanks for the offer of help and sorry for not being able to point you directly in the desired, uhh, direction. It you're still undeterred after all the above, here's how you (or anyone) can help. However, currently I'm immersed in INFO.c tools programming and subsequently transferring their output to the wiki, so combined with real-life interference it may take a while before I can return to bot-based map page updates. --Xymph (talk) 07:32, 12 July 2017 (CDT)


 * Thank you; that is quite informative. There's certainly no rush, and I hope the bold text didn't imply otherwise.  (I did reread the discussions first, so I'm glad I hadn't overlooked something obvious in your updates.)


 * I'm learning that "active" is a floating-point quantity :>  so e.g. I will hesitate to join a project systematically playtesting megawads.  That said, I already had your list bookmarked because some of those articles appeared hastily during an RFC, which I suspected would cause trouble for somebody.  Now that I know it's XymphBot, its previously successful PWADs can serve as examples to follow.


 * Re "decompression" &mdash; AFAICT it's a common experience that long, tedious jobs may be followed by an unrelated mindless task to clear one's head, before one can return to a different demanding activity. It's hard to plan for, especially with real-life concerns present, so it's another reason bots are great.  :D    Ryan W (living fossil) 16:44, 12 July 2017 (CDT)

&larr;&larr;&larr;

Hi again. I've changed (hopefully improved) Template:Map skel slightly; does that affect XymphBot at all? I dimly remembered that it did, but now I can't find the original discussion (oops...), only several cousins: If I'm barking up the wrong tree again, just say so. :>  I did not have to change any headings to match our newest articles, which presumably means you and the other "map regulars" had already applied your usual thoroughness. In theory there might be additional tricks available, e.g. reading the slot automatically from the page title, but I would like to keep the markup simple because non-regulars sometimes create map pages (especially after Cacowards). Ryan W (living fossil) 00:31, 17 July 2017 (CDT)
 * A tiny IRC excerpt, very inconclusive but did mention the garbled introduction (now fixed)
 * This thread in which you were using the template as a reference, but for manual edits
 * Parameters to templates within the template, which sounds resolved (and AFAICT wouldn't interfere either way, unless we tried to pass default parameters)


 * You may be remembering the remark at the end of Template:Things. But DMMPST includes its own copies of the templates it uses, so there is no direct consequence. However for consistency, I similarly dropped the ftp: stub and added the Adding custom music wikilink; and adopted the dummy NavboxTemplateGoesHere line for which I had to update navboxBot.php as well. Thanks for the – indeed – improvements. --Xymph (talk) 08:28, 17 July 2017 (CDT)

"Revert misguided spots formatting"
Hi Xymph. This function appears not to detect multiple letters within the same parentheses. For example, here the text was unchanged. Ryan W (living fossil) 06:52, 16 August 2017 (CDT)


 * Good catch, thanks. Fixed for all Ultimate Doom maps, found none in the Doom2 map pages. Let me know if you find some in another map series. --Xymph (talk) 08:20, 16 August 2017 (CDT)

November featured article thread
Which "both articles", RyanW?  &mdash; Hmm, probably just being sloppy with verb tenses again. "Both articles will have been featured when all is said and done, so there's no harm done to the original goal of showcasing the good work in both". Does that make sense? Ryan W (living fossil) 17:49, 10 November 2017 (CST)
 * I don't dare to say 'no' anymore at this point. :-P Yes --Xymph (talk) 02:38, 11 November 2017 (CST)

Report on Wikipedia linking?
Hi Xymph,

Ideas for future scripts, anyone?  &mdash; okay, what about a list of all links to Wikipedia meeting certain criteria?

There was a brief IRC discussion about knock-on effects of Wikipedia deletions. It can be argued that any game-related article aside from the main AAA releases is at risk. I would like to remediate this somehow, and as Quasar pointed out, that is much easier before the article is deleted. :7

It is of course your decision whether this sounds interesting (and whether you have time!). I have vague ideas about implementation details, which can and should be finalized through general discussion. I am asking you first, however, because if we don't have a bot then my initial proposal will change. :D

Thanks for reading, as always. Ryan W (living fossil) 17:03, 18 March 2018 (CDT)


 * Time, indeed, is of the essence :) and now that my .exe/iwad project is done, I wanted to turn to other things on my to-do list, on DoomWiki and elsewhere. But this sounds like a multi-day project rather than a multi-month one, so it's doable. Can you elaborate on what the script should do, across which pages against what criteria, producing what kind of output? --Xymph (talk) 06:51, 19 March 2018 (CDT)


 * I can respond, but it creates no obligation on your part. :>


 * Output would be a one-to-many mapping like this:

! Wikipedia page !! Linked from
 * || Aspect ratioCategory:Node buildersChristoph Oelckers (Graf Zahl)Doom 3
 * || Aspect ratioFile:Familyguy.jpgShareware
 * }
 * || Aspect ratioFile:Familyguy.jpgShareware
 * }


 * On any page where "encyclopedic" content is expected, sources might be added, so the territory would be namespaces 0, 4, 6, 10, 12, 14 except


 * Criteria provide the most room for debate. If I did this manually, I would want two things:
 * The ability to identify every link, even if added indirectly (interwiki, transclusion, etc).
 * A conservative first pass to filter out less likely candidate links. Total volume is a bit too large to digest otherwise.
 * I assume #1 can be addressed by requesting pages in HTML format. #2 is far more heuristic; options include
 * For each Wikipedia article, read Wikipedia's category system to identify related areas (gaming, computing, scripture, ...).
 * List only links formatted locally as references (citation tags, specific templates, ...).
 * List only links under whitelisted section titles.
 * I would choose the last approach. The second would be very incomplete (indeed one motivation is to clean up sloppy citations from a decade ago).  The first would be beating the bot's head against the scale and nonlinearity of Wikipedia's categories, at least until it got blocked.  I did manage to compile promising titles already:
 * In this context, a "section" ideally comprises all text until the next header at the same level. The next header at any level is likely 99% accurate however.
 * In this context, a "section" ideally comprises all text until the next header at the same level. The next header at any level is likely 99% accurate however.
 * In this context, a "section" ideally comprises all text until the next header at the same level. The next header at any level is likely 99% accurate however.
 * In this context, a "section" ideally comprises all text until the next header at the same level. The next header at any level is likely 99% accurate however.
 * In this context, a "section" ideally comprises all text until the next header at the same level. The next header at any level is likely 99% accurate however.
 * In this context, a "section" ideally comprises all text until the next header at the same level. The next header at any level is likely 99% accurate however.
 * In this context, a "section" ideally comprises all text until the next header at the same level. The next header at any level is likely 99% accurate however.
 * In this context, a "section" ideally comprises all text until the next header at the same level. The next header at any level is likely 99% accurate however.
 * In this context, a "section" ideally comprises all text until the next header at the same level. The next header at any level is likely 99% accurate however.
 * In this context, a "section" ideally comprises all text until the next header at the same level. The next header at any level is likely 99% accurate however.
 * In this context, a "section" ideally comprises all text until the next header at the same level. The next header at any level is likely 99% accurate however.
 * In this context, a "section" ideally comprises all text until the next header at the same level. The next header at any level is likely 99% accurate however.
 * In this context, a "section" ideally comprises all text until the next header at the same level. The next header at any level is likely 99% accurate however.
 * In this context, a "section" ideally comprises all text until the next header at the same level. The next header at any level is likely 99% accurate however.
 * In this context, a "section" ideally comprises all text until the next header at the same level. The next header at any level is likely 99% accurate however.
 * In this context, a "section" ideally comprises all text until the next header at the same level. The next header at any level is likely 99% accurate however.
 * In this context, a "section" ideally comprises all text until the next header at the same level. The next header at any level is likely 99% accurate however.
 * In this context, a "section" ideally comprises all text until the next header at the same level. The next header at any level is likely 99% accurate however.
 * In this context, a "section" ideally comprises all text until the next header at the same level. The next header at any level is likely 99% accurate however.
 * In this context, a "section" ideally comprises all text until the next header at the same level. The next header at any level is likely 99% accurate however.
 * In this context, a "section" ideally comprises all text until the next header at the same level. The next header at any level is likely 99% accurate however.
 * In this context, a "section" ideally comprises all text until the next header at the same level. The next header at any level is likely 99% accurate however.
 * In this context, a "section" ideally comprises all text until the next header at the same level. The next header at any level is likely 99% accurate however.
 * In this context, a "section" ideally comprises all text until the next header at the same level. The next header at any level is likely 99% accurate however.
 * In this context, a "section" ideally comprises all text until the next header at the same level. The next header at any level is likely 99% accurate however.
 * In this context, a "section" ideally comprises all text until the next header at the same level. The next header at any level is likely 99% accurate however.
 * In this context, a "section" ideally comprises all text until the next header at the same level. The next header at any level is likely 99% accurate however.
 * In this context, a "section" ideally comprises all text until the next header at the same level. The next header at any level is likely 99% accurate however.
 * In this context, a "section" ideally comprises all text until the next header at the same level. The next header at any level is likely 99% accurate however.
 * In this context, a "section" ideally comprises all text until the next header at the same level. The next header at any level is likely 99% accurate however.
 * In this context, a "section" ideally comprises all text until the next header at the same level. The next header at any level is likely 99% accurate however.
 * In this context, a "section" ideally comprises all text until the next header at the same level. The next header at any level is likely 99% accurate however.
 * In this context, a "section" ideally comprises all text until the next header at the same level. The next header at any level is likely 99% accurate however.
 * In this context, a "section" ideally comprises all text until the next header at the same level. The next header at any level is likely 99% accurate however.
 * In this context, a "section" ideally comprises all text until the next header at the same level. The next header at any level is likely 99% accurate however.
 * In this context, a "section" ideally comprises all text until the next header at the same level. The next header at any level is likely 99% accurate however.
 * In this context, a "section" ideally comprises all text until the next header at the same level. The next header at any level is likely 99% accurate however.


 * This post represents only my personal hypotheses, subject to later consensus-building and any needed compromises regarding what is or is not feasible to code.   Ryan W (living fossil) 17:34, 19 March 2018 (CDT)


 * Further idea: this probably needs a throttle to avoid mass image uploads, which link pages such as through templates, and in practice have section structures varying by user.  If the search returns more than 25 results (say) for the same external link in the same namespace, just write  or something.  If it's clearly an unusual case, I'll deal with it manually.    Ryan W (living fossil) 23:18, 22 March 2018 (CDT)


 * The past two days of work on the script resulted in a fair amount of progress. The last run went over 5 of the 6 namespaces, excluding main – I don't have an idea of the link volume in the latter yet. Because runs over that many pages take a long time, the script logs its progress like this:

Attempting to log in... Success. collecting Wikipedia namespaces: 34 processing 'Doom Wiki'... processed:   43 pages processing 'File'... 100 200 300 400 500 processing 'File' from File:32in24_MAP09_hilltop.png... 100 200 300 400 500 processing 'File' from File:AnimFace.gif... 100 200 300 400 500 processing 'File' from File:Cchest4_MAP13.png... 100 200 300 400 500 processing 'File' from File:CommunityChest-map12-mud.png... 100 200 300 400 500 processing 'File' from File:Congestion1024-map19-end.png... 100 200 300 400 500 processing 'File' from File:DTWID-LE_E2M8_map.png... 100 200 300 400 500 processing 'File' from File:Doom3_weapons_07.jpg... 100 200 300 400 500 processing 'File' from File:Doom_v0.3_thestore.png... 100 200 300 400 500 processing 'File' from File:E3M4_map.png... 100 200 300 400 500 processing 'File' from File:Eternity-e2m1-s.png... 100 200 300 400 500 processing 'File' from File:H2H-Xmas_MAP31_map.png... 100 200 300 400 500 processing 'File' from File:Heretic-e3m4-snail.png... 100 200 300 400 500 processing 'File' from File:Interception_MAP04_map.png... 100 200 300 400 500 processing 'File' from File:Kansams_Trial_E2M7_map.png... 100 200 300 400 500 processing 'File' from File:MM_MAP02_Secret.png... 100 200 300 400 500 processing 'File' from File:MementoMori2-map04-down.png... 100 200 300 400 500 processing 'File' from File:NDCP-map23-torches.png... 100 200 300 400 500 processing 'File' from File:Origwad_2.png... 100 200 300 400 500 processing 'File' from File:Psx-command-control-start.png... 100 200 300 400 500 processing 'File' from File:Requiem-map27-end.png... 100 200 300 400 500 processing 'File' from File:Scythe2_MAP03_map.png... 100 200 300 400 500 processing 'File' from File:Stardate20X6.png... 100 200 300 400 500 processing 'File' from File:TheEvilUnleashed-e3m5-cage.png... 100 200 300 400 500 processing 'File' from File:Vile_Flesh_MAP11_map.png... 100 200 processed: 12260 pages processing 'Template'... 100 200 300 400 500 processing 'Template' from Template:MAYhem_1500_21-30... 100 200 300 400 processed:  940 pages processing 'Help'... processed:    8 pages processing 'Category'... 100 200 300 400 500 processing 'Category' from Category:Easter_eggs... 100 200 300 400 500 processing 'Category' from Category:MrGuyTodd_levels... 100 200 300 400 processed: 1491 pages


 * Some info about what listWikiLinks.php does:
 * init namespace lists and patterns; at first I used an API call to get Wikipedia's namespace list, then hard-coded it to save a call; in the exception list, and  were also needed
 * the regex patterns are: , , and ''
 * from all selected namespaces, lists of pages are queried at 500 per batch and each page is then fetched and searched for the link patterns; this is the most time-consuming part, so a progress marker is logged every 100 pages
 * all data is collected in an array (keyed by Wikipedia path) of arrays (with DoomWiki paths), which are all sorted with flags SORT_NATURAL and SORT_FLAG_CASE
 * the tables are generated by traversing the data array for each Wikipedia namespace; for entries with more than 20 DW links, a more value is appended
 * Filtering of WP links is not implemented yet, I'm not sure that's wise. For one thing, quite a few WP links occur in the intro (section 0) of a page, so that should be in your list too. For another, perhaps the link volume isn't overwhelming, I am now running the script on main space only to find out.
 * About avoiding mass uploaded images, the script could skip filenames ending in "_map.png" and "_title.png" to save processing time indeed. Or what (else) did you have in mind?
 * Do you now want to launch a general discussion before I continue? --Xymph (talk) 05:41, 27 March 2018 (CDT)


 * I added the main space data. Okay, yes, it's voluminous. :) --Xymph (talk) 06:51, 27 March 2018 (CDT)


 * There's so much progress here, people can't help being inspired to further suggestions! :>   I hope you won't interpret it as criticism if I make suggestions before testing any bibliographic edits &mdash; it's a big job and I don't want people to have to revert me later.


 * Regarding mass uploaded images, you have apparently handled this, by counting links in template space once per template instead of once per transclusion.


 * You're probably correct to parse wikitext instead of HTML; it provides more fine-grained results. Note that:
 * also points to Wikipedia.
 * I'm a regex newbie, but the above doesn't seem to include MediaWiki's case insensitivity on the first character. E.g. {{c| {{wp }} and {{c| {{Wp }} are the same.  {{wp|fair use}} and  are the same.  Further, interwiki prefixes are completely case-insensitive.
 * Similarly, {{wp|Blake Stone: Aliens of Gold}} and {{wp|Blake_Stone:_Aliens_of_Gold}} are the same.
 * I endorse Gez's idea of a third column: direct links are far more likely to be unmarked sources, at least on the articles I've seen.


 * If the consensus is not to filter mainspace links (time for that general discussion :D )  then there will be many more ideas and many more possible mass edits to implement them.  I personally still intend to focus on the task in the OP, and would heuristically skip entries clearly outside it ({{wp|Blu-Ray}}, {{wp|Lava}}).  If no one else made use of the remainder, then anything I missed would simply remand to a more general link rot project in the far future.


 * Hope that makes some sense, and thanks as always for your patience.   Ryan W (living fossil) 19:56, 27 March 2018 (CDT)

Hey, a suggestion: would it be possible to have a third column listing the link types used? More generally, a bot capable of looking through the wiki pages to find any and all instance of direct links to domains that have interwiki entries would be useful for maintenance. --Gez (talk) 06:25, 27 March 2018 (CDT)


 * Sure, you mean whether a DW page uses wp template, interwiki or https link for a WP link? How should the third column look like, for example? And what if more than one method is used for the same WP link on a page?
 * The general suggestion would be a separate bot script. --Xymph (talk) 06:42, 27 March 2018 (CDT)
 * I figure the third column could just contain a simple marker, like D for direct link, I for interwiki link, and T and template link. If more than one method is used, then you get more than one marker. If there's a direct link and a template link, then it'd show DT for example. --Gez (talk) 07:03, 27 March 2018 (CDT)

&larr;&larr;&larr;

First, to clear up a misconception, bot scripts aren't crawling the site's HTML pages. They communicate via the MW API and thus, when working with page content, they indeed retrieve and store wikitext, just like a human using the edit form. And in the Template namespace, fetching pages therefore gets the template wikitext; nothing transcluded. I wouldn't even want to process HTML for page content, that would be error-prone and unnecessarily complicated.

Thanks for the feedback, the interwiki pattern was updated to '{{c| \[\[Wiki(?:pedia)?:(.+?)(?:\|.+?)?\]\] }}'. All patterns are matched using the i modifier, so case-insensitively. But the WP paths found are case-sensitive, and stored that way in the data structure. The script doesn't know which one of those is the canonical WP page. To collate WP paths to some degree, the script now converts all underscores to spaces, to deal with Blake_Stone-type situations. But I'm not sure no WP path legitimately includes an underscore, and whether that would result in a broken WP link in the first column of the tables.

Also, paths starting with a letter always use a capital initial in WP's URL, so the first letter is now uppercased to collate [Ff]air use-type situations. But I can't safely collate "Alice in Chains" and "Alice In Chains", so in those situations there are still multiple entries. Further, in File space, pages ending in " map.png" were skipped, saving about half an hour of processing time.

Lastly, the third column is present, showing very few direct links, most of which I fixed already. No DW pages mixed link types for the same WP path, either. After the following log:

Attempting to log in... Success. collecting Wikipedia namespaces: 34 processing 'main'... 100 200 300 400 500 processing 'main' from main:Counterattack... 100 200 300 400 500 processing 'main' from main:E1M3:_Toxin_Refinery_(Phobos_Revisited)... 100 200 300 400 500 processing 'main' from main:E4M9:_Bean_Oversight_(Doom_the_Way_id_Did:_The_Lost_Episodes)... 100 200 300 400 500 processing 'main' from main:IRC... 100 200 300 400 500 processing 'main' from main:MAP02:_Security_(H2H-Xmas)... 100 200 300 400 500 processing 'main' from main:MAP06:_Colossus_(The_Rebirth)... 100 200 300 400 500 processing 'main' from main:MAP10:_Eureka_Signs_(BTSX-E2)... 100 200 300 400 500 processing 'main' from main:MAP15:_Descent_to_Core_(Speed_of_Doom)... 100 200 300 400 500 processing 'main' from main:MAP21:_Aberinkula_(MAYhem_2048)... 100 200 300 400 500 processing 'main' from main:MAP27:_Emerald_Spire_(Sunlust)... 100 200 300 400 500 processing 'main' from main:MAP34:_Abstraction_(Claustrophobia_1024_2)... 100 200 300 400 500 processing 'main' from main:Psychophobia... 100 200 300 400 500 processing 'main' from main:The_Doom_Movement_Bible... 100 200 300 processed: 6874 pages processing 'Doom Wiki'... processed:   43 pages processing 'File'... 100 200 300 processing 'File' from File:32in24_MAP09_hilltop.png... 100 200 300 processing 'File' from File:AnimFace.gif... 100 200 300 processing 'File' from File:Cchest4_MAP13.png... 100 200 300 processing 'File' from File:CommunityChest-map12-mud.png... 100 200 300 400 500 processing 'File' from File:Congestion1024-map19-end.png... 100 200 300 processing 'File' from File:DTWID-LE_E2M8_map.png... 100 200 processing 'File' from File:Doom3_weapons_07.jpg... 100 200 300 processing 'File' from File:Doom_v0.3_thestore.png... 100 200 300 400 processing 'File' from File:E3M4_map.png... 100 200 300 processing 'File' from File:Eternity-e2m1-s.png... 100 200 300 processing 'File' from File:H2H-Xmas_MAP31_map.png... 100 200 processing 'File' from File:Heretic-e3m4-snail.png... 100 200 300 processing 'File' from File:Interception_MAP04_map.png... 100 200 300 processing 'File' from File:Kansams_Trial_E2M7_map.png... 100 200 processing 'File' from File:MM_MAP02_Secret.png... 100 200 300 400 processing 'File' from File:MementoMori2-map04-down.png... 100 200 300 400 processing 'File' from File:NDCP-map23-torches.png... 100 200 processing 'File' from File:Origwad_2.png... 100 200 300 processing 'File' from File:Psx-command-control-start.png... 100 200 300 400 processing 'File' from File:Requiem-map27-end.png... 100 200 300 processing 'File' from File:Scythe2_MAP03_map.png... 100 200 300 processing 'File' from File:Stardate20X6.png... 100 200 processing 'File' from File:TheEvilUnleashed-e3m5-cage.png... 100 200 300 processing 'File' from File:Vile_Flesh_MAP11_map.png... 100 processed: 12260 pages skipped:   3465 map pages processing 'Template'... 100 200 300 400 500 processing 'Template' from Template:MAYhem_1500_21-30... 100 200 300 400 processed:  940 pages processing 'Help'... processed:    8 pages processing 'Category'... 100 200 300 400 500 processing 'Category' from Category:Easter_eggs... 100 200 300 400 500 processing 'Category' from Category:MrGuyTodd_levels... 100 200 300 400 processed: 1491 pages

the new tables for all DW namespaces are online. --Xymph (talk) 12:58, 28 March 2018 (CDT)


 * Thank you! I'm slightly out of breath, watching it grow.  :>   Maybe I spend too much time thinking about the global topology of the wiki, but I sense this could lead in several useful directions, not all to do with modifying links.


 * They communicate via the MW API  &mdash; I did know that, and in that very documentation I kept seeing mentions of "parsed" text.  I hoped there was a feature which could make your task easier by reducing the number of different cases to search for, i.e. converting all links into direct links.  Of course I accept your word that wikitext is easier; as I said, in hindsight that should be easier for everyone else also.


 * But I'm not sure no WP path legitimately includes an underscore  &mdash; I'm reasonably sure, based on this experiment, that equivalence between spaces and underscores is implemented at a fairly low level.  I can even type the spaces into my URL bar and still get where I intended: {{c| https://en.wikipedia.org/wiki/Death of Michael Jackson }}   Does that work in general?  Nope: {{c| https://www.youtube.com/watch?v=VxqaI c9j g }}


 * But I can't safely collate "Alice in Chains" and "Alice In Chains"  &mdash; True enough.  So far none of these points is enormously important  :D  however:


 * Thanks for the feedback  &mdash; It's the least I can do.  Something is still wrong, because the table omits the link that started my train of thought: this one.


 * Ryan W (living fossil) 18:12, 28 March 2018 (CDT)


 * Ow, that's because Template:Wikipedia wasn't mentioned to far, and I didn't realize it either. It's now covered by regex '{{c| {{wikipedia(?:\|(?:title=)?(.+?)(?:\|.+?)?)?}} }}' and type S (Source). Besides with a positional or named parameter, it can be used parameterless which presented a small problem, but that's now solved. See the updated list. --Xymph (talk) 10:58, 29 March 2018 (CDT)


 * D'oh! I saw "exception list" above, and somehow assumed it was part of the regex even though I couldn't see it.  (I did page through templates at some point to check if there were any more.  There are not.)


 * I think that's all of my questions, so we'll see what others have to say. :>    Ryan W (living fossil) 16:36, 29 March 2018 (CDT)

Interwiki links
Copied from Wikipedia topic: More generally, a bot capable of looking through the wiki pages to find any and all instance of direct links to domains that have interwiki entries would be useful for maintenance. --Gez (talk) 06:25, 27 March 2018 (CDT)


 * To verify, this script needs to search pages for the URLs in the Interwiki table (up to the $1 parameter), and show which interwiki prefix should be used instead? --Xymph (talk) 12:26, 27 March 2018 (CDT)
 * Yeah, I figure that should be enough. The only potential subtlety would be to watch out for http/https, i.e. there could be an http direct link to a site that has https in the interwiki table (or vice-versa). --Gez (talk) 13:42, 27 March 2018 (CDT)


 * If we're talking about maintenance, the reverse of this (an analogue of Special:LinkSearch for interwikis only) would occasionally be useful as well, e.g. when a site goes down permanently. Again a low priority, unless someone finds a live example.    Ryan W (living fossil) 20:00, 27 March 2018 (CDT)


 * Not sure what info a "reverse" script (or addition to the current script) should generate – if you'd like something done, please elaborate.
 * The current script was pretty easy to derive from the previous one, and produces the latest table after this log:

Attempting to log in... Success. collecting unique Interwiki list: 135 processing 'main'... 100 200 300 400 500 processing 'main' from main:Counterattack... 100 200 300 400 500 processing 'main' from main:E1M3:_Toxin_Refinery_(Phobos_Revisited)... 100 200 300 400 500 processing 'main' from main:E4M9:_Bean_Oversight_(Doom_the_Way_id_Did:_The_Lost_Episodes)... 100 200 300 400 500 processing 'main' from main:IRC... 100 200 300 400 500 processing 'main' from main:MAP02:_Security_(H2H-Xmas)... 100 200 300 400 500 processing 'main' from main:MAP06:_Colossus_(The_Rebirth)... 100 200 300 400 500 processing 'main' from main:MAP10:_Eureka_Signs_(BTSX-E2)... 100 200 300 400 500 processing 'main' from main:MAP15:_Descent_to_Core_(Speed_of_Doom)... 100 200 300 400 500 processing 'main' from main:MAP21:_Aberinkula_(MAYhem_2048)... 100 200 300 400 500 processing 'main' from main:MAP27:_Emerald_Spire_(Sunlust)... 100 200 300 400 500 processing 'main' from main:MAP34:_Abstraction_(Claustrophobia_1024_2)... 100 200 300 400 500 processing 'main' from main:Psychophobia... 100 200 300 400 500 processing 'main' from main:The_Doom_Movement_Bible... 100 200 300 processed: 6874 pages processing 'Doom Wiki'... processed:   43 pages processing 'File'... 100 200 300 400 500 processing 'File' from File:32in24_MAP09_hilltop.png... 100 200 300 400 500 processing 'File' from File:AnimFace.gif... 100 200 300 400 500 processing 'File' from File:Cchest4_MAP13.png... 100 200 300 400 500 processing 'File' from File:CommunityChest-map12-mud.png... 100 200 300 400 500 processing 'File' from File:Congestion1024-map19-end.png... 100 200 300 400 500 processing 'File' from File:DTWID-LE_E2M8_map.png... 100 200 300 400 500 processing 'File' from File:Doom3_weapons_07.jpg... 100 200 300 400 500 processing 'File' from File:Doom_v0.3_thestore.png... 100 200 300 400 500 processing 'File' from File:E3M4_map.png... 100 200 300 400 500 processing 'File' from File:Eternity-e2m1-s.png... 100 200 300 400 500 processing 'File' from File:H2H-Xmas_MAP31_map.png... 100 200 300 400 500 processing 'File' from File:Heretic-e3m4-secret.png... 100 200 300 400 500 processing 'File' from File:Interception_MAP03_map.png... 100 200 300 400 500 processing 'File' from File:Kansams_Trial_E2M6_map.png... 100 200 300 400 500 processing 'File' from File:MM_MAP01_map.png... 100 200 300 400 500 processing 'File' from File:MementoMori2-map03-plus.png... 100 200 300 400 500 processing 'File' from File:NDCP-map19-water.png... 100 200 300 400 500 processing 'File' from File:Odessa_3.png... 100 200 300 400 500 processing 'File' from File:Psx-central-processing-02.png... 100 200 300 400 500 processing 'File' from File:Requiem-map22-cacos.png... 100 200 300 400 500 processing 'File' from File:Scythe2-map28-afrit.png... 100 200 300 400 500 processing 'File' from File:Speed_of_Doom_MAP24_map.png... 100 200 300 400 500 processing 'File' from File:TheEvilUnleashed-e3m1-chasm.png... 100 200 300 400 500 processing 'File' from File:Vanguard_MAP06_map.png... 100 200 processed: 12292 pages processing 'Template'... 100 200 300 400 500 processing 'Template' from Template:MAYhem_1500_21-30... 100 200 300 400 processed:  940 pages processing 'Help'... processed:    8 pages processing 'Category'... 100 200 300 400 500 processing 'Category' from Category:Easter_eggs... 100 200 300 400 500 processing 'Category' from Category:MrGuyTodd_levels... 100 200 300 400 processed: 1491 pages


 * Some info about what listInterLinks.php does:
 * init namespace lists and patterns; DW paths in the exception list are the previous five, plus ,  and ''
 * the interwiki list is fetched and the URLs are converted to regexes, e.g. the first one becomes ""; also, some interwikis/URLs occur in the list twice, so it is deduped from 157 to 135
 * pages are fetched from namespaces analog to listWikilinks and matches for each regex are stored in an array (keyed by interwiki prefix) of arrays (keyed by DW path)
 * the arrays are then sorted by those keys – but the direct links remain stored in the order they were encountered in that page – and the table is generated
 * On the first full pass, file pages ending in " map.png" weren't skipped yet but that's enabled now to save time in future runs. If more/different features are needed, let me know. For one thing, the regex was a bit limited, I've already expanded it to "". --Xymph (talk) 04:50, 3 April 2018 (CDT)

Templatable links
Moved from Wikipedia topic: And while I'm on that general topic, a third script could search for direct links that could be templated -- I'm thinking about stuff like direct links to forums or archives, notably. There's no hurry, though. --Gez (talk) 07:03, 27 March 2018 (CDT)


 * Okay, one thing are a time. :) --Xymph (talk) 12:26, 27 March 2018 (CDT)


 * This one should be uncontroversial, but "no hurry" is an understatement. :>   IMO a bot performing style edits would have to sweep at regular intervals, and I don't assume Xymph or anyone else can commit to that (wishful thinking solution: someone in the Doom community figures out how to host bots as autonomous processes, like the WMF does).    Ryan W (living fossil) 17:04, 27 March 2018 (CDT)


 * Actually, running scripts regularly and posting their output to list pages automatically would be trivial for me, and less so for anyone else. ;-) But multiple admins/editors are monitoring Recent Changes already and when incorrect styles and links are fixed right away, regular runs won't even be necessary. An incidental run every 4-6 months or so would be enough to catch anything falling through those cracks.


 * The new script was really easy to derive from the previous one, and produces the latest table after this log:

Attempting to log in... Success. collecting external templates list: 20 processing 'main'... 100 200 300 400 500 processing 'main' from main:Cory_Scott_(NiTROACTiVE)... 100 200 300 400 500 processing 'main' from main:E1M3:_The_Gatehouse_(Heretic)... 100 200 300 400 500 processing 'main' from main:E4M5:_The_Witching_Hour_(Crusades)... 100 200 300 400 500 processing 'main' from main:Hub_4:_Desolate_Garden... 100 200 300 400 500 processing 'main' from main:MAP02:_Nullth_Precinct_(Community_Chest)... 100 200 300 400 500 processing 'main' from main:MAP05:_Wasted!_(NDCP2)... 100 200 300 400 500 processing 'main' from main:MAP09_(2_Sectors)... 100 200 300 400 500 processing 'main' from main:MAP14:_Vile_Gear_(Swift_Death)... 100 200 300 400 500 processing 'main' from main:MAP20:_Speedtraps_for_the_Bee_Kingdom_(BTSX-E2)... 100 200 300 400 500 processing 'main' from main:MAP26:_Teleport_Control_(Daedalus:_Alien_Defense)... 100 200 300 400 500 processing 'main' from main:MAP32:_Ultimative_Geheimnis_(Whispers_of_Satan)... 100 200 300 400 500 processing 'main' from main:Porkalator... 100 200 300 400 500 processing 'main' from main:Terminatrix... 100 200 300 400 processed: 6916 pages processing 'Doom Wiki'... processed:   43 pages processing 'File'... 100 200 300 400 500 processing 'File' from File:32in24_MAP09_hilltop.png... 100 200 300 400 500 processing 'File' from File:AnimFace.gif... 100 200 300 400 500 processing 'File' from File:Cchest4_MAP12_Deathmatch.png... 100 200 300 400 500 processing 'File' from File:CommunityChest-map12-end.png... 100 200 300 400 500 processing 'File' from File:Congestion1024-map17-switches.png... 100 200 300 400 500 processing 'File' from File:DTWID-LE_E2M1_map.png... 100 200 300 400 500 processing 'File' from File:Doom3_backpack.png... 100 200 300 400 500 processing 'File' from File:Doom_pinky_2.jpg... 100 200 300 400 500 processing 'File' from File:E3M1_heretic.png... 100 200 300 400 500 processing 'File' from File:Eternally_Yours_MAP12_map.png... 100 200 300 400 500 processing 'File' from File:H2H-Xmas_MAP24_map.png... 100 200 300 400 500 processing 'File' from File:Heretic-e3m1-storage.png... 100 200 300 400 500 processing 'File' from File:Infinity_E1M8_Map.png... 100 200 300 400 500 processing 'File' from File:KansamsTrial-E2M7-start.png... 100 200 300 400 500 processing 'File' from File:MM8BDM_Gameplay_1.png... 100 200 300 400 500 processing 'File' from File:MementoMori2-map01-start.png... 100 200 300 400 500 processing 'File' from File:NDCP-map17-hex.png... 100 200 300 400 500 processing 'File' from File:Obituary_MAP04_map.png... 100 200 300 400 500 processing 'File' from File:Programador1_Strife.png... 100 200 300 400 500 processing 'File' from File:Requiem-map16-elemental.png... 100 200 300 400 500 processing 'File' from File:Scythe2-map13-center.png... 100 200 300 400 500 processing 'File' from File:SpeedOfDoom-map28-redskullkey.png... 100 200 300 400 500 processing 'File' from File:Talosian_Incident_MAP15_map.png... 100 200 300 400 500 processing 'File' from File:Vae_Victus_2_MAP02_map.png... 100 200 300 processed: 12341 pages processing 'Help'... processed:    8 pages processing 'Category'... 100 200 300 400 500 processing 'Category' from Category:Earth_(WAD)_levels... 100 200 300 400 500 processing 'Category' from Category:MrGlide_levels... 100 200 300 400 processed: 1493 pages


 * Some info about what listExternLinks.php does:
 * init namespace lists and patterns; pages in the exception list are just  and 
 * from External link templates the following were manually added to the patterns list (the others don't contain direct links):

$externlist['Competnftp'] = 'https?://(?:www\.)?doom\.com\.hr/temp/compet-n/'; $externlist['Competnmap'] = 'https?://(?:www\.)?doom\.com\.hr/index.php\?page=compet-n_database&cndb='; $externlist['Competnwad'] = 'https?://(?:www\.)?doom\.com\.hr/index.php\?page=compet-n_database&wad_id='; $externlist['Deb-pkg']   = 'https?://packages\.debian\.org/'; $externlist['Drdforums'] = 'https?://forum\.drdteam\.org/viewtopic.php\?'; $externlist['Dsda']      = 'https?://doomedsda\.us/wad'; $externlist['Dsdaftp']   = 'https?://doomedsda\.us/lmps/'; $externlist['Dwforums']  = 'https?://(?:www\.)?doomworld\.com/vb/thread/'; $externlist['Dwforumst'] = 'https?://(?:www\.)?doomworld\.com/vb/post/'; $externlist['Edgeeg']    = 'https?://edge\.sourceforge\.net/edit_guide/'; $externlist['Idftp']     = 'https?://mirrors\.syringanetworks\.net/idgames/'; $externlist['Idgames']   = 'https?://(?:www\.)?doomworld\.com/idgames/'; $externlist['Idgamesmm'] = ' ftp://ftp\.fu-berlin\.de/pc/msdos/games/idgames/ '; $externlist['Lc']        = 'https?://lccn\.loc\.gov/'; $externlist['Legacywiki'] = 'https?://legacywiki\.net/index\.php/'; $externlist['Realm667']  = 'https?://(?:www\.)?realm667\.com/index\.php\?option=com_content&view='; $externlist['Vvwiki']    = 'https?://(?:www\.)?vavoom-engine\.com/wiki/index.php\?title='; $externlist['Wip']       = 'https?://(?:www\.)?wadsinprogress\.info/\?a=listwads&wad='; $externlist['Zanforums'] = 'https?://(?:www\.)?zandronum\.com/forum/showthread\.php\?tid='; $externlist['Zdforums']  = 'https?://forum\.zdoom\.org/viewtopic\.php\?';


 * pages are fetched from namespaces analog to listInterlinks and matches for each regex are stored in an array (keyed by external template) of arrays (keyed by DW path); sorting and outputting remained the same


 * On the initial pass, templates being matched by their own template's link were skipped, but because some direct links occur in multiple templates a handful of them were still included as false positives. So for the full run over all namespaces, Template space was skipped.
 * For future runs, '' is added to the exception list as direct links do belong on that page. As usual, file pages ending in " map.png" will also be skipped.
 * If I missed any external link templates in the patterns list, please let me know. --Xymph (talk) 14:29, 12 April 2018 (CDT)

Level has serious issues with popular mod loaded - put in "bugs", "trivia" or not at all?
Hey Xymph, I'm currently working on the walkthrough for. While playing through the level, I found that mods where boxes of rockets could be shot and blown up (e.g. Brutal Doom v21 beta) caused serious problems in this level, as one would telefrag the box (as part of the level progression) and die in the explosion. Hence, I'm planning to add the following line in into the page:

"This level should not be played with mods that allow boxes of rockets to be shot and blown up as if they were barrels (or where possible, have the setting turned off in the mod), as in the course of getting to each of the keys, the player will teleport onto a spot with a box of rockets already there. With such mods loaded, the player will telefrag the box of rockets and be hit by a point-blank explosion, which will likely kill them."

The question is - should this line go into "bugs", "trivia", or should it not be on the page at all? --15FiftySeven (talk) 09:51, 7 May 2018 (CDT)
 * Personally I wouldn't put it there. I would mention the issue instead on the Brutal Doom article. --Gez (talk) 12:03, 7 May 2018 (CDT)


 * The PWAD was designed for Boom-compatible ports, so when an unrelated mod has an unexpected gameplay effect, that's certainly not a map bug, and possibly not even something that needs to be mentioned on the PWAD page or the specific map page. Gez's suggestion works for me too. --Xymph (talk) 12:56, 7 May 2018 (CDT)

Shedding data from map articles?
Please double-check the recent changes: These are all tagged with the revision text "use secrets & map data from textmapStats.php" and have atypically large negative deltas. --Quasar (talk) 17:22, 18 June 2018 (CDT)
 * https://doomwiki.org/w/index.php?title=MAP04:_Water_control_(Doomworld_Mega_Project_2013)&curid=33083&diff=167921&oldid=167880
 * https://doomwiki.org/w/index.php?title=MAP11:_ffffabolous_secret_level_(Doomworld_Mega_Project_2013)&curid=33089&diff=167933&oldid=167887
 * https://doomwiki.org/w/index.php?title=MAP15:_NUKAGE_PROCESSING_(Doomworld_Mega_Project_2013)&curid=33093&diff=167941&oldid=167891
 * https://doomwiki.org/w/index.php?title=MAP18:_SUBBSE_(Doomworld_Mega_Project_2013)&curid=33096&diff=167944&oldid=167894
 * https://doomwiki.org/w/index.php?title=MAP19:_Trial_And_Error_(Doomworld_Mega_Project_2013)&diff=prev&oldid=167945
 * https://doomwiki.org/w/index.php?title=MAP02:_Twilight_of_the_Gods_(Doomworld_Mega_Project_2013)&diff=next&oldid=167878