User:Xymph/DMMPST

As usual on my user pages, some rambling and (non-)thinking out loud can be expected. ;)

Historical discussions
An index of past discussions about map and thing data:
 * 2011: Strange amounts of vertices in DOOM map articles
 * 2012: Map Info
 * 2012: Once again with map data
 * 2013: Map data
 * 2014: Calculating item counts
 * 2014: Thing_Spawn
 * 2015: Not all monster-like thing types listed
 * 2015: Vertex counts (GL_VERT)
 * 2016: Map info (redux)
 * 2016: Map thing statistics
 * 2016: Things template Usage & styling
 * 2016: Feedback on this DMMPST page
 * 2017: Thing appearance stats

2016-07-30
After a lengthy, real-life induced hiatus since its initial discussion, my DMMPST (DooM MaP StaTistics) tool has taken shape in the past week, and can now fill out DoomWiki templates for mapdata and things sections as well as complete map skeletons, for all classic Doom variants and Heretic.

Some samples of the latter: Doom II's MAP06 (comparison), Heretic's E3M3 (comparison), Sunlust's MAP11 (comparison)

Obviously, some bits of info need further (manual) updating before committing such a skeleton to the public wiki – like the PWAD name for non-stock maps, and the episode name for (Ultimate) Doom and Heretic based maps – but that's to be expected.

While it's not its primary purpose, the tool can also display some basic stats about a level, e.g. that same MAP11:


 * 1) DOOM ][  Map statistics for level MAP11: Chasing Suns

MapDim:	 MaxX	  MinX	 SizeX	  MaxY	  MinY	 SizeY 2055	 -4864	 6919	  2309	 -2288	  4597

Struct:	Things	Vertxs	Before	LinDfs	TelePt	SidDfs	Sectrs	Secret 657	 10297	 7396	  8429	    13	 14258	  1068	     3

Secrets:  442	   647	   990

For the tool, the biggest remaining task is probably settling on the contents and layout of the Things table, as discussed here. To help this discussion along, I can hand out beta copies of DMMPST to a few DoomWiki regulars who are willing to test it AND participate in that topic, so that it will (hopefully) move along more quickly than the usual glacial pace here. ;) Participants can email me via my homepage.

Thing tables
Details to be hashed out in the Thing tables include:
 * Order of items within classes:
 * Enemies roughly by cast sequence / increasing hit points (Doom); or by alphabet (Heretic)?
 * Weapons by slot - no contest.
 * Ammo by weapon/slot - no contest.
 * Powerups by class (armor, health, artifact/item) or alphabet?
 * Keys by alphabet - no contest.
 * Decorations, player starts, and other miscellaneous things - skip them?


 * Multiplayer:
 * One column (classic maps), or multiple columns or a separate table (coop, deathmatch) for modern maps?
 * How does one determine the map type anyway?
 * Does classic multiplayer and deathmatch take skill levels into account?


 * Should multiple columns with identical values be combined using colspan?

2016-07-31
A small DMMPST update now produces Thing tables with the extra multiplayer column for classic maps, see e.g. Heretic E1M1 (comparison).

I found the description of the two extra Boom thing flags (bits 5 & 6) confusing at first: 'If you want a thing to be only available in Single play, you set both the "not in DM" and "not in COOP" flags.' This suggests backward incompatibility, as classic maps won't have those flags enabled for Things intended to show up in single-player only (nor the classic MP flag, bit 4).

After examining the source code, function P_SpawnMapThing in P_MOBJ.C, it turns out that description is indeed cutting some corners: // check for appropriate skill level if (!netgame && (mthing->options & 16)) //jff "not single" thing flag return;

//jff 3/30/98 implement "not deathmatch" thing flag if (netgame && deathmatch && (mthing->options & 32)) return;

//jff 3/30/98 implement "not cooperative" thing flag if (netgame && !deathmatch && (mthing->options & 64)) return;

if (gameskill == sk_baby) bit = 1; else if (gameskill == sk_nightmare) bit = 4; else bit = 1<<(gameskill-1);

if (!(mthing->options & bit)) return; In words: when in singleplayer mode, it's still just bit 4 (and the skill setting) that determines whether a Thing is shown; only in multiplayer mode do the Boom flags distinguish between deathmatch and/or coop Things.

Is there a Boom(-compatible) map in the wiki that already includes a multiplayer (DM/coop) Things table so I can see how that looks?

Multiplayer
I realized that yesterday's multiplayer-related details and questions originated from a lack of understanding of Doom's multiplayer features. As a mostly-SP focused player who dabbled only a few times in IDMAP01, I had completely forgotten that coop play also came with the classic game from the beginning. Somehow the idea had settled in my mind that coop was only added in source ports, together with those Boom flags. Reading up on that MP page as well as the P_SpawnMapThing code made me understand at last, and also cleared up some aspects of Gez's discussion for me.

Thus I now also understand that the skill levels are indeed relevant in all MP modes too, and a single column such as in the Heretic tables does simplify the Thing tables somewhat (perhaps a choice made there for space reasons).

2016-08-02
The DMMPST beta (testers still invited) can now output Things tables for vanilla multiplayer and Boom-compatible cooperative / deathmatch. A sample of the first is Doom E1M1 (comparison), clearly showing the extra weapons, ammo and berserk pack in MP.

A brief search of the idgames archive turned up Boom-compatible map 102.WAD with these things. In coop there are two extra arch-viles and an extra cyberdemon, deathmatch has additional weapons & ammo and an extra megasphere. I manually verified the cyberdemons, chaingun and megaspheres in SLADE3, so the tables are probably correct (but more testing is welcome).

Ordering
I have yet to locate a Heretic PWAD page in the wiki (anyone?). Pages for stock Heretic maps order items per these classes: Monsters, Weapons, Ammo, Items/Powerups, Keys. The Doom Things template instead uses: Monsters, Items/Powerups, Weapons, Ammo, Keys. The Heretic order seems more logical to me, as monsters are fought with weapons, and keys are a kind of item too. Therefore DMMPST uses that same class order for the above E1M1 tables.

The Backpack is categorized a powerup in Doom tables, while the Bag of Holding is considered a collection of ammo in Heretic tables. I have a mild preference for the latter approach.

Should a class header be included even if it contains no items, e.g. a level without keys such as Doom II MAP07? Perhaps this shows more clearly that the class was not accidentally forgotten but really contains no items, but I'm not set on it.

Doom tables sort monsters roughly by cast sequence / increasing hit points, Heretic by alphabetical name. A comparison of both approaches:

Here I like the Doom order more. Shall we adopt this for Heretic too?

For weapons and ammo, the Doom & Heretic ordering by slot is fine, but hasn't been applied consistently in the past. See e.g. the aforementioned MAP07 where the super shotgun is last, and ammo is seemingly randomized. And for example in Plutonia MAP28, there's no apparent method to the ordering madness in almost all classes – with DMMPST that looks like this.

For items/powerups, the order in the Doom template is a bit odd: armors at increasing percentages (1%, 100%, 200%), armor/health 200% (megasphere), health decreasing (100%, 25%, 10%, 1%), berserk (100% health but predominantly a powerup), backpack, and then the remaining powerups in alphabetical order. More logical to me would be to order health by increasing percentage too, followed by the megasphere, like this:

This is the order I used in DMMPST so far, but moving backpack to the ammo class (as already proposed above) would put the megasphere adjacent to the alphabetical powerup list, which makes more sense to me in this regard as well.

2016-08-04
I've added the Hazards class to DMMPST's Thing tables, see Doom II MAP23 (comparison) – obviously :) – and Heretic E1M8 (comparison).

2016-08-11
After addressing some integer overflow issues ;), DMMPST can now process really large PWADs such as Vrack2BiggestEver (from the download section of the DeePsea homepage) and Vrack2SuperHuge (direct download).

The Things tables are also placed closer together by defining "width=33%" for the first, singleplayer table. This is done only when there are two or three tables, SP/MP or SP/Coop/DM (where it has no effect, as the columns already take up 33% each).

More importantly, DMMPST now supports Hexen and Strife, for example Hexen MAP23 (= Hub 4: Gibbet) and Strife MAP20 (comparison).

While the basics are in place, there are still plenty details to resolve in the ordering, formatting and linking departments:
 * What should the order of entries within each class be for each game?
 * Five tables side-by-side for Hexen is perhaps a bit much, so maybe something should be made collapsible, or the multiplayer tables should be below the singleplayer ones?
 * Strife tables don't aggregate generic minor characters such as Peasants into single totals. Also, shop props are included in the counts. Probably not going to bother with either.
 * Wikilinks for classes and entries should probably be made only once per Things section, for readability and SEO aspects?

For Heretic and Strife, I wasn't planning to replace the Things tables so carefully hand-crafted by Jartapran, but since they don't include (complete) multiplayer information, I probably will do that eventually.

For the Doom I intend to replace the Statistics section for all IWADs' map pages, to ensure they are consistent in content and style for both Map Data and Things (including multiplayer).

And of course, one major goal of this project is to finally provide Things tables for all Hexen map pages, and updated Map Data tables too (as those are still missing the vertexbefore value).

2016-08-14
DMMPST now wikilinks each Thing class and entry name only once. To demonstrate, most previous example pages have been regenerated:


 * 102.WAD
 * Plutonia MAP28
 * Doom II MAP23
 * Heretic E1M8
 * Vrack2BiggestEver
 * Hexen MAP23
 * Strife MAP20

Also, for all games monsters/enemies are now (mostly) sorted by hit points.

2016-08-15
Similarly to the Doom and Heretic tables above, here are the monsters/enemies in Hexen and Strife ordered by alphabet and hit points (with a few exceptions for Strife):

DMMPST currently uses the latter order, but feedback from Strife specialists (Quasar? ;)) whether this makes sense would be welcome.

To better assess the current ordering for all classes in all games, here is a complete overview of all classes and things. This also shows the wikilinks used for all entries, though as before in actual Things tables any second and subsequent identical links are pruned.

Some Hexen and Strife things don't have a default wikilink because they occur only in the same level as a previous, related thing (e.g. Hexen's planets and clock gears, and Strife's power coupling). Others don't have one because a suitable page doesn't exist (Gas grenade, Becoming Acolyte?) or I haven't found it yet (Force field guard?). My sources for these thing lists were the DeePsea Things browser (Search -> Find/F4 -> Thing/6 -> Browse -> Type), SLADE3's config files (config/games/include/things_hexen.cfg & things_strife.cfg) and, occasionally, the ZDoom wiki.

Hexen tabs
Upon a suggestion by Gez, I experimented with the Tab2 template as a way of showing multiple Things tables such that map pages aren't overwhelmed, especially for Hexen.

For vanilla Doom/Heretic maps with just two tables, each could be put in a tab like for Doom II MAP23. But a width parameter needs to be set for the entire table so it doesn't fill out to the entire page body, and that's not really portable as the various games have Thing entries of widely varying lengths and the required space depends on skins, browser font settings (Ctrl-+) and other factors too.

Also, putting just two tables in a tab container seems overkill, even three tables easily fit together within reasonable browser windows. Lastly, in mobile view (tested on my tablet) the whole Tab2 template is simplified to the contents of just the first tab. So this approach should not be used for just two or three tables – i.e. Doom, Heretic and Strife.

For Hexen it can be used to put the three class-based singleplayer tables in one tab, and the coop/deathmatch tables in the other, like for MAP23. On mobile, again only the SP tab's content is visible, but that might be acceptable.

Having just two tables in the second tab has the undesirable effect that the second (DM) table jumps quite far to the right. However, the nested col-break doesn't accept its usual width parameter, even when escaped like. No idea if that can be solved. Also, because the coop/DM tables are often a bit longer than the SP ones, content below the tabs jumps up and down while switching tabs. No biggie for me.

However, DMMPST so far is able to process all games in a generic way with a single Things template and some decision logic in the code. Breaking up Hexen's tables into two groups would make that much harder, if not require a separate processing branch – which I would like to avoid if possible. So I'm not sure yet whether the tabbed Hexen tables are the way to go. Opinions, anyone?

Stealth monsters
The Stealth monsters supported by some source ports were not yet handled, but are now, like in Plasmaplant (comparison). Curiously, the counts differ for quite a few entries. I'm not sure why that is, but the most likely reason is incorrect tallying of the multiplayer flags in that official table. DeePsea doesn't support stealth monsters, but in comparing its numbers for a variety of other thing types in multiple classes with DMMPST's, the latter always correlate.

2016-08-16
A few improvements, I think:
 * Combining three identical values into a single value row, like in Plasmaplant. Centering all the single cell values also looks better than leaving those left-aligned.
 * Adding the ambient sounds class to Heretic tables, e.g. E1M8.

Most layout and ordering aspects are homing in on their final state, I'd think. The biggest remaining issue are the Hexen tables: tabbed, or show them some other way?

2016-08-25
Summarizing various small improvements over the past week and a half:
 * The skill levels link above the Things table is now anchored to the game-specific section of that page; the same goes for class headers linking to the Item page.
 * Health & Armor is now a separate class with the non-countable items in Doom & Heretic, and all such items in Hexen & Strife (which don't count items and have no intermission screens).
 * The class header for power-ups, artifacts and items in all games is simply Items; for Doom & Heretic it consists (mostly) of the countable items.
 * Hexen's puzzle items are grouped with its keys, and similarly for Strife's quest items. Makes sense because of their nature, and because on most levels the classes have only one or a few entries.
 * Strife's Gold items moved from quest items to the generic ones.

To demonstrate, most previous example pages have been regenerated:


 * All classes and things
 * 102.WAD
 * Plutonia MAP28
 * Doom II MAP23
 * Heretic E1M8
 * Vrack2BiggestEver
 * Hexen MAP23
 * Strife MAP20
 * Plasmaplant

Strife poses a special problem to DMMPST's generic Things counting in that many of the Peasant, Rebel and Beggar things are level-dependent minor characters which have dedicated wiki pages. DMMPST currently doesn't handle these, and I doubt it can without introducing a copious plate of spaghetti code...

So despite Jartapran's consent to replace Things tables for all existing stock maps, I'm not sure that's feasible for Strife without losing valuable information.

In the tabbed Hexen table, prettytable won't take a style parameter just like col-break didn't take width, so at the moment only the row values spanning three columns are centered. Centering all the other cell-threesomes is feasible, at the expense of a lot of additional style attributes.
 * Hexen MAP23 (tabbed)

If tabs are not the way to go after all, then a suitable alternative may be to put the three SP tables on one row and the two MP ones on another: This also avoids the formatting and mobile issues that plague the tabbed variant, but makes the Things section take up a big extra chunk of page real estate.
 * Hexen MAP23 (2 rows)

All things considered (it's easier to deal with in DMMPST too), I'm leaning towards the last approach.

2016-09-14
Lack of updates here did not imply lack of progress: Strife-related coding, waiting for feedback, excursions into INFO.C-land, and real-life distractions all took their time. :)

The topic of Hexen tables was the easier one to sort out. After some further discussion I settled on my initial preference of generating them in two rows.

Handling the map-specific Strife characters took a lot more effort, but eventually I was able to code this in a relatively pasta-devoid manner, and I feel DMMPST output for Strife's IWAD maps is now suitable to supersede the existing manually-compiled Things tables.

The complete translation of generic actors (acolytes, beggars, peasants, and rebels) to specific wiki-linked characters per level is mapped out in this table. After the translations for a given level, the translated (i.e. customized) actors are sorted alphabetically within their class (enemies, other characters), while the generic actors follow within each class in their fixed default order.

As demonstration serves the level with the highest number of such translations, MAP02 Town. In the Enemies class, five named acolytes are listed first, followed by the generic ones, and then the remaining enemies. Similarly, the named major and minor characters are listed alphabetically before the general peasant population. Although entries within each class as a whole are not completely sorted, it seems to me this approach presents the significant information in the clearest way.

Character translations occur only on stock maps. If one or more PWADs are loaded, regardless of whether statistics are generated for a map inside or outside those PWAD(s), translation of Strife characters is disabled and the Things table may need manual adjustment.

And with that, if there is no further feedback, I consider Things generation in DMMPST complete, and it's time for the next phase: preparing a bot-based approach to updating the Statistics sections of all maps in the official IWADs. I probably will also apply manual updates to some sparse other map pages, and still plan to code appearance statistics into DMMPST too.

2016-10-10
After a few weeks of bot scripts development and testing, and improving their underlying Wikimate framework, the DMMPST project is reaching its primary goal: providing complete and consistent Statistics sections for all stock levels and uploading them automatically. The first public release of the DMMPST tool has been uploaded to idgames archive, and the first series of map pages have been updated: Ultimate Doom, Doom II, Heretic (including SSR), Strife and Sunlust. More as time permits.

2016-10-13
The remaining uploads for stock levels have been completed: TNT: Evilution, The Plutonia Experiment, Hexen and DDC, Master Levels, as well as Obituary. More megawads coming as time permits. For DOOM II PWADs that contain levels beyond MAP32, DMMPST currently does not load those correctly, so there will be an update to enable that.

Anyway, this completes the primary goal of this project, and further updates to this page / blog are unlikely, or will at best be few and far between. Thanks to everyone reading along, and in particular to those providing feedback (even though sometimes it was like pulling teeth ;-P).

2016-10-17
A &mdash; very nitpicky, sorry &mdash; comment about one, err, thing led to another. On deathmatch(-only) maps like id Map01 and Hexen DDC's Hub 4 – and on any map suitable for DM, really – the number of Deathmatch starts would be valuable information to include in the Thing tables. Then, while we're at it, the same might apply to Teleport landings, in single-player and multiplayer (coop/DM) tables. And to flesh out the miscellaneous items even further, two destructible (but non-hazardous) things: Hexen's destructible tree and Strife's wooden barrel. And maybe Heretic's key gizmos and teleport glitters too. Any others (e.g. in modern source ports) that I overlooked?

If there's sufficient consensus about all that, or at least no opposition..., the next question is where to put them in the tables. A separate class, or combined with an existing one in order to avoid another class with very few entries? If the latter, the logical candidate is the equally small class Hazards. What header would describe their combination in a suitable manner? I've tried out a few possibilities in the All classes and things page, and listed more below, including some that were used before:


 * Hazards & Map spots
 * Hazards & Others (nah)
 * Hazards & Misc.
 * Hazards & more (nah)
 * Hazards & Other parts (too long)
 * Technical
 * Working parts
 * Markers
 * Miscellaneous (the safe choice)
 * 

The latter entries would no longer allow linking Hazards to its category, but I suppose that's no big loss. So, opinions, anyone? :)

2016-10-18
After trying a few other options in the overview page, I think only "Hazards & Misc." and "Miscellaneous" work as suitable headers above a combined class. Anyone got a preference between the two?

2016-10-22
Further discussion and feedback led to the inclusion of all of the aforementioned items, and to using "Miscellaneous" as class header.

At first processing Doom-in-Hexen format levels seemed problematic, however it turns out that identifying and loading a Doom PWAD in Hexen-format was quite easy to support, but only if all maps in such a PWAD are in that format. Mixed formats (like 4maps) will probably require a complete rewrite of my/DEU's PWAD loading code, which I may defer til later.

I was hesitant to include any items, like coop starts and voodoo dolls, that require computation, to avoid cluttering up the generic counting and/or table building code. Fortunately I found a clean solution: a processing step between counting and table building, which then zeroes out all the original player start counts, thus preventing that they show up in the tables. From the original counts "Coop starts" and "Voodoo dolls" are now computed at imaginary Thing IDs, see the big overview. But I'm still not sure whether to handle voodoo dolls for 1-8 players as one or 1-8 values. The former results in one row of 4 dolls for, say, Plutonia MAP28 which has 1 for each player 1-4; and also for counter.wad which has 4 dolls for player 1. I'm thinking the latter approach – which would give four rows of 1 doll each for MAP28 – may be more accurate, but that could potentially result in 8 rows, although such maps are probably rare. Any feedback on that? Edit: One row it is.

There are no Team starts in the Hexen section of SLADE3's things_zdoom.cfg, nor could I find any Hexen levels using the CTF Standard, and therefore I left them out of the Hexen list.

Next to the existing mechanism to skip a class (like Keys) in a particular mode (like DM), I've also added a way to skip one thing from a class in a mode, needed so DM starts show up only in the classic Multiplayer or Boom/Hexen Deathmatch tables. Naturally, Coop starts now appear only in the MP or Boom/Hexen Coop tables, but in which table(s) should the Team starts be included? Edit: MP or DM tables, just like DM starts then.

Lastly, for now, I've added a command-line option to exclude SP/Coop tables, making it easier to generate MP/DM-only output for deathmatch maps like id Map01 and Hexen:DDC Hub 4.

2017-05-19
The recent server migration – interrupting my train of tho... err... updates :) – provided the incentive to return to DMMPST coding, and implement the promised thing appearance statistics.

These traditionally comprise a table with map names where the thing first occurs and another table with their total counts per IWAD. Those tables only cover the single-player campaign, but both first occurrences and total counts can differ when playing cooperatively, as those campaigns usually provide more resources and increased opposition. So I decided to implement both modes in the appearance statistics generated by DMMPST.

Vanilla Doom, Heretic and Strife things have just a multiplayer-only flag while Hexen things have distinct single-player, cooperative and deathmatch flags, and can be further distinguished by player class. Since some things (like the Mystic Urn) exist in both Heretic and Hexen, for valid comparisons the single-player tables use only Hexen's single-player flag and ignore the player class flags, while the multiplayer tables use only Hexen's cooperative flag and not the deathmatch flag.

Traditionally the tables only cover the IWADs as a whole, but first occurrences are relevant per episode and Hexen hub, while total counts per episode/hub are an interesting addition too. After all, they are all played from a pistol start or equivalent. So DMMPST generates that information for the pertaining IWADs, italicizing the episode/hub names and bolding the IWADs' total counts (bolding is not applied when no episodic/hub-based IWADs are included in the tallies). When combining the registered Doom/Heretic IWAD with their complete commercial variants (Ultimate Doom and Shadow of the Serpent Riders), the common episodes are listed only once, followed by the registered summary, the additional episode(s), and the commercial summary.

For first occurrences, DMMPST takes the playing order of maps including secret levels into account, so e.g. DOOM II maps 31/32 are checked after map 15. It doesn't actually affect the results for any DOOM II things as none of the ones that first occur in maps 31/32 do also occur in 16-30, but it's the correct method. ;) Doom/Heretic secret levels are also checked in their pertaining episodes' order. Many of Hexen's maps are not necessarily played in a fixed order due to the hub system, so these are simply checked in the order listed by the wiki.  Lastly, a fixed order for Strife's maps is not defined, so I'm not sure whether first occurrences can meaningfully be determined and thus DMMPST simply uses numeric map order.

As usual with DMMPST-generated templates, wiki pages (game, episode, hub and map names) are linked only once per type of statistics, although at this time the templating mechanism results in one set of repeat links for the game/episode/hub names in the second type of statistics.

With all the theory covered, how does it look in practice? :) Here are a couple of examples:


 * The super shotgun in Doom II and Final Doom (comparison)
 * The baron of Hell in all Doom variants (comparison)
 * The weredragon in Heretic (comparison)
 * The aforementioned mystic urn in Heretic and Hexen (comparison)
 * The scanner in Strife (no comparison available)

So, any feedback?

2017-05-20
The following changes have been implemented:
 * Heretic and Hexen IWADs are no longer considered compatible, so statistics tables for things appearing in both need to be generated separately.
 * This allowed the Section header to become Episode (Doom & Heretic) or Hub (Hexen); and likewise the M/C header became Cooperative (for Hexen) or Multiplayer (the other games).
 * For episodic/hub-based IWADs, the map names with first appearances are listed only for the episodes/hubs and not repeated for the IWAD.
 * Hexen map names that are identical to their hub names now have the correct "(map)" postfix.
 * The column layout of the tables is fixed to avoid alignment issues.

2017-05-23
After sleeping on it for a few nights :) I decided to use italics for things counts in episodes/hubs (rather than bold for the IWAD totals), and updated the above examples accordingly. Unless further feedback arrives soon, the appearance statistics feature is thus ready for roll-out on thing pages.