Doom engine games and source ports make use of several different map formats. There are currently four commonly used formats, each using different syntax for storing a map's data. Different source ports can utilize a map format in different ways, meaning a project in a given map format does not necessarily guarantee compatibility with any specific source port.
Doom map format
The oldest format, as originally devised by id Software, sometimes referred to as vanilla Doom format (or limit removing format if the map surpasses the static limits of vanilla Doom). It is a binary format used by the classic Doom series, but also by Chex Quest, Heretic and Strife, with some differences in their sets of line specials. Maps created using this format will be compatible with all source ports. This format does not support ACS, line specials cannot have arguments, things cannot have specials, arguments, and other features present in more advanced formats.
"Boom format" is a colloquial term referring to maps using features introduced with Boom, such as deep water effects, translucent walls, conveyor belts, generic linedef types and other features. Similarly, "MBF format" refers to maps which make use of MBF's features beyond those offered by Boom. These formats are the same as the Doom map format, however maps which use these extra features are not compatible with vanilla Doom, but will run in any source port with Boom or MBF compatibility.
Hexen map format
Developed by Raven Software to support the enhanced features of Hexen, most notably Action Code Scripting, this is an extension of the Doom map format. ZDoom pioneered using this format for the other games, making it known as "Doom-in-Hexen", "Heretic-in-Hexen", and so on. This is somewhat of a misnomer: only the map format is affected, a "Doom-in-Hexen" map is still a map for Doom and will not work properly in Hexen.
The ZWadConv and WinZWadConv utilities convert Doom-formatted maps to ZDoom's Doom-in-Hexen map format. For this reason, the Hexen format is a good choice for updating a vanilla or limit removing wad if one wants to add advanced features to the project, such as 3D floors or slopes.
A Hexen-format map has the same sequence of lumps as a Doom-format map, followed by a BEHAVIOR lump for compiled ACS, and optionally an uncompiled SCRIPTS lump at the end. However, the THINGS and LINEDEFS lumps are different and incompatible (the size of a single element is respectively 20 bytes instead of 10 and 16 bytes instead of 14), so even if a Hexen-format map does not use ACS, the BEHAVIOR lump is necessary to prevent the map from being interpreted incorrectly.
Most ZDoom-family ports feature compatibility with Doom-in-Hexen format maps.
Doom 64 map format
When Midway Games developed Doom 64, they chose to add more complex events to their levels than what was seen in the original game. To deal with these additions, they developed their own expanded mapping format for the game that presented numerous enhancements and brand new features, including: colored lighting for sectors previously seen in PlayStation Doom, generic linedef types, a system of compiled scripts for each map referred to as macros, and several new specials for sectors and flags for actors, including the ability to have actors be spawned in via triggers, or for the actors themselves to activate triggers upon dying (for monsters) or being picked up (for items). Over the years the mapping format was reverse-engineered and implemented in Doom64 EX and later the 2020 re-release of the game. Unlike the Hexen and UDMF formats, which came to support multiple games, the Doom 64 mapping format is exclusively used for Doom 64 custom maps.
Universal Doom Map Format
The UDMF, or "textmap" format, was developed specifically with extensibility in mind and does not suffer from the limitations that affect the binary formats. Ports which support UDMF format include GZDoom, Eternity, 3DGE, K8Vavoom and Zandronum. Each of these source ports has support for a variety of advanced mapping features such as 3D floors and deep water which can be taken advantage of using UDMF format.
A UDMF map has the following lumps: map header, TEXTMAP, ZNODES, REJECT, DIALOGUE, BEHAVIOR, ENDMAP. Of those, only the header, TEXTMAP and ENDMAP are mandatory; all others are optional, though they may be needed for certain features.