Ingame world maps, level maps, map layouts.. and their format

Anything relating to CD-i can be discussed in this forum. From the multiple hardware iterations of the system to the sofware including games, reference, music and Video CDs. Maybe you hold an interest in Philips Media and the many development houses set up to cater for CD-i if so then this is the forum.
Post Reply
User avatar
Shikotei
Burn:Cycle Activated
Posts: 42
Joined: Mon Mar 01, 2010 5:01 pm
Location: Netherlands
Contact:

Ingame world maps, level maps, map layouts.. and their format

Post by Shikotei » Mon Sep 04, 2017 3:30 pm

Perhaps I should've done this a lot sooner... but better late than never!
The many games I've played had incredible maps such as overworld maps, dungeon maps, race-layout maps, or a complex system of multiple areas making one giant world.
Games such as Alice In Wonderland is one of those latter. Oh the times I got lost (or family for that matter) trying to find my way forward (after having to backtrack 10+ screens)...
Zelda's Adventure has one of the most beautiful maps out on the CD-i, but many of those available on the 'net are low quality and small in dimensions, and simply don't do it justice.
Sometimes having a map would have made things so much easier in games such as Dimo's Quest and Micro Machines.

For games with a giant map I used a script to divide the ginormous image into hundreds and hundreds of smaller images to make it viable for viewing on the web.
The viewer of this map has a bit of (experimental) JavaScript. It may not work on every browser out there, but at least it works on Firefox 54, Chrome 60, IE11 (must allow ActiveX though..) and probably some versions that are a little older.
Use your mouse (perhaps it works with fingers on a tablet/phone) to scroll through the world.
There's a minimap to the right to show you where you are.
You can also drag the minimap for fast-travel!

I'll add a little bit of information on how the map data is stored/encoded. It's not supposed to be a full explanation, but just a bit of insight on what incredible (sometimes complex) methods were used.

Zelda's Adventure
Giant map
Original size: 12288x8000 pixels and about 72MB.
Each piece is 198x200 pixels and range from 1kB-5kB (nigh empty) to 125kB (the biggest). A grid of 4x4 pieces is shown. 2520 pieces total.
Encoding method:
- DYUV, one screen at a time (384x240 pixels per screen)
- 305 screens total in over.rtf
The file over.rtf contains WAY MORE than just images! Audio sectors detected.

Dimo's Quest
List of maps.
Size per map: 1152x704 pixels, ranging between 17kB and 115kB.
Encoding method:
- 1 level in levNN.rtf (NN=01..51)
- Level is 72 by 44 tiles
- Stored diagonally mirrored (Xout=Yin, Yout=Xin)
- Level tiles are indexed, 2 bytes per tile
- 1 set of tiles in gfxsetM.blk (M=NN/10+1) (ie N=34 -> M=2).
- Tile set is CLUT stored

The Apprentice
List of maps.
Size per map: 360x3980 pixels, ranging between 461kB and 904kB.
Encoding method:
- 1 level in mapN_M.dat (N=1..7, M=1..3 for N<=6, M=1..5 for N=7)
- 3 layers per map: background, foreground (platforms, decorations, etc), interactibles
- Background: 2 bytes per tile, 6 tiles per row, 200 rows (1 wide, 4x3 wide, 1 wide)
- Foreground: 2 bytes per tile, 16 tiles per row, 200 rows
- Interactibles: 2 bytes per tile, 16 tiles per row, 200 rows
- Layer tiles are indexed (background has gaps: tiles that are 3 'normal' tiles wide 'cost' 3 indices -> subsequent background: 0,3,6,9 instead of 0,1,2,3)
- Tiles are stored compiled in 2 pieces (top half, bottom half), 20x10 pixels per piece (20x20 total)
- Tiles (all layers) are stored in levelN.dat (N=1..6, b (b for bonus levels: map7_M.dat M=1..5))
levelN.dat contains WAY MORE images than just the level tiles! Monsters, Marvin, effects (basically every required image for that level).
Only the first tower was 'fully' completed. It is the only one where I replaced the interactibles with the actual interactibles. The other towers (and bonus levels) only have their index.

List of games that have CLUT-stored maps
These images are easy to extract and thus I won't show 'em here. Besides, there's no added insight or overview on 'where to go' for these games.
- Link: The Faces of Evil
- Zelda: The Wand of Gamelon

The list:
Zelda's Adventure big overworld map
Zelda's Adventure overworld and dungeon maps, with sprites
Dimo's Quest level maps
The Apprentice level maps
Micro Machines level maps
Alice In Wonderland world map
Buzz Off level maps
Room Service level maps
Full Attack level maps
Skid Kart level maps
Joukisen map
Mega Maze level maps
Christmas Country level maps
Lucky Luke level maps

Games that are added later will be on the above list, while the details will be in their own post.

Currently working on processing Micro Machines (list of maps) and Alice In Wonderland (giant map).

-- edit 08/09/'17 --
Added MicroMachines and Alice In Wonderland to the list of games.

--edit 09/11/'17 --
Added Family Games II: Buzz Off and Room Service to the list of games.

--edit 02/02/'18 --
Added Family Games I: Full Attack, Skid Kart, and Joukisen to the list of games.

-- edit 16/06/'19
Added Mega Maze, Christmas Country, and Lucky Luke The Video Game

-- edit 09/07/'19
Updated links to https and renamed both links to Zelda's Adventure maps.

-- edit 18/12/'19
Updated links to reflect new website
Last edited by Shikotei on Wed Dec 18, 2019 7:24 pm, edited 6 times in total.

User avatar
Shikotei
Burn:Cycle Activated
Posts: 42
Joined: Mon Mar 01, 2010 5:01 pm
Location: Netherlands
Contact:

Re: Ingame world maps, level maps, map layouts.. and their format

Post by Shikotei » Fri Sep 08, 2017 5:27 pm

Well, this took quite a bit longer than expected. Partially due to MicroMachines' weird CLUT pixel format, but mostly due to Alice's stupidly obscene connection.
Anyway:

Micro Machines
List of maps.
Size per map: 3072x3072 pixels, ranging between 212kB and 1478kB.
Encoding method:
- 1 level in ROUNDN.M (N=1..7, M=1..3 for N=1..5, M=1..5 for N=6..7) (with a few exceptions)
- Level is 32 by 32 blocks
- Level blocks are indexed (only 0x00 to 0x7F are used: 0x80 to 0xFF are treated as 0x00 to 0x7F), 1 byte per block
- Block is 6 by 6 tiles, indexed (1 byte per tile)
- Block is defined in BLOCKS/N.BLK (N=BEDROOM,BREAKFST,GARAGE,GARDEN,PATIO,POOL,SAND,SCHOOL,SPORTS) (first block of 'index' data)
- Tile is 16x16 pixels
- Tile is stored in BLOCKS/N.BLK (N=BEDROOM,BREAKFST,GARAGE, etc) (second block of 'index' data)
- Tile is CLUT stored - color index is 0x00 to 0x8F, index 0x80 to 0xFF are treated as 0x00 to 0x7F. These indices indicate special purpose (races go below it, fall/drown/sink, or transported elsewhere).

Alice In Wonderland
Giant map
Original size: 8200x9600 pixels and 4MB in size. Small size (MB) because over 90% is empty space.
Each piece is 200 by 200 pixels and range from 1kB-5kB (nigh empty) to 21kB (the biggest). A grid of 4x4 pieces is shown. 1968 pieces total.
The green lines indicate where each door (and sometimes map edge) leads to. Arrows indicate one-way tickets. The brown arrows are guestimated (the only ones that weren't in the Let'sPlay I watched).
Encoding method:
- All screens in alice_rooms.rtf (320x160 pixels per screen, 251 screens total)
- 1 screen is 40 by 20 tiles, indexed (2 bytes per tile)
- Tile is 8x8 pixels (CLUT stored)
- Tile is stored in fpc.rtf
- fpc.rtf contains 21 sets of tiles
- Tile index has unknown relation to set index
- Each screen had 21 variations, correct one hand-picked.

The list:
Micro Machines level maps
Alice In Wonderland world map

User avatar
Shikotei
Burn:Cycle Activated
Posts: 42
Joined: Mon Mar 01, 2010 5:01 pm
Location: Netherlands
Contact:

Re: Ingame world maps, level maps, map layouts.. and their format

Post by Shikotei » Thu Nov 09, 2017 12:12 pm

I had hoped to have done this faster, but I've been stuck at Family Games II - Room Service for a long while.
This time I've looked at the Family Games II games and even though there's a bunch of games on it, there's really just two that I think require a map.
Mortal Pong, Power Bucket, and the Exploding Pizza games don't even have maps (just game screens)!

Buzz Off
List of maps.
Size per map: 9300x512 pixels, ranging between 2075kB and 2646kB. Level 8 is 8800px wide though.
Encoding method:
- 1 level in levelN.rtf (N=1..8 )
- Level is stored as CLUT7, in sections of 100px wide (93 sections (88 for level 8 )).
- Level is stored as single background with interactibles visible
- Lower half contains collision data (used colors to make this clear).

Room Service
List of maps.
Size per map: 2848x2848, 2720x2720, 2800x2800, ranging between 718kB and 1116kB, and between 751kB and 1161kB with control numbers.
Encoding method:
- 1 level in roomN.rtf (N=1..3) (/RTF/ROOM/<file>)
- Level is variant in size, but always 2 bytes per tile. (178x178, 170x170, 175x175)
- Level size (nr tiles by nr tiles) is stored in first 4 bytes of roomN.rtf (N=1..3)
- Tiles are CLUT stored, 16x16 pixels.
- Color table in roomN.dat (N=1..3) (/DATA/ROOM/<file>)
- Tile index is bitwise rotated right 1 bit. 7 MSbs are suspected control/action bits.
IE: source=0x9096 -> rotate left 1 bit -> index=0x212D&0x01FF = 0x012D, control=0x212D&0xFE00 = 0x2000
(I could be wrong and have switched the MSB and LSB, in which case the tile index is the 9 MSb and the control bits the 7 LSb
IE: source=0x9680 -> index=0x9680&0xFF80 >> 7 = 0x012D, control=0x9680&0x007F = 0x0000)

Shown numbers on map are the control values. Some reveal the hidden coins in Level 3, but not the changes made by picking up keys. Those are probably in the .dat file.
Interesting tidbit is that in Level 3, not all teleport pads (yellow 'W' markers) are active. Most notably the area with the ton of time and coins!
Oh how many times I tried to get there, testing out every teleport pad in the giant field (32 pads). Just saw that all but one goes to 'home'.

The list:
Buzz Off level maps
Room Service level maps

User avatar
Shikotei
Burn:Cycle Activated
Posts: 42
Joined: Mon Mar 01, 2010 5:01 pm
Location: Netherlands
Contact:

Re: Ingame world maps, level maps, map layouts.. and their format

Post by Shikotei » Fri Feb 02, 2018 10:17 am

Family Games I, probably the easiest-to-acquire game on the console. Except I couldn't find an ISO on the 'net, so ripped my own disc.
Turns out a BD-burner can read these just fine!
There's only 3 games on it that I think would benefit from a map: Full Attack, Skid Kart, and Joukisen.
The other games (Where's that Sound, Captain Alphabet, DRAW stamp COLOR, SUPERSLIDE, Morphon Invasion, Wizard, Hot Rocks) simply don't have maps, or just have single screens.

Did anyone ever complete Full Attack? I never got past the Icy level (3rd level).
Once you reach the 8th level of Skid Kart and manage to find enough flags, you can't do much until time runs out (there's no exit!). The one thing that does change, is the ability to take sweet sweet revenge on the blue and pink cars as you can now destroy THEM!

Full Attack
Mostly the same encoding as Family Games II's Buzz Off, but (apart from the size difference) there's no monster collision data.
List of maps.
Size per map: 18700x408 pixels, ranging between 775kB and 964kB. Level 3 is 1679kB though.
Encoding method:
- 1 level in attack_levelN.rtf (N=1..4)
- Level is stored as CLUT7, in sections of 100px wide (187 sections).
- Level is stored as single background with interactibles visible
- Lower half contains collision data for map, not enemies.

Skid Kart
List of maps.
Size per map: 1552x992 pixels, ranging between 54kB and 152kB.
Encoding method:
- 1 level in skidkart_levelN.rtf (N=1..8 ), first block of data
- Level is 32 by 32 tiles
- Level tiles are indexed, 2 bytes per tile
- Tile is 16x16 pixels
- Tile is stored in skidkart_levelN.rtf (N=1..8 ), second block of data
- Tile is CLUT stored in skidkart_skid.rtf
I marked the 'secret' entrances to the (often poorly) hidden pathways. Turns out that not everything is accessible.

Joukisen
Single map
Size of map: 3024x2240 pixels, 540kB.
Encoding method:
- Level in yokosan_level.rtf, first block of data
- Level is 32 by 32 tiles
- Level tiles are indexed (only 0x0000 to 0x0FFF are used: 0x1000 to 0xFF00 are action codes?), 2 bytes per tile
- Tile is 16x16 pixels
- Tile is stored in yokosan_level.rtf, second block of data
- Tile is CLUT stored in yokosan_yoko.rtf


The list:
Full Attack level maps
Skid Kart level maps
Joukisen map

--edit 21/02/'18
Spelling & emoticons fix

User avatar
Shikotei
Burn:Cycle Activated
Posts: 42
Joined: Mon Mar 01, 2010 5:01 pm
Location: Netherlands
Contact:

Re: Ingame world maps, level maps, map layouts.. and their format

Post by Shikotei » Sun Jun 16, 2019 9:11 am

Mega Maze! One of the best puzzle games for the CD-i. And one with a TON of levels.
There's 65 level codes, but 165 levels! Many have two stages per level, some even more.
The game is ridiculously stored in a single file (apart from the general stuff [bumper, abstract, etc]).

My first attempt at decoding the full map data yielded only half the information I needed.
I found the map itself, the colors (there's two color tables, a red and a blue), but not the sprites.
This was back in November 2017. Now, in May/June 2019, I've resumed my efforts in getting the rest.
I've made some progress, but still haven't found the sprites. What I did find? Odd methods of building the levels.

Mega Maze
List of maps.
Size per map: 384x280, ranging between 5kB and 17kB.
Encoding method:
- everything in menu.rtf (seriously, the WHOLE flipping game is in that file!)
- First two sectors contains a list of indices (probably the sub-files) based on sector number (can't just split, gotta use the indices).
- Used indices to split menu.rtf into 181 smaller files. Most are 112 sectors (165x game map), three are fewer, the rest are bigger (up to 4848 sectors!).

[s]Each map-file contains a meta-map (where the marbles can move), the CLUT-stored map itself, the CLUT-table and interactibles.[/s]
Each map-file contains:
- Meta map (where the marbles can move)
- The color scheme (256 colors)
- The map itself (CLUT-stored)
- Tilenumbers-based map, preceded by the width and height of the map (32x28). 2 bytes per tile
- The shadows tile information (yes, the shadows of the tiles are stored in a second layer!) preceded by the number of shadow tiles. Each tile has 3 numbers: code, x-location, y-location.
- The interactibles tile information (same format as the shadows tiles)
- Another interactibles tile information (same format). The numbers are different, but not the locations. Behavior codes?

There are a few things I could not find... the start location of the blue marble (the player). Although looking at the map will often make it clear where you start, it's a little disappoiting to miss this.
I found tile numbers, but no tiles. I found sprite numbers, but no sprites.

Each map I made is made from 2 images: the CLUT-stored map, and the (homegrown) interactibles map.
With homegrown I mean that I didn't find the sprites, and so made my own.
These two images are stacked in HTML with a bit of CSS magic.
The order of the maps have absolutely no correlation with the order of levels in the game.

Christmas Country
One of the few games that I feel was not completed. Rushed to market too soon. Biggest reason? No ingame music other than the main menu, world introduction, boss-fights, and credits. And the first three are almost identical!
Still, it's a game with a lot of secret passages, hidden areas, and 19 levels (four in each of the four worlds, and three bonus levels).

List of maps.
Size per map: from 7200x240 to 16000x240, ranging between 88kB and 247kB.
Encoding method:
- 1 level data in levelN.rtf (N=1..16)
- 1 level data in bonusN.rtf (N=1..3)
Each level data file contains:
- Color table (indexed)
- Map width and height in number of tiles (2 bytes per number)
- Map tile codes (2 bytes per tile) (7 control bits, 9 index bits)
- Tiles (CLUT-stored) (16x16 pixels)

I've made two versions of each map, one with control codes, one without. The control codes are useful to discover hidden objects and pathways.


Lucky Luke - The Video Game
This game... was a (bleep) to figure out. At least map-wise a (bleep). The game itself is pretty fun to play! One of the ones I'd recommend to give a try.
What many people (at least the walkthroughs on YouTube I found) do not know, or never found out, is that this game HAS SECRETS!
If you duck/lay down at certain points in each level, there's a warp to a secret part of the level. The full layouts have shown there are TWO in EACH level. Only exceptions are the horseback and minecart levels (you can't duck here anyway).

As for why I spent the better part of a month figuring out where and how the maps are stored... I'll explain:
I first tried this in May 2018 and got .. little. I found the map data and the map tiles (pieces of tiles), but not how the pieces were lined up to make a tile.
I did find this little easter egg while poking around.

The level data is stored in a single file per level (yay!), split in multiple blocks.
After thoroughly going through this data time and time again, incrementally identifying each little blob of data I gained the following knowledge:

The normal levels (Town/Indians/Circus) (1.10MB to 1.13MB) contents:
- Block data (2 blocks)
- 1)Level behavior data
- 1)Sprite list of offsets (complex structure)
- 1)Sprites data (player, UI, items)
- 1)Sprite list of offsets (complex structure)
- 1)Sprites data (enemies)
- 1)Color tables (4 tables, 128 colors each, day/night/?/?)
- 1)CLUT list of offsets (4 bytes each)
- 1)CLUT data (2x20 pixels per image, hundreds images: background tile-parts)
- 1)CLUT list of offsets (4 bytes each)
- 1)CLUT data (4x20 pixels per image, hundreds images: foreground tile-parts)
- 1)Sprite list of offsets (4 bytes per offset)
- 1)Sprites data (foreground objects, split into 4x20 pixel pieces)
- 2)MPEG list of offsets (4 bytes each)
- 2)MPEG data (audio only -> sound effects)
- 2)Map data (4 maps, 8 bytes header per map) (background, foreground, items, action)

Background tiles are 2x20, but 10 are needed to make 1 tile usable for the background map. This map is half as wide as the foreground map.
Foreground tiles are 4x20, but 5 are needed to make 1 tile usable for the foreground map.
As for which 5 tiles to concatenate, it's not the sequential 5 (0..4, 5..9, 10..14), but is defined by the header (which I ignored the first 20 tries).
Why did I ignore the header? Because it was a list of numbers starting at 0 and going sequentially up in steps of 1. I didn't notice it repeated the first number and stopped being linear 50 numbers in.
For instance, the first tile (blank) is a concatenation of pieces 0,0,0,0, and 0.
The first 32 tiles:

Code: Select all

  0   0   0   0   0
  1   2   3   4   5
  6   7   8   9  10
 11  12  13  14  15
 16  17  18  19  20
 21  22  23  24  25
 26  27  28  29  30
 31  32  33  34  35
 36  37  38  39  40
 41  42  43  44  45
 46  47  48  49  49
 49  50  51  49  49
 50  52  53  54  49
 49  49  49  49  49
 49  49  49  55  56
  0  57  58  59  60
 61  62  63  64  65
 66  67  68  69  70
 71  72  73  74  75
 76  77  78  79  80
 81  82  83  84  85
 86  87  88  89  90
 91  92  93  94  95
 96  97  98  99 100
101 102 103 104   0
105 106 107 108 109
110 111 112 113 114
115 116 117 118 119
120 120 120 120 120
121 122 123 124 125
  0 126 127 128 129
130 131 132 133 134
As you can see, it's a bit of a mess. All my other attempts failed miserably and created a garbled mess of everything.

Items... I haven't found the correlation between item code and sprite number. The header data for these sprites is too complex (it's not a straight number list).
I did try to circumvent this problem by concatenating 5 sprites into 1 (nearly all of these pieces are 4px wide... see the pattern?) and use the new ones.
This worked like a charm!
My program cannot concatenate tiles that are not 4x20 properly without file-specific instructions and will result in a tile that is wrong. Only 2 or 3 are 'broken' like this (there's hundreds of tiles in a level).

Action codes: I've made a map of the codes to indicate where the secret warp points are. The codes to look for are 23 and 25 (use the checkboxes to make searching easier).
All tiles are CLUT stored, making this fairly easy to make into a map.
These levels have 4 layers:
- Background
- Foreground
- Items
- Action codes (secret warp, color table change, platform/wall, enemies)


The horseback levels (3 and 5) (5 are 800kB, 1 is 802kB) setup is mostly similar, but has key differences:
- Block data (2 blocks)
- 1)Level behavior data
- 1)Sprite list of offsets (complex structure)
- 1)Sprites data (player, UI, items)
- 1)Color tables (4 tables, 128 colors each, normal/fog/?/?)
- 1)CLUT list of offsets (4 bytes each)
- 1)CLUT data (4x20 pixels per image, thousands of images: background tile-parts)
- 1)Sprite list of offsets (4 bytes per offset)
- 1)Sprites data (most are 4x20, some smaller, one is 0x0)
- 2)MPEG list of offsets (4 bytes each)
- 2)MPEG data (audio only -> sound effects)
- 2)Map data (3 maps, 8 bytes header per map) (foreground, items, action)
The minecart levels (6) (all 3 are 376kB) setup is the same as horseback levels (just fewer images/sprites)

The foregroundtiles are each 4x20, but to be used for the map 5 of them will have to be horizontally concatenated to 20x20 pixels.
Most of the item sprites are 4x20 pixels (some are less in width).


These levels have 3 layers:
- Foreground
- Items
- Action codes

All map data (of all 6 levels) have an 8 byte header containing tile width/height (in pixels) and map width/height (in number of tiles).

Anyway, enough is enough and I'll list the additions. Suffice to say it's been rather delightful when I got the results I was hoping to get!
Fun facts: I used 63 map datafiles (the layout) and 27130 tiles (made from 5 or 10 pieces each) (totaling just over 34MB) to generate 108 layers, so you can see them separated.

So to sum this up:
- Every level has 4 layers: background, foreground, items, action(interactibles);
- Levels 3,5,6 have no background layer
- All tiles have two color schemes (normal and an alternative: night,storm,fog)
- Background tiles are made from 10 pieces. Each piece is 2x20. CLUT stored.
- Foreground tiles are made from 5 pieces. Each piece is 4x20. CLUT stored in levels 1,2,4
- Foreground tiles are made from 5 pieces. Each piece is 4x20. Sprite stored in levels 3,5,6
- Item tiles are made from 5 pieces. Each piece is 4x20. Sprite stored
- Background layer is 240x15 tiles.
- Foreground layer is 480x15 tiles.
- Item layer is 480x15 tiles.
- Action layer is 480x15 tiles.
- Action layer contains enemy locations, platform data, interactibles, and more

The list:
Mega Maze level maps
Christmas Country level maps
Lucky Luke level maps

User avatar
Shikotei
Burn:Cycle Activated
Posts: 42
Joined: Mon Mar 01, 2010 5:01 pm
Location: Netherlands
Contact:

Re: Ingame world maps, level maps, map layouts.. and their format

Post by Shikotei » Fri Jun 21, 2019 10:02 pm

omegalfa wrote:
Mon Jun 17, 2019 11:22 am
Shikotei....
F**k off......... this is an absolutely amazing work....!
I don't even have an idea of the time you spend doing this!

Congratulations!
Hmm.. I never really kept track of how much time I needed to get a single game worth of maps.
Wouldn't really make sense at first anyway; I would have to make the decoding and extraction programs first (which cost way more time than any game so far).

In total I think I spent well over 400 hours fiddling with code, raw data and unknown CD-i formatting (those compiled sprites were time consuming demons).
This did result in tools that can make extracting a game with good sector-data a matter of minutes.

In this posts there is sad news too though:
The original giant map of Zelda's Adventure has been removed. Not due to copyright claims (does Viridis still exist?), or other official reasons.
I've simply replaced it by something different.
You see I only have 500MB worth of space on my site and the latest addition simply is too big to fit.
Don't worry about losing the chance to see Zelda's world of Tolemac. The replacement is ALSO Tolemac!
The total free space on my site is less than 1 MB.. nothing can be added for now.

Allow me to explain: The original giant map shows only the overworld. The new map includes the dungeons.
And the sprites.
And the metadata map.
Oh yes, I've found these!
You see, the game files over.rtf and under contain the overworld and underworld maps. Trouble is that they're part of these giant files and not neatly separated.
Well, I've gone and done it. I found a way to separate each screen with all the required data.
I have extracted from these datafiles:
- The DYUV encoded screen (384x240 pixels)
- The RLE encoded metadata (384x240 pixels)
- A sprite datablob (with multiple complex header blobs)
- Two 2-second audio files (background sounds)

The sprite datablobs headers have been fully parsed and enabled me to precisely extract the sprites. These are neatly grouped by header-depth.
As a result, with a little HTML and JavaScript, I now can show you which sprites can be found on which screen.
Including unused sprites. For example, has anyone ever seen the surfer on the west coast near the volcano?
Or ever acquired a trumpet from the talking mushroom north of the cave that has the Harp?
Or a trident on the southeast border between the swamp and the beach?
Or the fishing net near there?

Numbers? Overworld has 305 screens and 3973 sprites. Underworld has 184 screens and 2211 sprites.

I might make a more detailed post on how the files were encoded, but right now I'm gonna watch another episode of Robin of Sherwood (1983).

Instructions
There's two layers (background and metadata), 1 overworld, 7 dungeons, and Ganon's Gauntlet to view. All screens and sprites accounted for.
The directional texts move the viewer 1 screen in that direction.
The minimap (red blocks) can be used to instant-travel there.
The green block is where you are now.
You can visit black blocks, but there's no map data.

--edit 01/07/'19
Format typo
Last edited by Shikotei on Mon Jul 01, 2019 5:12 pm, edited 1 time in total.

User avatar
Bas
CDinteractive Admin
Posts: 3041
Joined: Mon Jun 20, 2005 11:14 am
Location: the Netherlands
Contact:

Re: Ingame world maps, level maps, map layouts.. and their format

Post by Bas » Sun Jun 23, 2019 1:05 am

I'm sure that either Devin (if he is ever around) or Omegalfa can help you out. Devin is still hosting the CDinteractive Network with plenty of space. Omegalfa has with djkoelkast The World of CD-i with probably also enough space, but I don't know about that. Once they'll see this, one of them will surely help you out, I loved the big Tolemac world map!

Who has space to host his stuff?

User avatar
Shikotei
Burn:Cycle Activated
Posts: 42
Joined: Mon Mar 01, 2010 5:01 pm
Location: Netherlands
Contact:

Re: Ingame world maps, level maps, map layouts.. and their format

Post by Shikotei » Mon Jul 01, 2019 5:11 pm

Bas wrote:
Sun Jun 23, 2019 1:05 am
I'm sure that either Devin (if he is ever around) or Omegalfa can help you out. Devin is still hosting the CDinteractive Network with plenty of space. Omegalfa has with djkoelkast The World of CD-i with probably also enough space, but I don't know about that. Once they'll see this, one of them will surely help you out, I loved the big Tolemac world map!

Who has space to host his stuff?
omegalfa wrote:
Mon Jun 24, 2019 5:31 pm
We would love to host your work.
We have 1000GB space on The World of CD-i server.

Just contact me if you accept.
Your offers for hosting are heartwarming, and nearly convinced me to join forces.
I thank you for this kindness!
One of the reasons I won't accept them, however, is that my hosting package is very very outdated (it started on May 29th in 2007) and could do with an update.
A short overview of the package:
- 500MB storage
- 10GB traffic per month
- no PHP, Python, ASP, Ruby, or other serverside language
- no databases
- 25 e-mail addresses
- 3 SFTP
- 20 sub domains
- 3,62 Euro per month

I've eventually decided to upgrade this to:
- 10GB storage
- no traffic limit
- PHP 7.3, CGI, Perl, Python, Ruby
- 5 databases (1GB) + phpMyAdmin
- 2000 e-mail addresses
- 5 SFTP
- 100 sub domains
- SSL certificate (yay for https!)
- 6,00 Euro per month
There are huge increases in all aspects for just a little bit more money. Tonight they'll put it through.

With the inclusion of PHP in the serverside languages, I can now add a whole lot more functionality to the website.
I might even try to find a way to get my decoding tools (as far as they exist in PHP) online.

User avatar
Bas
CDinteractive Admin
Posts: 3041
Joined: Mon Jun 20, 2005 11:14 am
Location: the Netherlands
Contact:

Re: Ingame world maps, level maps, map layouts.. and their format

Post by Bas » Tue Jul 02, 2019 11:45 am

that's even better, great to hear that! I'm looking forward to more Shikotei stuff!

User avatar
Shikotei
Burn:Cycle Activated
Posts: 42
Joined: Mon Mar 01, 2010 5:01 pm
Location: Netherlands
Contact:

Re: Ingame world maps, level maps, map layouts.. and their format

Post by Shikotei » Tue Jul 09, 2019 1:46 pm

Bas wrote:
Sun Jun 23, 2019 1:05 am
.. I loved the big Tolemac world map!
Then you'll be pleased to know it's back :mrgreen:
I even fixed an old mistake (south of the swamp I switched two screens).

As promised I'm going to explain how the overworld (and dungeon) maps, sprites, audio, and metadata is stored on the Zelda's Adventure disc.
Be prepared for a wall of text, code, and a few images!

First off, the files in question are 'over.rtf' and 'under.rtf'. They are 41512 and 24645 sectors in size (or 97636224 and 57965040 bytes).
So far I've been using fairly simple methods of decoding files, looking only at the type of sector (audio/video/data) and acting accordingly.
But these two files are far too big and chaotically built to simple split them by groups of sectors.
I've therefore used a new method: split them by EOR flag.

The EOR (End Of Record) flag is present in the subheader of each sector. Mostly unset, this flag can indicate the end of a record (a subfile if you will) within a file.
Using this flag I got 610 subfiles.
The first thing I noticed was that exactly half the subfiles are the same size (7 sectors), and contained only empty sectors.
The file numbers (I numbered each subfile starting from 0 up to 609) of the empty subfiles are all odd numbers (1,3,5,7,9).
All other subfiles are only 3 sizes: 6 are 137 sectors, 297 are 129 sectors and 1 is 116 sectors.

Code: Select all

over-subfile 0.dat      297KB
over-subfile 1.dat       17KB
over-subfile 2.dat      297KB
over-subfile 3.dat       17KB
over-subfile 4.dat      297KB
over-subfile 5.dat       17KB
...
over-subfile 608.dat    297KB
over-subfile 609.dat     17KB
Next step was to split each subfile into the different sector types, and ignore further empty ones.
Audio wasn't too hard:
- Further split sectors (by file number, channel number, bits per sample, sample frequency, mono/stereo) into groups of the same format
- Decode groups with the ADPCM decoder and output to WAV

Video wasn't too bad either:
- Further split sectors by encoding type (CLUT4, CLUT7, CLUT8, RL3, RL7, DYUV, RGB555(lower), RGB555(upper), QHY, Reserved) I found RL7 and DYUV
- RL7 requires CLUT palette
- DYUV requires start YUV values per line

For RL7 I initially use a greyscale palette so I can at least see what this might be. Almost all RLx images are 384 pixels wide, but can vary in height.
In this case, they're all 384x240. With the greyscale palette I could see that there would not be a normal color table present in these files: too little indicating that this isn't an image, but action codes.

Same goes for DYUV: I can generate the start YUV values (black or grey) and perhaps get the properly decoded DYUV image.
In this case, only a handful decoded properly (the others require specific values).
Because I attempted decoding these images earlier I know that there are start YUV values in the data sectors.

Data was.. interesting. And challenging.
First thing I did was dump the grouped data sectors (only the payloads) and see if there are similarities. The sizes are all over the place (from 36 to 67 sectors), so anything simple just went out the window.
After some poking around, comparing the data subfiles, using the knowledge I've gained from dozens of other sessions, I found the following structure:
-(1) 1 sector with start YUV values
-(2) 8 sectors with unknown data (QHY data suspected)
-(3) 1 sector with header data
-(4) 10 sectors with audio (but only n are used)
-(5) 4 sectors with header data
-(6) n sectors with sprite data (cause of the varying sizes)
-(7) 4 sectors with header data
-(8) 1 sector with color table data
-(9) 1 sector with header data
-(10) remainder of the file (audio data, but the last x sectors aren't completely used)

Examples of content are of a single data subfile:
Image
The DYUV image of the example file

Group (1) has the start YUV values be directly used to decode the DYUV image.
Group (2) has the exact same first 240 bytes as group (1)... I suspect this block (17280 bytes) contains the QHY data for the DYUV image. For now it's not possible to check this because I don't have that decoder (yet?).
Group (3) has some form of header data of unknown format. The plain text does have some indicators of what it might be related to: play voice info tree cycle canyon.
Group (4) has audio data with an 8 byte header containing (00 00 09 00) and can be decoded using the ADPCM decoder.
Group (5) has some form of header data of unknown format. Plain text found: sp_desc sp_groups sp_cast
Group (6) has a header (I've figured out the format) followed by sprite data (also of known format).
Group (7) has some form of header data of unknown format. Plain text found: SWITCH BRIDGE
Group (8) has the sprites' CLUT palette with a 4 byte header (00 00 01 00). It is unindexed and has 256 color entries.
Group (9) has some form of header data of unknown format. Plain text found: offsets
Group (10) has audio data similar to group (4), but the sound is incomplete: soundgroups stop halfway the last sector payload

So groups 2,3,5,7,9 are of unknown use.
Groups 1 and 8 are used for DYUV and RL7.
Groups 4 and 10 are audio.
Leaving group 6:

The following is the actual C# code used to extract the sprite data. This data still needs to be decoded.

Code: Select all

IList<int> lstStartOffsetsSubSub = new List<int>();
IList<string> lstStartOffsetsSubSubName = new List<string>();

IList<int> lstStartOffsetsMain = new List<int>();
int readOffsetMain = readOffset;
{
    int nrFiles = Parser.ArrayToInt(Functions.GetNBytes(dataData, readOffsetMain, 4)); readOffsetMain += 4;

    lstStartOffsetsMain.Add(Parser.ArrayToInt(Functions.GetNBytes(dataData, readOffsetMain, 4))); readOffsetMain += 4;
    int sizeHeader = Parser.ArrayToInt(Functions.GetNBytes(dataData, readOffsetMain, 4)); readOffsetMain += 4;
    lstStartOffsetsMain[0] += sizeHeader;
    Console.WriteLine("M[" + (readOffsetMain-4).ToString("X") + "] = " + sizeHeader.ToString("X"));

    for (int k = 1; k < nrFiles; k++)
    {
        int startOffsetMain = Parser.ArrayToInt(Functions.GetNBytes(dataData, readOffsetMain, 4)); readOffsetMain += 4;
        Console.WriteLine("M[" + (readOffsetMain-4).ToString("X") + "] = " + startOffsetMain.ToString("X"));
        lstStartOffsetsMain.Add(startOffsetMain);
    }
}

for (int k = 0; k < lstStartOffsetsMain.Count(); k++)
{
    Console.WriteLine(" ");
    Console.WriteLine("M: " + lstStartOffsetsMain[k].ToString("X"));
    IList<int> lstStartOffsetsSub = new List<int>();
    int readOffsetSub = readOffset + lstStartOffsetsMain[k];
    {
        int nrFiles = Parser.ArrayToInt(Functions.GetNBytes(dataData, readOffsetSub, 4)); readOffsetSub += 4;

        int unk = Parser.ArrayToInt(Functions.GetNBytes(dataData, readOffsetSub, 4)); readOffsetSub += 4;
        int sizeHeader = Parser.ArrayToInt(Functions.GetNBytes(dataData, readOffsetSub, 4)); readOffsetSub += 4;
        if (nrFiles > 0)
        {
            lstStartOffsetsSub.Add(sizeHeader);
            Console.WriteLine("S[" + (readOffsetSub - 4).ToString("X") + "] = " + sizeHeader.ToString("X"));
        }
        for (int l = 1; l < nrFiles; l++)
        {
            int startOffsetSub = Parser.ArrayToInt(Functions.GetNBytes(dataData, readOffsetSub, 4));
            Console.WriteLine("S[" + readOffsetSub.ToString("X") + "] = " + startOffsetSub.ToString("X"));
            lstStartOffsetsSub.Add(startOffsetSub);
            readOffsetSub += 4;
        }
    }

    for (int l = 0; l < lstStartOffsetsSub.Count(); l++)
    {
        Console.WriteLine("S: " + lstStartOffsetsSub[l].ToString("X"));
        //IList<int> lstStartOffsetsSubSub = new List<int>();
        int readOffsetSubSub = readOffset + lstStartOffsetsMain[k] + lstStartOffsetsSub[l];
        {
            int nrFiles = Parser.ArrayToInt(Functions.GetNBytes(dataData, readOffsetSubSub, 4)); readOffsetSubSub += 4;
            Console.WriteLine("S: nr = " + nrFiles);
            int unk = Parser.ArrayToInt(Functions.GetNBytes(dataData, readOffsetSubSub, 4)); readOffsetSubSub += 4;
            int sizeHeader = Parser.ArrayToInt(Functions.GetNBytes(dataData, readOffsetSubSub, 4)); readOffsetSubSub += 4;
            if (nrFiles > 0)
            {
                lstStartOffsetsSubSub.Add(sizeHeader + lstStartOffsetsMain[k] + lstStartOffsetsSub[l]);
                lstStartOffsetsSubSubName.Add(String.Format("{0}-{1}-{2}", k, l, 0));
                Console.WriteLine("SS[" + (readOffsetSubSub - 4).ToString("X") + "] " + lstStartOffsetsSubSub[lstStartOffsetsSubSub.Count() - 1].ToString("X"));
            }
            for (int m = 1; m < nrFiles; m++)
            {
                int startOffsetSubSub = Parser.ArrayToInt(Functions.GetNBytes(dataData, readOffsetSubSub, 4));
                lstStartOffsetsSubSub.Add(startOffsetSubSub + lstStartOffsetsMain[k] + lstStartOffsetsSub[l]);
                lstStartOffsetsSubSubName.Add(String.Format("{0}-{1}-{2}", k, l, m));
                Console.WriteLine("SS[" + readOffsetSubSub.ToString("X") + "] " + lstStartOffsetsSubSub[lstStartOffsetsSubSub.Count()-1].ToString("X"));
                readOffsetSubSub += 4;
            }
        }
    }
}

Console.WriteLine(" ");
Console.WriteLine("sprite offsets: "+String.Join(" ", lstStartOffsetsSubSub.Select(x => x.ToString("X")).ToArray()));

for (int j = 0; j < lstStartOffsetsSubSub.Count(); j++)
{
    int spriteLength = Parser.ArrayToInt(Functions.GetNBytes(dataData, readOffset + lstStartOffsetsSubSub[j], 4)) - 4;
    byte[] spriteData = new byte[spriteLength];
    System.Buffer.BlockCopy(dataData, readOffset + lstStartOffsetsSubSub[j] + 4, spriteData, 0, spriteLength);
    //dctSpriteData.Add(String.Format("subfile {0}-spriteData-{1:d2}-{2:X4}.dat", i, j, lstStartOffsetsSubSub[j]), spriteData);
    dctSpriteData.Add(String.Format("subfile {0}-spriteData-{1}.dat", i, lstStartOffsetsSubSubName[j]), spriteData);
}

//move subfile readpointer to end of last sprite data
if (lstStartOffsetsSubSub.Count() > 0)
{
    readOffset += lstStartOffsetsSubSub[lstStartOffsetsSubSub.Count() - 1]; //start of last sprite
    readOffset += Parser.ArrayToInt(Functions.GetNBytes(dataData, readOffset, 4));  //length of last sprite
}

//move subfile readpointer to next sector payload (multiples of 0x800)
readOffset += (0x800 - readOffset % 0x800);
And one-line explanations of the two helper functions (ArrayToInt and GetNBytes).

Code: Select all

public static int ArrayToInt(byte[] arr) {...} //Convert a byte array to a single integer
public static byte[] GetNBytes(byte[] input, int start, int nrBytes) {...} //Get 'nrBytes' bytes from array 'input' with offset 'start'
The Console.WriteLine functions output to the command window, and is visible to the user.
The sprite block started at offset 0xC800 (readOffset).
For the above image I got the following feedback:

Code: Select all

M[C808] = 14
M[C80C] = 1B68
M[C810] = 255C

M: 14
S[C81C] = C
S: C
S: nr = 1
SS[C828] = 2C

M: 1B68
S[E370] = C
S: C
S: nr = 4
SS[E37C] = 1B8C
SS[E380] = 1E00
SS[E384] = 2074
SS[E388] = 22E8

M: 255C
S[ED64] = 10
S[ED68] = 320
S: 10
S: nr = 1
SS[ED74] = 2578
S: 320
S: nr = 1
SS[F084] = 2888

sprite offsets: 2C 1B8C 1E00 2074 22E8 2578 2888
Below the grouping of the Main header, the Sub headers, the SubSub headers, and the sprites (two colors).
Each sprite blob starts with 4 bytes that contain the length of the actual sprite (excluding those 4 bytes).
Image
The full datablob containing sprites and their headers

The sprites are uniquely encoded:

Code: Select all

public static Asset ZeldasAdventure(byte[] spriteData, string fileName, Dictionary<int, byte[]> colorTable)
{
    //reserve screen-sized byte array (a single sprite will never be this big) and prefill with 'background'
    byte[] imageData = new byte[384 * 240];
    for (int j = 0; j < 384 * 240; j++)
    {
        imageData[j] = 0xFF;
    }
    
    //decode pixel shifted sprite
    int i = 0;
    int writeOffset = 0;
    while (i < spriteData.Count())
    {
        int skipLength = Parser.ArrayToInt(Functions.GetNBytes(spriteData, i+=2, 2));
        int nrBytes = Parser.ArrayToInt(Functions.GetNBytes(spriteData, i+=2, 2)) * 4;

        writeOffset += skipLength;
        System.Buffer.BlockCopy(spriteData, i, imageData, writeOffset, nrBytes);
        writeOffset += nrBytes;
        i += nrBytes;
    }
    
    //determine number of rows required for sprite and copy sprite to smaller datablob
    i = writeOffset + (384 - writeOffset % 384);
    byte[] sprite = new byte[i];
    System.Buffer.BlockCopy(imageData, 0, sprite, 0, i);
    
    //determine width and height of sprite (and extract datablob of these dimensions)
    int width = 0;
    int height = 0;
    imageData = Functions.ExtractSprite(@sprite, 384, ref width, ref height);
    
    //decode CLUT of sprite
    return new Asset(fileName, CLUT(imageData, width, height, colorTable));
}
Helper function 'Functions.ExtractSprite' loops through each byte in the array and checks if the value isn't background (0xFF).
The data is processed as follows:
Image
The first 304 bytes of the sprite datablob, showing the header, subheader, subsubheader, and first 12 lines of the bridge sprite

Upon completion, the width and height of the content is determined and the sprite is extracted from the blob.
Image
The first decoded sprite in a 384x240 byte blob

The highlighted area is the extracted sprite (156 x 111 pixels) ready to be CLUT decoded.
There is just one step left to show:
In the above image you can notice that the bridge sprite data is quite.. blocky. That's because the pixel shifting works with groups of 4 pixels.
This is resolved by overriding several colors in the color table. This is also why I used a grey background:

Code: Select all

lstColorTable[0][0xFF] = new byte[3] { 50, 50, 50 };
lstColorTable[0][0x80] = new byte[3] { 50, 50, 50 };
lstColorTable[0][0xA0] = new byte[3] { 50, 50, 50 };
lstColorTable[0][0xC0] = new byte[3] { 50, 50, 50 };
lstColorTable[0][0xE0] = new byte[3] { 50, 50, 50 };
The result:
Image

Processing the rest of the sprite blob yields the following sprites:
Image Image Image Image
Image
Image

Not every sprite requires every color override, so a few have grey pixels where they shouldn't be.
Image

The whole extraction process for 'over.rtf' (from ISO to 5193 assets) takes roughly 30 seconds. The time to extract all 2952 assets from 'under.rtf' is only 15 seconds.

If anything it unclear I will elaborate more.

This was probably the biggest post I've ever made in any forum

-- edit 23-12-'19
Fixed image URLs due to site revamp.

-- edit 18-08-'20
Fixed image URL and link URL (to overworld map)
Last edited by Shikotei on Mon Dec 23, 2019 11:16 am, edited 1 time in total.

Post Reply