Project Perfect Mod Forums
:: Home :: Get Hosted :: PPM FAQ :: Forum FAQ :: Privacy Policy :: Search :: Memberlist :: Usergroups :: Register :: Profile :: Log in to check your private messages :: Log in ::


The time now is Thu Mar 28, 2024 1:38 pm
All times are UTC + 0
Digest in maps
Moderators: Community Tools Developpers
Post new topic   Reply to topic Page 1 of 1 [24 Posts] Mark the topic unread ::  View previous topic :: View next topic
Author Message
RP
Commander


Joined: 12 Jul 2012
Location: Mapping God Heaven

PostPosted: Tue Oct 14, 2014 12:39 pm    Post subject:  Digest in maps Reply with quote  Mark this post and the followings unread

I know this perhaps should not go here, but I'm asking it for both Tiberian Sun and Red Alert 2.

How is the [Digest] section calculated in map files?
I need to know this to create valid maps for multiplayer.

From what I've read, it is used to validate whether 2 players have the exact same map.
If they don't the game would throw an error "Scenario is corrupt" or something.
Removing [Digest] also prevents the validation from happening, but it is safer when it exists.

Please read;
I don't want to be pointed to FA2 because "it does it for you",
I need to know how I can 'manually' create a valid Digest value.

_________________


Mental Omega 3.0 Mission creator - Creator of FinalOmega: APYR 3.0 Map Editor

/ppm/'s stupidity

Back to top
View user's profile Send private message
Lin Kuei Ominae
Seth


Joined: 16 Aug 2006
Location: Germany

PostPosted: Thu Oct 16, 2014 9:50 am    Post subject: Reply with quote  Mark this post and the followings unread

I would assume it's some kind of hash value of the whole map.

The question is, if the game uses the same algorithm as FA to create this value and then checks if the Digest hash and the calculated maps hash are the same
or
if it only checks the hashs of the 2 maps and the Digest hashs (but not map hash against Digest hash), to see if they are different.

In the second case, you should be able to simple write there your own hash value.
In the first case, you would indeed need the algorithm the game uses to create the Digest value manually.


A simple test would be
-create a map
-write in the Digest only something like 1=2wingdangdudel4=
-give the map a partner
-change in your map something (like remove a tree)
then start the game

If it starts, you know that the manually set hash value isn't working.

_________________
SHP Artist of Twisted Insurrection:  Nod buildings

Public SHPs
X-Mech Calendar (28 Mechs for GDI and Nod)
5 GDI, 5 Nod, 1 Mutant, 1 Scrin unit, 1 GDI building

Tools
Image Shaper______TMP Shop______C&C Executable Modifier

Back to top
View user's profile Send private message
CCHyper
Defense Minister


Joined: 07 Apr 2005

PostPosted: Thu Oct 16, 2014 1:41 pm    Post subject: Reply with quote  Mark this post and the followings unread

I might have old information, but if you check the digest of FinalSun produced maps, I think its a generic one/taken from a large map.

Back to top
View user's profile Send private message
RP
Commander


Joined: 12 Jul 2012
Location: Mapping God Heaven

PostPosted: Thu Oct 16, 2014 4:13 pm    Post subject: Reply with quote  Mark this post and the followings unread

@CCHyper: This is not the case, it's not 1 for all.

@LKO:
I have tested it with someone on CnCNet.
If his [Digest] is different than mine; game says not all players have the map
If his [Digest] is the same as mine, but I modified the map (different map hash?); game says not all players have the map

I don't know how this works in XWIS or non-LAN games, it seems that if either of the hashes doesn't match,
the game will transfer the host's map to everyone.

So:
Mine             His
[Digest] =/= [Digest]
[Map] == [Map]
Game detects difference, host sends over the map.

Mine             His
[Digest] == [Digest]
[Map] =/= [Map]
Game detects difference, host sends over the map.

Mine             His
[Digest] == [Digest]
[Map] == [Map]
But: [Digest] =/= [Map]
Game detects difference, host sends over the map.


I have not found any relation between [Digest] and the map hash other than above.
I think it's quite hard to properly test because this has no effect on singleplayer maps.

One could draw the preliminary conclusion that the map hash and [Digest] are actually compared and thus need to be valid.

One note:
If [Digest] is not present in the map file, it will not (at least for custom mp maps) be displayed in the map list.

_________________


Mental Omega 3.0 Mission creator - Creator of FinalOmega: APYR 3.0 Map Editor

/ppm/'s stupidity

Back to top
View user's profile Send private message
Bittah Commander
Defense Minister


Joined: 21 May 2003
Location: The Netherlands

PostPosted: Thu Oct 16, 2014 6:40 pm    Post subject: Reply with quote  Mark this post and the followings unread

Is it the client or the actual game that says not all players have the map? In other words; do you get to see the ingame loading screen before you get the error or is an error displayed before the game even starts?

If the error comes from the client, then the issue obviously isn't with the game, but the client and you're best off contacting FunkyFr3sh about this (since he made and maintains the client).

_________________

Back to top
View user's profile Send private message Send e-mail ModDB Profile ID YouTube User URL Facebook Profile URL
RP
Commander


Joined: 12 Jul 2012
Location: Mapping God Heaven

PostPosted: Thu Oct 16, 2014 7:08 pm    Post subject: Reply with quote  Mark this post and the followings unread

It's not CnCNet 5, forgot to mention that, sorry. The talk about hashes might be confusing, too, I just copied LKO's terminology.
It's the game that says it though, before loading any map.

_________________


Mental Omega 3.0 Mission creator - Creator of FinalOmega: APYR 3.0 Map Editor

/ppm/'s stupidity

Back to top
View user's profile Send private message
E1 Elite
General


Joined: 28 May 2013

PostPosted: Thu Oct 16, 2014 7:27 pm    Post subject: Reply with quote  Mark this post and the followings unread

How did you test 3rd condition? Have you copied a Digest from another map into the current
one and transferred to the other player?

Finalsun/FA2 does not modify an existing digest value, when the map gets modified. If those
works then the idea may be is to keep it unique per file, not to decode and verify the checksum.
When the digest is removed and then saving the map in FS/FA2, it creates a new one different
from the old one everytime. Checksum are also not expected to be inside the same file itself.

Back to top
View user's profile Send private message
RP
Commander


Joined: 12 Jul 2012
Location: Mapping God Heaven

PostPosted: Thu Oct 16, 2014 8:01 pm    Post subject: Reply with quote  Mark this post and the followings unread

I changed the Digest of the file after I created / saved it to some horseshit line "lololderp".

We both used the same Digest value, so the map files themselves were identical.
The Digest value wasn't equal to what the game generates, so it detected changes.
Maybe when it finds that out it just assumes they're not the same (the maps) because the Digest has been tampered with.

I don't know how FA2 does it, it seems a tad bit inconsistent...

_________________


Mental Omega 3.0 Mission creator - Creator of FinalOmega: APYR 3.0 Map Editor

/ppm/'s stupidity

Back to top
View user's profile Send private message
E1 Elite
General


Joined: 28 May 2013

PostPosted: Thu Oct 16, 2014 8:35 pm    Post subject: Reply with quote  Mark this post and the followings unread

Format looks like 1= and then 27 alphanumeric chars (upper + lower and also + and /) followed by
= at the end.

Should have tried from another map or by removing the entry and saving it in FA2 to regenerate.
And then transferring the map file to the other player (not from in-game). It could turn out to be a
randomly generated entry to a 28 char format to make it unique.

Back to top
View user's profile Send private message
CCHyper
Defense Minister


Joined: 07 Apr 2005

PostPosted: Thu Oct 16, 2014 8:42 pm    Post subject: Reply with quote  Mark this post and the followings unread

I am taking a guess that the "=" is a artefact from Base64 engine processing a " " (space).

Back to top
View user's profile Send private message
Blade
Cyborg Commando


Joined: 23 Dec 2003

PostPosted: Thu Oct 16, 2014 10:12 pm    Post subject: Reply with quote  Mark this post and the followings unread

Mix files appear to use the SHA1 digest for their checksums, at a guess I would suggest that this may be a base64 encoded SHA1 digest of the map before it has the digest section added.

Back to top
View user's profile Send private message
RP
Commander


Joined: 12 Jul 2012
Location: Mapping God Heaven

PostPosted: Thu Oct 16, 2014 10:44 pm    Post subject: Reply with quote  Mark this post and the followings unread

CCHyper told me it is a Base64'd SHA1 hash.
Based on that information I implemented methods to generate and write a digest.

Basically what it does is create a random hash, write 0x80 at the end and then Base64 it.
The last byte, 0x80, is always translated to '=' in Base64.

That's also why you quite often see '=' or '==' at the end of Overlay(Data)Pack:
It uses Format80 and the terminator/last byte is usually 0x80, used for the commands the compression uses.

I have tested the generated Digest on my PC and laptop, both use the same configuration (using Dropbox), so everything is the same.
The game (on PC, as host) did not tell me that Laptop did not have the map, it seems it saw both maps as a proper map.

_________________


Mental Omega 3.0 Mission creator - Creator of FinalOmega: APYR 3.0 Map Editor

/ppm/'s stupidity

Back to top
View user's profile Send private message
Blade
Cyborg Commando


Joined: 23 Dec 2003

PostPosted: Fri Oct 17, 2014 7:56 am    Post subject: Reply with quote  Mark this post and the followings unread

If the hash is random, what other information gets sent to decide if the maps are not the same then when the digest matches but the contents of one are altered? File length? Wouldn't it be possible to adjust for example tech levels on units whose tech level is declared in a map that way (turn an 11 or -1 into 10 for example) such that you could build but opponent couldn't?

Back to top
View user's profile Send private message
CCHyper
Defense Minister


Joined: 07 Apr 2005

PostPosted: Fri Oct 17, 2014 8:55 am    Post subject: Reply with quote  Mark this post and the followings unread

RULES and ART crc's are generated and compared, map file size/checksum. Ill look into if there are any more.

Back to top
View user's profile Send private message
Blade
Cyborg Commando


Joined: 23 Dec 2003

PostPosted: Mon Oct 20, 2014 8:01 am    Post subject: Reply with quote  Mark this post and the followings unread

On a slightly tangental topic, does anyone know what the WW format80 encoder does more aggressively than the code from XCC? If you convert for example a CPS image from RA to pcx and back the image is always slightly larger. I assume that at least edwin has a WW format80 encoder built into it since it needs to encode to it for the pack sections in a map?

Back to top
View user's profile Send private message
CCHyper
Defense Minister


Joined: 07 Apr 2005

PostPosted: Mon Oct 20, 2014 9:12 am    Post subject: Reply with quote  Mark this post and the followings unread

Blade wrote:
On a slightly tangental topic, does anyone know what the WW format80 encoder does more aggressively than the code from XCC? If you convert for example a CPS image from RA to pcx and back the image is always slightly larger. I assume that at least edwin has a WW format80 encoder built into it since it needs to encode to it for the pack sections in a map?


What is format40/80 and what sections use this compression? I'll look into it.

Back to top
View user's profile Send private message
Blade
Cyborg Commando


Joined: 23 Dec 2003

PostPosted: Mon Oct 20, 2014 9:48 am    Post subject: Reply with quote  Mark this post and the followings unread

format80 is the compression format used for most images in the game, http://eob.wikispaces.com/Format+80 has an overview of the format on disk and how it is decoded. MapPack and OverlayPack are I believe first format80 compressed followed by base64 encoded to be recorded in the map ini file. http://cnc.wikia.com/wiki/Red_Alert_File_Formats_Guide briefly mentions how its used in the *Pack sections when it discusses base64 decoding.
Format40 is used in shp and wsa files, though I don't believe XCC even tries to use it in shp files, just format80 compressing all frames to save processing time testing which format is better. It is something like a compressed xor between the target image and some reference image.
As far as reference code for format40, I guess the only tool that would have WW code in it would be the recently leaked shp builder tool that leaked with the RA2 infantry models.

Back to top
View user's profile Send private message
CCHyper
Defense Minister


Joined: 07 Apr 2005

PostPosted: Mon Oct 20, 2014 11:03 am    Post subject: Reply with quote  Mark this post and the followings unread

Format40 and Format80 are the incorrect names, it looks very much like it is LCW as these sections are parsed through the LCWPipe/Straw. (Straw = Read/Get, Pipe = Write/Put).

EDIT: Yup, its definitely LCW compression/decompression. CPS file uses a version that seems to have been written in ASM (leftover from DOS perhaps).

EDIT2: I can not find anything related to LCW on the internet, but LZW appears a lot. Does anybody know of the format/s?

Back to top
View user's profile Send private message
RP
Commander


Joined: 12 Jul 2012
Location: Mapping God Heaven

PostPosted: Mon Oct 20, 2014 2:32 pm    Post subject: Reply with quote  Mark this post and the followings unread

Did you check whether it used all 5 command bytes for compression, Blade?

It could be like in my implementation, only using 0 (raw copy) and 4 (copy byte X n times).
It's the quickest way to get 'proper' compression.
The other command bytes take some time to figure out for writing because you have to look through the destination.
Writing the overlay packs is really easy with these 2 commands because about 80% is always 255.

_________________


Mental Omega 3.0 Mission creator - Creator of FinalOmega: APYR 3.0 Map Editor

/ppm/'s stupidity

Back to top
View user's profile Send private message
Blade
Cyborg Commando


Joined: 23 Dec 2003

PostPosted: Mon Oct 20, 2014 2:59 pm    Post subject: Reply with quote  Mark this post and the followings unread

My code is largely cribbed from XCC at the moment though I wrote an equivalent for part of it that was still in assembly to make it a touch more portable. XCC looks like it uses all the commands (XCC actually has variants of format80 that do as you suggest) and it still generates format80 data larger than the original files. I could log the decisions the decoder and encoder make as they cycle a file, but there are thousands of bytes in a file and I may loose the will to live looking for where the encoder made a different choice and then working out why.

Back to top
View user's profile Send private message
RP
Commander


Joined: 12 Jul 2012
Location: Mapping God Heaven

PostPosted: Mon Oct 20, 2014 4:29 pm    Post subject: Reply with quote  Mark this post and the followings unread

I wouldn't worry about it that much.
The only difference is that the XCC variant takes up a bit more diskspace, in the end every SHP is equally large regardless of its compression.

Don't mind logging, I've been through it while working on the encoder, it's a bitch #Tongue

_________________


Mental Omega 3.0 Mission creator - Creator of FinalOmega: APYR 3.0 Map Editor

/ppm/'s stupidity

Back to top
View user's profile Send private message
ApolloTD
Commander


Joined: 19 Nov 2003

PostPosted: Tue Oct 21, 2014 8:00 am    Post subject: Reply with quote  Mark this post and the followings unread

Blade wrote:

As far as reference code for format40, I guess the only tool that would have WW code in it would be the recently leaked shp builder tool that leaked with the RA2 infantry models.


Sorry to say this seems false, that shp tool will only encode the SHP (TS) format (C1 & C3 RLE) that does not use the format40/format80 methods that original SHP (TD-RA1) shps used thus can't be used to encode them.

More chance with maybe RA1 map editor...

Back to top
View user's profile Send private message
Blade
Cyborg Commando


Joined: 23 Dec 2003

PostPosted: Tue Oct 21, 2014 11:57 am    Post subject: Reply with quote  Mark this post and the followings unread

I didn't realise that the TS SHP format didn't use format80/40. Is the TS shp format and compression documented somewhere?

Back to top
View user's profile Send private message
CCHyper
Defense Minister


Joined: 07 Apr 2005

PostPosted: Tue Nov 18, 2014 9:53 am    Post subject: Reply with quote  Mark this post and the followings unread

From Denzil Long directly...

Denzil Long wrote:
"LCW was a custom Westwood LZ sytle compression. LC = Louis Castle, W = Westwood (Louis Castle wrote this compression code the the VERY early garage days."


Now we know what the name is, who wrote it and what style it is based on!

EDIT:
Louis Castle wrote:
Wow. Yes, I wrote it.  
Lempel - Castle - Welch.  A bit tough to explain in a message but I'll give it a shot.


The plot thickens!

Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic Page 1 of 1 [24 Posts] Mark the topic unread ::  View previous topic :: View next topic
 
Share on TwitterShare on FacebookShare on Google+Share on DiggShare on RedditShare on PInterestShare on Del.icio.usShare on Stumble Upon
Quick Reply
Username:


If you are visually impaired or cannot otherwise answer the challenges below please contact the Administrator for help.


Write only two of the following words separated by a sharp: Brotherhood, unity, peace! 

 
You can post new topics in this forum
You can reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © phpBB Group

[ Time: 0.1648s ][ Queries: 11 (0.0091s) ][ Debug on ]