Project Perfect Mod Forums
:: Home :: Get Hosted :: PPM FAQ :: Forum FAQ :: Privacy Policy :: Search :: Memberlist :: Usergroups :: Register :: Profile :: Log in to check your private messages :: Log in ::


The time now is Thu Mar 28, 2024 4:04 pm
All times are UTC + 0
Shapeset, a tool for C&C, Red Alert and Sole Survivor shps.
Moderators: Community Tools Developpers
Post new topic   Reply to topic Page 1 of 1 [11 Posts] Mark the topic unread ::  View previous topic :: View next topic
Author Message
Blade
Cyborg Commando


Joined: 23 Dec 2003

PostPosted: Tue Jun 23, 2015 2:15 pm    Post subject:  Shapeset, a tool for C&C, Red Alert and Sole Survivor shps. Reply with quote  Mark this post and the followings unread

Okay, here is yet another tool for encoding C&C and Red Alert .shp files. Its command line and only works on a sequence of pcx files, so why should you care? Read on...

As I've previously mentioned in other threads, I've been working on code for the LCW and XorDelta compression methods (otherwise known as format80 and format40) that are used mostly in the older C&C games. I've also commented on the poor implementations of LCW encoding and almost total lack of implementations of XorDelta.

For example if you ask XCC to extract and then recompress the temperate advanced guard tower icon from C&C95, it adds 104 bytes to the file because it doesn't implement LCW as well as possible. It gets worse if you do a multiframe file, such as the e3.shp file from Red Alert's lowres.mix. There an XCC compressed file grows by 16511 bytes, almost half the file size again. This is due to a lack of XorDelta where only the differences between frames are efficiently encoded. XCC does have code to encode to encode to XorDelta, but as far as my testing can tell, it doesn't work, resulting in crashes on many data sets. This is also probably why XCC has never supported encoding WSA files which require XorDelta as part of the format.

This may seem like I'm picking on XCC, but the OpenRA.Utility is even worse. It doesn't even implement full LCW making each frame larger than the uncompressed data would be (essentially forcing the algorithms worse case). They really shouldn't offer the tool with TD shp creation capability until they improve their LCW implementation at least. Red Horizon is in a similar situation to XCC where only LCW is used.

Enter Shapeset, its -k switch will apply the same type of XorDelta key framing that the original Westwood tool did. What is more, the efficiency of the XorDelta implementation is slightly better than the Westwood one was so you can shave bytes even off of existing files (though the difference is slight and I don't recommend doing this). This is the tool that you need to run at the end of your work flow to generate the final .shp if you want a properly compressed file that doesn't waste space and memory while the game is running.

Edit:
New version which handles PCX files where the width and pitch don't match. Also, a companion program WSASet which operates in a similar manner but generates WSA files.



shapeset.zip
 Description:

Download
 Filename:  shapeset.zip
 Filesize:  20.79 KB
 Downloaded:  35 Time(s)


Back to top
View user's profile Send private message
Matthias M.
Stealth Laser Trooper


Joined: 15 Jun 2012
Location: Germany

PostPosted: Sat Jun 27, 2015 8:37 am    Post subject: Re: Shapeset, a tool for C&C, Red Alert and Sole Survivor shps. Reply with quote  Mark this post and the followings unread

Can you share your source code and allow us to integrate your algorithm in OpenRA's command line utility?

Back to top
View user's profile Send private message
RP
Commander


Joined: 12 Jul 2012
Location: Mapping God Heaven

PostPosted: Sat Jun 27, 2015 11:29 am    Post subject: Reply with quote  Mark this post and the followings unread

I'm geting in line for this, too Razz

It's written in C++ right?
If you don't like sharing the source code, a .LIB or .DLL would work, too.

_________________


Mental Omega 3.0 Mission creator - Creator of FinalOmega: APYR 3.0 Map Editor

/ppm/'s stupidity

Back to top
View user's profile Send private message
Blade
Cyborg Commando


Joined: 23 Dec 2003

PostPosted: Sat Jun 27, 2015 4:01 pm    Post subject: Reply with quote  Mark this post and the followings unread

I would have no issues with providing these as open source and generally encourage people to do so. However I can't curently share the source for these applications as they have been developed as part of, and link against, a larger project that is in development. Hopefully when complete it will be release as open source (probably GPL v2 and higher) at which point the source for these tools will also be released.

I will discuss with the other members of the project if they are happy to release at least the compression codecs seperately from the main project. The compression code is more C like than C++ and uses a lot of pointer arithmetic and the LCW code could probably do with being rewritten a bit cleaner before any release however.

Failing that, if it should appear that the project isn't going to be completed for some reason, I will release the source needed to build at least these tools at that point.

Back to top
View user's profile Send private message
Blade
Cyborg Commando


Joined: 23 Dec 2003

PostPosted: Fri Aug 28, 2015 10:58 am    Post subject: Reply with quote  Mark this post and the followings unread


Back to top
View user's profile Send private message
ApolloTD
Commander


Joined: 19 Nov 2003

PostPosted: Sat Jan 09, 2016 10:13 am    Post subject: Reply with quote  Mark this post and the followings unread

Well, seems this LCW(?) Compression is entirely useless beyond the size of 256x256 regardless what tool I use, including this one.

I was hoping to save space but no can do as output is screwed into index 0s in the middle for some strange reason.

Last edited by ApolloTD on Sat Jan 09, 2016 4:16 pm; edited 1 time in total

Back to top
View user's profile Send private message
tomsons26lv
Cyborg Artillery


Joined: 30 Dec 2009
Location: Latvia

PostPosted: Sat Jan 09, 2016 3:19 pm    Post subject: Reply with quote  Mark this post and the followings unread

ApolloTD wrote:
Well, seems this LCW(?) Compression is entirely useless beyond the size of 256x256 regardless what tool I use, including this one.

I was hoping to save space but no can do as output is screwed into index 0s in the middle for some strange reason.

TD shps have a resolution limit. I was testing the limits before Shapeset was out publicly. Since it was a while ago don't remember how much exactly anymore, but it is a bit higher then 256x256 but really close to that, beyond it the image gets corrupted.

_________________
Tiberian Dawn, Red Alert, Tiberian Sun ,Red Alert 2,Renegade, Command & Conquer 3,Tiberium and Tiberium Wars and Westwood related image & video archive
https://picasaweb.google.com/113361105083292812413?noredirect=1

Skype live:tomsons26
Don't forget to state who are you otherwise i'll ignore the invite

Back to top
View user's profile Send private message Visit poster's website
ApolloTD
Commander


Joined: 19 Nov 2003

PostPosted: Sat Jan 09, 2016 3:46 pm    Post subject: Reply with quote  Mark this post and the followings unread

tomsons26lv wrote:

TD shps have a resolution limit. I was testing the limits before Shapeset was out publicly. Since it was a while ago don't remember how much exactly anymore, but it is a bit higher then 256x256 but really close to that, beyond it the image gets corrupted.


Well, 288x288 for example not work but I guess this limit is why Westwood abandoned that shp version format then as art needed to be bigger.

This limitation kinda further outlines it has little use apart from TD/RA1 scale use, unfortunate design.

Doesn't VQA encoding use LCW principle too and yet it uses far greater resolutions without issue...

Back to top
View user's profile Send private message
Blade
Cyborg Commando


Joined: 23 Dec 2003

PostPosted: Sun Jan 10, 2016 11:04 pm    Post subject: Reply with quote  Mark this post and the followings unread

Its because that implementation of the LCW compression format changes the format for data greater that 64KiB to use a relative offset for a few of the commands and the data is flagged as being in the new format by starting with a null byte rather than a short copy command. In the original, no image over 320x200 is encoded to LCW and so it doesn't implement the decompression for it. TS does have a decompressor for the extended format in its VQA decoder however.

What game and format are you trying to encode to LCW to save space? If you already have shp files encoded by XCC or something similar then I could modify shapeset not to use the extended format for larger frames which should support the earlier games, but then you would run into a limitation on the largest compressed frame size which is also ~64KiB (stored as an unsigned short).

Back to top
View user's profile Send private message
ApolloTD
Commander


Joined: 19 Nov 2003

PostPosted: Mon Jan 11, 2016 9:27 am    Post subject: Reply with quote  Mark this post and the followings unread

Well, I thought using it in OpenRA since can support both shp types and LCW can compress similar frame artworks better. No doubt ORA only supports it just like originals instead of VQA level support...

That compressed frame size 64 limit might allow up to ??? resolutions that would need experimentation to see if its worthwhile enough.

Back to top
View user's profile Send private message
Blade
Cyborg Commando


Joined: 23 Dec 2003

PostPosted: Mon Jan 11, 2016 3:05 pm    Post subject: Reply with quote  Mark this post and the followings unread

OpenRA does have support for the extended format as a flag passed into the decoder, but a quick look at their source suggests that only the VQA decoder checks for the null byte and then passes the flag to tell it which mode to operate in.

If the TD Shape decoder was updated to check for the null for the start of LCW frames, it should be able to load larger sprites.

It may even ignore the Delta field in the header allowing it to use even larger frames, though they would technically be out of format spec and I may make future versions of ShapeSet error out if it generates a frame that is too large.

Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic Page 1 of 1 [11 Posts] Mark the topic unread ::  View previous topic :: View next topic
 
Share on TwitterShare on FacebookShare on Google+Share on DiggShare on RedditShare on PInterestShare on Del.icio.usShare on Stumble Upon
Quick Reply
Username:


If you are visually impaired or cannot otherwise answer the challenges below please contact the Administrator for help.


Write only two of the following words separated by a sharp: Brotherhood, unity, peace! 

 
You can post new topics in this forum
You can reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © phpBB Group

[ Time: 0.1687s ][ Queries: 13 (0.0109s) ][ Debug on ]