New patching format, UPS

Discussion of hacking, editing and developments in Advance Wars games.
Remember, no Rom Requests
User avatar
RadioShadow
Rank: Holding the Mega Drive controller
Location: UK, England

New patching format, UPS

Post by RadioShadow » Fri Apr 04, 2008 3:22 am

At the moment, the current format for release hack patches is by using IPS but the problem with IPS though, is that it can only patch files that are 16 MB in size or less. I'm aware this news isn't very exiting but from what I read, the patching program is really good and I may update the current patches to use UPS.

No Mac support unfortunately but there is XP, Vista (THANK GOD) and Linux.

Download here.
The creator byuu says:

Okay, it's been talked about forever. Now it's finally ready for prime time.

Background:

The goal of UPS is different from NINJA / xdelta / etc. It is designed to be a direct replacement for IPS. The difficulty of implementing it is equal to IPS (it has trickier pointers, but no RLE magic.)

The spec for UPS supports any file size, though the binary is currently limited to 4GB files due to libc limitations. But yes, thanks to a condensed pointer encoding method, the file size can be infinite. UPS is completely future proof in that regard.

Now, UPS is designed for patching files, not performing black magic voodoo. It won't perform advanced clipping methods to move data around and restructure files like xdelta will. However, it's possible for anyone to implement UPS, whereas there has only ever been one xdelta implementation. Guess why. If you need this kind of magic, then you probably want to use xdelta or bsdiff instead.

UPS also doesn't try and support every last console on the planet. The general rule is to make patches without headers when they aren't needed (eg SNES copier), and with headers when required (eg iNES.) The platform-specific detection will be part of NINJA.

UPS doesn't compete with NINJA, they will complement each other. NINJA3 will use UPS internally as the raw patch data, and will handle detection and support for each individual system, as needed. NINJA3 is to OGG as UPS is to Vorbis, for instance.

UPS doesn't waste time with compression tricks such as RLE coding. The reason is because ZIP, 7-Zip, etc etc have always and will always do it better. It's better to just have a bigger patch that gets packed down more by compression. Most emulators support patches inside ZIP files anyway. This has the added benefit of making the file format easier for others to implement.

Finally, UPS is a finalized spec. With roughly a year to review it, I've still not felt the need to change it in any way. So even though the UPS patcher and library have first release version numbers (0.xx), the patches created will work with all future versions. The spec is unchanging. Don't even ask. I've posted here for a good year asking for feedback, this is set in stone now.

Now, if you decide to make your own patcher, make sure that the output from your patcher is 1:1 identical with the reference library. If it has even one bit different, it's not a compliant patcher, and this is a license violation. Otherwise, my reference code is public domain. I need to do this because I don't want the format splintering like all the IPS "extensions". Let's not start that mess. UPS doesn't need any extensions, it does everything it needs to do right from the start.

Still, as an absolute failsafe, and to make the sig dword-aligned, the signature is "UPS1". If you want to modify the format, don't use "UPS2", or anything with "UPS" at the start, please. Change the name to something else.

---

Advantages:
- simple file format, easy for anyone to implement.
- automatic bi-directional patching. The same patch can both patch and unpatch a game.
- CRC32 checksums on the original, modified and patch files guarantees patches will not apply to the incorrect games. We use CRC32 as this is designed to test integrity, not to prevent malicious checksum conflicts. Odds of a false positive are 1:4 billion with CRC32, and CRC32 is 100x easier to implement than eg MD5SUM, etc. Very important for others implementing this algorithm.
- infinite file sizes. No more 16MB limitation as with IPS.
- Windows / Linux GUI patchers, core library written in ISO C++9x.
- all of this is public domain

---

byuu.cinnamonpirate.com/files/ups_v03.zip (Windows and Linux binaries)
byuu.cinnamonpirate.com/files/ups_v03.tar.bz2 (source)

The executables themselves have simplistic UIs, I was aiming for something as easy as LunarIPS.

Linux GUI:
Image

Windows GUI (yes, it supports XP / Vista themes, too):
Image

Both Windows and Linux binaries work on the command line, too. Use:
ups --create original.bin modified.bin patch.ups
ups --apply input.bin output.bin patch.ups

Run directly to get the UI.

---

The spec itself:

Code: Select all

/*****
 * file format:
 *****
 * [4] signature ("UPS1")
 * [V] X file size
 * [V] Y file size
 * [?] {
 *   [V] relative difference offset
 *   [?] X ^ Y
 *   [1] 0x00 terminator
 * }
 * [4] X file crc32
 * [4] Y file crc32
 * [4] Z file crc32
 *****/
If you want the more detailed spec, see the source code, libups/libups.hpp file. It contains all specific rules. And libups/libups.cpp has the reference implementation with the pointer length encoding / decoding algorithm. It's simple enough.

---

Lastly, I've taken the input of mostly everyone here into mind when making this, except obviously when conflicting. Most importantly, I'm of the same mind as Nightcrawler for how to replace IPS: simple format, simple tools, simple implementation. Simple all around. Just address IPS' limitations.

Now, if we're ever going to get rid of IPS, we need to rally behind a format. If you guys don't want UPS, fine. But we need to pick something. I hope it'll be UPS, of course. What we need is for as many people as possible to push a replacement format and get it out there. Otherwise we'll be stuck with IPS forever :/

User avatar
Xenesis
Tri-Star CO
Tri-Star CO
Rank: Hydrocarbon Inspector
3DS Code: 2535-4646-7163
Location: 0x020232DD

Re: New patching format, UPS

Post by Xenesis » Fri Apr 04, 2008 11:56 pm

Uh, RS.

You do realise that this is basically only relevant to those of us who want to patch N64/Playstation/Megasized DS Games right?

For hacking GBA games, unless you're retarded IPS is the best - Nothing's gonna be 16+mb for the most part and it can be used on anything.
IST wrote:Even the worst individual needs to discover the joys of a chicken statue that is also a pregnant blonde housewife.

User avatar
RadioShadow
Rank: Holding the Mega Drive controller
Location: UK, England

Re: New patching format, UPS

Post by RadioShadow » Sat Apr 05, 2008 2:23 am

It does mean you could make the rom 32MB (biggst size for a GBA rom) and creating patches would not be a problem. But if you do plan to hack the Advance Wars DS roms (which are 32mb), to make a patch, you're going to need to use UPS.

But as you say Xen, the IPS Patches are fine for the GBA. Just UPS looks like it preforms a better job overall.

And someone ported it to work on a Mac since they rule! :D
daemoncollector says:
Bin:
http://opennestopia.staulkor.com/UPS.zip

Src (Including XCode project):
http://opennestopia.staulkor.com/UPS_Src.zip

This is the OSX UPS patcher I wrote based on byuu's reference implementation, tested and verified working.

Who is online

Users browsing this forum: Multivac [Bot] and 0 guests