When trying to renew an old game, it's important to have a working thing relatively soon - otherwise, there's a great chance of your work getting wasted. So it's good you've turned towards RE ideas with potential of having full game usable without having it fully remade.
The DLL approach taken in KeeperFX:
- Works only if your executable has a relocation table - later versions of both Visual C and Watcom started to remove these. Without relocation table, the DLL will have ability to load only at one specific address in the virtual address space of the game - which means sometimes it will work, and sometimes not, depending on versions of other DLLs and their base addresses.
- Requires you to do your C rewrite from
main() and outer layers, towards the central functions which you will likely want to modify. Unless you will mix the DLL approach with some binary patching.
The ASM approach taken in SW Port:
- Allows you to modify / rewrite to C any function at any time - no need to start with the
main(). Because if you have the whole game disassembled, you can do calls back and forth between C and ASM.
- Allows you to port to other OSes without a full rewrite - ie, SW Port supports Linux and Mac OS (as long as they run on i386 arch with 32-bit user space)
- The 32-bit user space restriction will be easier to overcome in case OSes would stop supporting that. While it would require assembly changes, it is doable without a full rewrite.
- Achieving the first runnable state this way is a bit more labour-intense that with the DLL approach.
There's also a 3rd approach - you'd find it in Magic Carpet HD. That project author just used Ida Pro to generate C code out of the whole game, and then continued fixing the generated C until it started working. I consider this approach very risky - you need to put a lot of effort to make the game work. Also, the generated C code has a very low quality this way, and it may negatively influence further progress of the project. Plus, the brute-force C conversion introduced a lot of bugs.
General points:
- For both methods, you will find useful tools already developed in the respective projects (ie. export table updated for DLL approach, and C-to-watcom bi-directional calling wrapper for ASM approach).
- The DLL approach required some specific conditions to work, while disassembling the whole game is a general approach which can be used for any game.
For accessing global variables and functions between original game and remade C code:
- In KeeperFX, the variables and functions from within DLL were just exported with a prefix (
_DK_ specifically), and could be easily used outside, like any DLLIMPORT. Calling convention change was handled outside of the DLL, as an ASM block within C. Not sure if there was a helper macro to hide the details.
- In SW Port, any variables/functions which are used in assembly but also in C, need to be properly defined (as
.global with decor) - the port source has macros for that already made. And for C-to-Watcom conversion, there is a tool which generates a wrapper C function - you need to add the function to the wrapper, and then you call it normally in C. The wrapper will get arguments from _cdecl call and put them into proper registers (you have to define the arguments when adding function to the wrapper).
Basically:
- C variables are decorated, so to access ASM-defined vars in C you need to add that decoration to your ASM vars.
- For C-to-Watcom calls, you get up to 4 vars from stack and place it into registers, and that's it.
- For Watcom-to-C calls, you have to modify the assembly to just use stack and no registers.
- While juggling the stack and registers, you need to mind clobbering/spoiling value within registers by functions. Watcom convention says - any register which was used as parameter, can have its value overwritten.
