5

Assume I were to be reversing some game which processes all of it's movement client side and have determined some function to be of the type:

bool __thiscall Player::CanJump(Player *this)

that I have determined to be a member of the Player object's vtable. Now lets assume I wanted to edit that object's vtable to point to my own dll injected implementation so that I could jump whenever I wanted. I could declare it as

bool __fastcall CanJumpReplacement(Player *player) {
    return true;
}

and replace the Player's vtable entry with a pointer to this function. This works as expected, but why should I use the fastcall convention here? Fastcall is used almost exclusively for this purpose from what I can tell, but I was reading through the calling conventions and cdecl seems to be a much closer match to thiscall than fastcall. Both calling conventions succeeded in replacing the function.

ahoward
  • 53
  • 3

1 Answers1

5

This question is a bit confusing.

Both __fastcall and __thiscall share that they use ecx as the first storage point. So either you implicitly say the class pointer will be in ecx (__thiscall) or you say the function is not a member function but has one argument - which also gets passed in ecx when using __fastcall so the class pointer still ends up in the right register.

Also, both calling conventions use callee cleanup so no problem here, too.

However, this only works for no-argument functions. If your function had an argument, it would end up in edx for __fastcall, but on the stack for __thiscall and therefore not work anymore.

Johann Aydinbas
  • 1,391
  • 7
  • 11
  • Ok, so basically fastcall is preferable for functions with no arguments, otherwise you would have to use cdecl/stdcall depending on context and read ecx manually to get the this pointer. – ahoward Nov 12 '19 at 05:27
  • That is a different question really. For my last project that required C++ member function hooking, I wrote 2 small wrapper functions defined as __declspec(naked) __stdcall which had inline assembly code that turned the this pointer from ecx to a standard stack argument on entry, and on exit took the this pointer from the stack and moved it into ecx again. – Johann Aydinbas Nov 17 '19 at 10:39
  • 1
    Doesn't the x64 Calling convention replace _thiscall anyway with _fastcall ? https://learn.microsoft.com/en-us/cpp/build/x64-calling-convention?view=msvc-170 x64 calling convention – Paul Gigel Feb 24 '23 at 13:12