5

I'm decompiling some Direct3D code that makes a lot of indirect calls to __stdcall functions.

For example:

call dword ptr [edx+0xC8h]

which is really:

pD3DDevice->SetRenderState();

IDA doesn't correctly guess the stack pointer change of these calls in every case, so I have to go through and Alt+K the correct SP value manually.

But after doing this I start running into a problem where one side of a branch will have the wrong SP value

enter image description here

I can Alt-K the first instruction with the erroneous SP value but this only takes effect on the next instruction.

Edit:

enter image description here

quitegiddy
  • 53
  • 5
  • Can you show how it looks like in text mode (instead of graph)? – Igor Skochinsky Nov 16 '20 at 07:27
  • 1
    @IgorSkochinsky Apologies for the extreme lateness of this reply (long story), but on the off-chance you're still willing to help, I've updated the OP with a text-mode screenshot which hopefully illustrates the problem. – quitegiddy Apr 17 '23 at 11:47
  • the new screenshot doesn't include the code from the first one but some unrelated(?) function. – Igor Skochinsky Apr 24 '23 at 15:18
  • 1
    @IgorSkochinsky Thanks for the response. I must have uploaded the wrong screenshot sorry about that. It's fixed it in the OP now. The right branch isn't out by quite as much as in the original screenshot for some reason. Taken from a fresh db with just the stack pointer corrected on the indirect calls and pD3DDevice named. – quitegiddy Apr 25 '23 at 12:52

1 Answers1

1

So, the problem is that you have two branches with different stack delta: E8 at 5018E4 (fall through from jnz) and F0 at 501c49 (destination of jnz). Normally they should be the same, however IDA failed to reconcile them, probably due to too many indirect calls in the function.

One peculiarity of manual stack delta adjustments is that they apply after the instruction at which you're setting them. In our case, we need to make 501C49 have the delta of E8, however we can't do it on the instruction itself but need to go to the previous one by address and not the control flow, i.e. the jmp at 501C44. Since that address has SPD=E0, specifying delta of -8 should work.

This is why such manipulations are best done in text view and not graph.

Igor Skochinsky
  • 36,553
  • 7
  • 65
  • 115
  • Thanks Igor, that's exactly what I was looking for. I must have been under the misconception that Alt-K would change the next instruction in the control flow rather than the next instruction in memory, which really wouldn't make any sense. Thanks again! – quitegiddy Jun 10 '23 at 08:20