5

Assume I use a software to encrypt data. How would I go about to find out with IDA or other RCE tools as to whether there is more than one key used during the encryption?

I am talking asymmetric encryption here, and it is possible that the software in question hides one or more master keys (or makes use of some already elsewhere on the system) and I want to find that out. How can I approach that task?

NB: you may assume I have determined the various algorithms in use.

0xC0000022L
  • 10,908
  • 9
  • 41
  • 79

2 Answers2

5

Statically, use IDA to find all cross-references to the "various algorithms in use" to determine how they find the keys they use.

Dynamically, use a debugger to set breakpoints on the "various algorithms in use", thereby allowing you to examine the callstacks and determine how they find the keys they use.

Jason Geffner
  • 20,681
  • 1
  • 36
  • 75
  • Well, dynamically is probably not a good idea, because I could overlook some mechanism whereby the program behaves different under debugger from when run normally (as some malware does). – 0xC0000022L Aug 29 '13 at 01:43
  • 1
    Some code might compute the addresses of the algorithm functions dynamically, so IDA might miss their cross-references. Unless you have the time to do a full static-analysis of the entire program, it's usually best to do your analysis both statically and dynamically to try to cover your bases. – Jason Geffner Aug 29 '13 at 16:50
5

Assuming the you are speaking about strong encryption, most of the algorithms are supposed to be indistinguishable even if you provided either the key or the clear text. So, it should not be possible to know that a given key is used just by looking at the result.

One example of this is the field of Kleptography where:

[...] the outputs of the infected cryptosystem are computationally indistinguishable from the outputs of the corresponding uninfected cryptosystem. Hence, in black-box implementations (e.g., smartcards) the attack is likely to go entirely unnoticed.

As you may have noticed, the Kleptography is safe only on black-box implementations, so there indeed room for detection in white-box attacks.

But, I have not enough experience in this topic to give you general advices about it. Except the fact that you should start to look at real World implementation of cryptographic backdoors before trying to detect it. You may want to explore some of these links:

And so on...

perror
  • 19,083
  • 29
  • 87
  • 150
  • I realize the issue about ciphertexts, but I'm talking about finding out in the program whether it encrypts it against more than the key I am giving it ;) – 0xC0000022L Aug 29 '13 at 01:44
  • If I understand well (tell me if I am wrong), you want to check it there is no backdooring process within the program in order to create a master key to access the clear text in addition to your own private key. – perror Aug 29 '13 at 11:36
  • that's correct. – 0xC0000022L Aug 29 '13 at 11:54
  • 1
    But, I am sure that you might get better answer in the Crypto Stack-Exchange website ! Just try to use the correct vocabulary and they will try to answer. :) – perror Aug 29 '13 at 12:00