There is no way that this works
You can only use a resurrection spell successfully on somebody whose soul is free and willing to be resurrected. This requires the soul to still exist in the first place. However, both of these factors are false for a soul that became a demon:
- The very soul sought out to return is no more.
- A soul that does not exist is clearly neither free nor willing to return.
But what if they still were souls?
Even if you (erroneously) assume that the demon still could be the soul sought, those two factors fail:
- Most demons are bound to other, more powerful demons, and thus are not free in almost all cases.
- The few free demons can just decide that they don't want to be resurrected: They would know the alignment and identity of the caster and realize their loss of power and say no (see below).
- Also of note is, that the typical age of a free demon is most likely older than dirt (or several centuries) and thus far beyond the 10-years-per-caster level of the most powerful resurrection spell True Resurrection.
Raise Dead:
In addition, the subject’s soul must be free and willing to return. If the subject’s soul is not willing to return, the spell does not work; therefore, a subject that wants to return receives no saving throw.
Even allowing the demon to count as the soul (which it shouldn't) the
3.5th Player Hand Book precludes it. The rule says that the soul can just deny it for any or no reason. They also very much know who is trying to resurrect it, making it nigh impossible to force anyone back from the afterlife:
Bringing Back the Dead: [...]
Revivification against One's Will: A soul cannot be returned to life if
it does not wish to be. A soul knows the name, alignment, and
patron deity (if any) of the character attempting to revive it and may
refuse to return on that basis. For example, if Alhandra the paladin is
slain and a high priest of Nerull (god of death) grabs her body,
Alhandra probably does not wish to be raised from the dead by him.
Any attempts he makes to revive her automatically fail. If the evil
cleric wants to revive Alhandra to interrogate her, he needs to find
some way to trick her soul, such as duping a good cleric into raising
her and then capturing her once she's alive again.1
By the way, that very same text was moved in 5e to the Dungeon Masters Guide: Magic in Your World > Bringing Back the Dead with just a slightly different example:
A soul can’t be returned to life if it doesn’t wish to be. A soul knows the name, alignment, and patron deity (if any) of the character attempting to revive it and might refuse to return on that basis. [...] If the evil cleric wants to revive Sturm to interrogate him, she needs to find some way to trick his soul, such as duping a good cleric into raising him and then capturing him once he is alive again.
Turning Souls into Demons
But back to the start of this side-track: being turned into a demon is just one of the numerous exceptions that turn souls into other beings or outright destroy it, at which point they are no longer souls but other beings and can't be revived because they are no longer souls, as I pointed out in the first paragraph. The Nine Hells turn their captured souls into Lemures and lesser Devils, Hades turns them into Larvae and so on.
In fact, to resurrect someone who died from a Narzugon's Hellfire Lance under 5e rules and who turned into a Larva needs to be killed first to be resurrected at all.
Conclusion
As a result, you can not use any resurrection spell on a demon successfully as they are not a valid target or unwilling, and the spell automatically fails - without telling the caster why the spell failed.
Footnotes
- Dungeons & Dragons Player's Handbook, Core Rulebook I v.3.5, 2003, p.171