Google offers the literal translation "Toutes mes excuses pour le retard dans la réponse" for "Apologies for my delay in responding", but does this sound correct, idiomatically?
"Désolé de ne pas avoir pu répondre plus tôt" sounds a little dishonest to me; I was able to respond sooner; I just forgot!
The closest translation of "sorry for the delayed reply" would be:
Désolé de la réponse tardive.
but that might sound a little formal.
Using pour for de is possibly an anglicism but common nowadays:
Désolé pour la réponse tardive.
Normally pour introduces the "victim" (je suis désolé pour vous) while de introduces the cause. Here is another suggestion where de is standard and that doesn't suggest you were prevented from doing it by something else:
Désolé d'avoir tardé à vous répondre.
"Toutes mes excuses pour le retard dans la réponse" — quite correct
"Désolé de ne pas avoir pu répondre plus tôt" — also correct (Not dishonest as I see it; whether caused by bad memory or something else an impairment is an impairment, but that's off topic.)
