This is an interesting discussion/debate that is currently being solved with live experiments in the form of martial arts competitions. Though the results have proven one being "better" than the other, there are a few exceptions. For the sake of simplicity I will refer to the palm facing down has a "horizontal fist", and palm facing sideways as a "vertical fist."
The twisting movement that your elbow makes is the major factor in the difference between these two techniques. I'm going to quote an article from expertboxing.com, because to paraphrase wouldn't do it justice:
The horizontal fist position is more powerful not so much because of
the rotation of the fist but rather the rotation of the elbow. With a
vertical-fisted punch, the elbow stays down and doesn’t support the
wrist until the very end where the arm is extended straight. Whereas
with the a horizontal-fist, the elbow rotates and lifts up immediately
supporting the wrist much sooner.
Having the elbow come up behind the fist is the key difference here.
The wrist has so much more support and delivers more power when the
elbow is behind the fist to give your forearm a straighter impact
angle. It’s important to know that punches can land with the arm bent
at varying angles. Rarely does the punch land when the arm is
completely straight. It’s more common to land punches with the arm
bent (at the elbow) to some degree, even if only minor.
Because punches are so often landed with a bent elbow, it’s generally
more powerful to land with a horizontal fist. When punches are landed
at very close range (with a very bent elbow), you’ll notice the power
output greatly favors a horizontal fist over a vertical fist. Think of
the optimum benchpress position, where you’re more powerful when your
elbows point out sideways (using the chest) supporting the wrists
better, rather than pointing downwards (using the triceps) towards
your feet.
One thing they don't mention is the hook. The hook is the one time that a vertical fist is preferred. The reason for that is the lateral movement. The hook has a built in "digging" motion to it. If one were to throw a hook with a horizontal fist you would too often land with your outer knuckles, losing most of your power. Throwing the hook with a vertical fist allows you to land with the middle of the fist for maximum power. Many boxing coaches will teach an almost slapping motion to beginners for hook technique.
Uppercuts fall under the same category, except a horizontal fist solves the problem. If you were to throw an uppercut with a vertical fist you would land with your outside knuckles, causing less damage, and taking more damage to your fist.
To summarize: for straight punches (and overhands), you want a horizontal fist. For hooks, vertical fist. Uppercuts are back to horizontal (but upside down of course). But wait! Power and speed aren't everything! There have been some excellent fighters that managed to mix in vertical fist straight strikes with success. The ability to throw in a deceptive vertical fisted strike right down the pipe is something Manny Pacquiao, Lyoto Machida, and many other fighters have incorporated to their game. One thing is true for all techniques in fighting: if it works, it's going to be used.
Source for the quote: https://www.expertboxing.com/boxing-techniques/punch-techniques/horizontal-punches-vs-vertical-punches
https://martialarts.stackexchange.com/questions/1649/what-are-the-advantages-of-using-a-vertical-fist-for-punching
– Andrew Jay Dec 20 '17 at 16:40