It's getting more and more confusing: I sometimes expect to add an -en, but instead I find out -es. For example, the colors follow a strong declension, instead of weak, but I don't get why.
Is there a simple rule to immediately know which declension I should use or is it a matter of memorization?
Also, is there a point for adjectives to have three declensions instead of just one, from a grammar point of view?
As has already been mentioned, colours are not fundamentally different from other adjectives.
I suppose what is confusing you is that you can have:
Der Wein ist rot.
Roter Wein ist besonders bekömmlich.
Dieser rote Wein ist besonders gut.
I hope I covered everything now. Wein ist nominative in all three cases, so why is the adjective different each time? One of the underlying principles of German seems to be to not repeat information that has already been given. So, looking at the three examples:
The first case is not declined at all, because the "ist" makes it sufficiently clear which word is described by "rot". Also, it would sound like a comparative form if you were to decline it in that situation. This is of course not a valid point for arguing why it developed that way, it's just a side remark.
Now for the two more interesting cases: In the first case you decline "rot" to show which word it belongs to. Specifying case, number and gender is usually quite sufficient for that, even in very long sentences with many words to choose from.
In the second case, "dieser" already carries all that information, so don't repeat it. "Dieser" and "Wein" form a small so-called bracket, another underlying principle of German, so for everything that goes into the bracket it is also clear where it belongs. (That's the same thing as pulling a construction like "bin gegangen" as far apart as possible in a sentence, this is also called sentence bracket (Satzklammer).) Therefore, you always have to decline the first word that belongs to something, to show where the bracket "opens".
One more thing to pay attention to: If you have more than one adjective, the situation becomes a little more difficult, because then you get for example:
Guter, roter Wein
and not
Guter rote Wein
Here this principle of leaving unnecessary information out doesn't work, because they are both of the same word type. A notable exeption to this rule is the dative (so here we follow the general rule stated above):
Mit starkem schwarzen Kaffee beginnt mein Tag.
Another thing to note: If we have something like "kein" or "mein" or "ein":
Mein roter Wein ist sehr gut.
Ein roter Wein ist was schönes.
Kein roter Wein konnte mich bis jetzt überzeugen.
Here, as you can see, "mein" and "kein" don't have an ending, so the information that would have gone into the ending has to go to the next candidate. If there is none, then the information is omitted:
Mein Wein ist sehr gut.
Ein Wein ist was schönes.
Kein Wein konnte mich überzeugen.
If they have an ending in a different case, then we are back to the general rule, so you don't have to repeat the information (and you shouldn't):
Diesem roten Wein spreche ich zu.
Keinem roten Wein würde ich den Vorzug über weißen geben.
This is what is sometimes called mixed declension, although as you have seen you don't need that construct to memorize when there is weak and when there is strong declension. As you have seen, there is no memorization needed at all, it's a fairly simple rule. One still has to memorize weak and strong declension though.
An interesting read in this context is: Flexion by Thieroff and Vogel (2012)
Not just colors but all attributive adjectives get declined
Attributive: Ich habe den neuen Film gesehen = I saw the new film
Predicate adjectives are not declined
Predicate: Der Film ist neu = The film is new
