It is not so much that there are no rules, more that there are a few of them that contradict. It really depends where you are and what road you are talking about.
If you are in the middle of nowhere on an open road, 'up the road' is your direction of travel - the village is a few miles up the road.
If you are sat a home discussing the trip, you go 'up north' or 'down south', as shown on a map.
but...
strictly speaking you go up to the capital city, so someone who lives up north, should go up to London even though it is down south.
In a city, you go 'down the road' if the building numbers are getting smaller, which usually means you are going towards the centre, or 'down town'. This still applies even if you are already that way and 'down town' is also 'up the road', or would be if you weren't in a city.
In practical use, the whole thing is very fuzzy, so it is hard to get it so wrong that people notice.