I have a personal preference for hexes over squares - the latter present some problems in terms of diagonal movement, and aliasing for certain area effects, that are just handled better with hexes.
So for systems like D&D 3.5 and 4e, and also less popular ones like FASA Star Trek, which use square grids rather than hexes, what are the results of substituting in hexes?
4th Edition, from experience
When inflicted on a 4e group as part of a curse, it gave everyone a headache and made combats incredibly long.
The hex based map presents incredible difficulties in calculating zones relative to the ease of calculating zones in a square map. Either zones in a hex grid are the same area as a square grid or they are the same approximation of a circle. Both have their difficulties in calculation. I found that with a group experienced in square maps, everyone had to rebuild their standard tactics from the ground up to deal with the different terrain.
On the advantages side, a hexagon approximates a circle far better than a square, and with wire templates, the difficulty in calculating zones will be reduced. It also offers odd cognitive dissonance (as observed) with the squiggly-lines problem: moving "across the grain" of the map will have a character rapidly oscillate between two different rows, annoying some players.
In the question of balance, bursts will generally effect fewer people (close burst 1 has max 6 targets instead of 8) and blasts may or may not effect more people, depending on how you calculate the area of the blast. A hex-diamond-shaped blast will offer a player greater "reach" than the normal game's blast, at the expense of "width" simply by the geometry of the hexes. Changing the definitions of blasts and bursts to be more "realistic" may help the problem, but will introduce non-trivial balance problems in both directions.
My general recommendation is to never inflict a hex grid on a group used to square grids, but it may not be a bad basis for a campaign if everyone wants a hex grid. It's probably more appropriate to use it with a simulationist system though.
3.5 from little experience
While running and playing in some play by post versions of 3.5, we tried using hexes. While 3.5 maps more ably to hexes because it's based on naturalistic geometry, the heuristic for calculating distance is slightly more difficult than "every other diagonal counts as 10 feet. Still, hexes are not a bad choice for 3.5 especially if using house-ruled ideas about facing.
Other tactical games
Fundamentally speaking, if a game is designed for real geometry it will play well with hexes as both squares and hexes can be mapped to circles and feet without too much trouble. A game with high amounts of abstract tactical design "baked in" (4e) will do less well, because much of that tactical design is based around assumptions of a square grid.
This might be a digression but there's always the option of using neither of them.
A tape ruler and some wargaming templates (warhammer or warmahordes) and you're good to go. Replace squares/hexes with inches and you have a pin point accurate system. Takes a little getting used to at first but it works smoothly once you get going.
For the GM, maps suddenly becomes a whole lot easier to make and you are much more free to do what you want without being constricted by grids, especially if you want to use terrain pieces as well.
The d20 SRD has a simple and useful section on using hexes instead of squares. In terms of ramifications it has this to say:
Using a hex-based grid changes relatively little about the game, but poses a mapping dilemma for the GM. Most buildings and dungeons are based on 90-degree and 45-degree corners, so superimposing a hex-based grid on a structure leaves the GM with many partial hexagons, not all of which are big enough for a Medium creature. Use this variant only if you’re comfortable adjucating these partial spaces on the fly.
I habitually use squares in constricted environments like alleys and most areas of dungeons, and hexes for open-movement or non-linear areas like wilderness, very-decayed ruins, and caverns. It works great to mitigate the mapping problem the SRD mentions. Given the clear area-of-effect templates in the SRD, there's also zero work to make existing spells and effects work properly on the hex map.
Hex vs Square boils down to several things.
Like Brian said, if you are using 4e with hexes you need to decide whether you want to continue using the square area of effects or go with a circular hex based area of effects.
I generally prefer hexes and use them for GURPS, Harnmaster, Traveller, etc. But for D&D style games, whether 4e, 3e or classic, I go with square because 90% of the material I have it drawn in square.
In a setting like city streets that are aligned north-south and east-west, traveling on a hexagonal grid in one direction will have similar experience to traveling along the diagonal in a square grid.
You might want to check out the topological implications. Here is an example: Go Board Game on Hexagonal and Triangular Grids.
