You are not hearing wrong. Spanish /g/ sounds different according to its position in the word; in technical terms, it has several phonetical realizations or allophones. The basic sound is what in English is called a "hard G", that it, the voiced velar stop [g]. This is the usual pronunciation of /g/ at the beginning of words and after most consonants.
However, Spanish /g/ between vowels is pronounced [ɣ], a voiced velar fricative. This is what you hear as close to [h] or [x] in the example with the word agua (and others). Some people and/or dialects further weaken this fricative to an approximant, especially in fast, careless speech, and some may even elide it altogether. This weakening is always clear between vowels, but sometimes takes place also after liquids (/l/, /r/).
You will be understood if you pronounce a hard G between vowels, but to native ears it will sound a bit too strong, almost like /k/.
The same thing that happens to /g/ happens also to /b/ and /d/ between vowels (see the relevant section on Spanish phonology in Wikipedia).
The other thing you noticed is that /g/ before a dipthong that begins with /u/ (phonetically [gw]) tends to get reduced as well. This is not universal. As Paco says in another answer, the two velar sounds seem to be interacting and the /g/ is dropped while the [w] is strengthened a bit. It is also common to find the opposite phenomenon: [w] by itself turning into [gw] at the beginning of words (prothesis): people will say [gweβo] for huevo, for example, and even write güebeo for hueveo.
This back-and-forth between [g] and [w] is how the Quechua word wanaku got borrowed into Spanish as guanaco. It is also (going farther back) how the modern Spanish word guerra "war" was derived from Proto-Germanic *werra: first a prothetic [g] was added, and then [gw] was simplified to [g] in some Romance languages (like Spanish) but not in others (like Italian).
