Each unit is compiled according to its API version. However, when a transaction starts, the API version will be set according to whatever initialized the transaction. For example, a Visualforce page's API version determines the running version for the entire transaction when viewing a Visualforce page. If you read a debug log, the top line shows the API version for the entire transaction.
In most cases, everything works just fine, but be aware that some edge cases exist. For example, if you use a describe in a higher-versioned class, generate a SOQL query string, then return that to a lower-versioned API class, it can introduce fields that don't exist. The example I remember best was when I had a 28.0 class that generated a dynamic SOQL for a 27.0 class, and I would get a "LastViewedDate does not exist" error in the query. In other words, each unit behaves as if it were from that API version (because it is).
For this reason, I advise that you try to keep all of your versions the same, and schedule some time every few releases to test out updating the API versions to keep everything current. It's okay to not jump to the bleeding edge API version (especially during the next release preview cycle), just try to keep within 2-3 versions of current.