Actually, that's not entirely accurate. In 3.5 at least and to the best of my knowledge your To hit against a given creature doesn't change based on their size. Their Armor Class, does however change based on a given Size modifier (+2 Tiny, +1 Small, 0 Medium, -1 for Large, -2 Huge, and so on). Note that is only based on the size of the defending creature, not their differences in size. You also gain an attack bonus or penalty based on your size, not size difference. To quote from the section on Small Characters:
A Small character gets a +1 size bonus
to Armor Class, a +1 size bonus on
attack rolls, and a +4 size bonus on
Hide checks.
So, when you factor in the bonus you're getting to hit based on your size if you're small, say, and the AC penalty a Large creature takes for being Large, you're effectively at a +2 to try and hit them.
If you take this and apply it to 4e, you basically throw out the whole idea of getting a bonus/penalty to hit based on your size, and the size of the monster is factored in when determining its AC Defense. It's not too much of a leap to go from -1 Size penalty to AC factored into a Monster's stat block in 3.5e, whereas in 4e they just did the factoring of the Monster's AC and didn't tell you about it.
Sources:
Combat Section d20srd.com
Small Creatures Section d20srd.com