I am running a sci-fi/fantasy game using Pathfinder. The setting is firearms and future weapons focused. It's also being played on a virtual battle-map with tokens and will be run very tactically.
How do I tactically use the cover rules in combat to create a realistic feel to a firefight both in and out of game?
Well, Pathfinder uses largely the same ruleset as d20 Modern did, which while not the best modern gun system ever is far from the worst. Here's how to get the most out of it for a firefight.
To make anything like a realistic gunfight, people have to be concerned about being shot. This means that people should have few hit points, and that guns should do a lot of damage. Keep levels low and advancement slow. Consider doing something like the E6 variant rules that cap normal level advancement. Do NOT use the piddly official Pathfinder gun rules. Use ones like the d20 Modern rules Brian linked - 2d6 should be minimum ever for a gun. If you're at a reasonably low level, 2d12 from a .357 Magnum is terrifying. A crit there can one-shot most characters of levels 6 and under. You can also try the d20 variant vitality/wound system. Oh,and remove AoOs for using a gun in melee.
Pathfinder has the same cover and concealment rules that all d20 variants have. Learn them and use them, especially partial/improved cover. Same with the terrain, vision and light, conditions, etc. rules. You need to become really familiar with most of those kinds of sections and be applying them all the time, not on an exception basis. Heck, even if you're shooting someone in the same motel room, do you see ALL of them? Or is enough of them behind a bed, tv, etc. that at least a small cover bonus is merited? You should be using Perception an awful lot. One of the distinctive elements of firearm combat is how hard it is to see all your foes and everything that's going on. Exert these rules to the utmost - and when in doubt, give defensive bonuses. If cover and concealment bonuses are so piddly compared to to hit bonuses that no one cares, they won't take cover - make sure defensive stuff helps. It's OK for most shots to miss. Strongly consider not sharing much information with the PCs, like whether they hit a foe or not (let them have a Perception check, but don't just say "ah yes you hit and do 10 points of damage" or "he is definitely down and dead") - this forces them to be careful and double-check what's going on.
You don't
As much as I'd like to recommend house-rules to provide for a "realistic gunfight" with the built-in cover-rules... I can't.
The problem with D&D in general, in light of this question, is that it promotes "heroic" games, where creatures routinely take (unless they're first level wizards) dozens of sword hits without being too much worse the wear. Your objectives are: sci-fi, realistic, very tactically, and interesting cover.
Let me address the points individually:
D&D can certainly do the more space-opera/fantasy style "sci-fi" settings, but you'll have to work to adapt the rules. I personally treasure my "Fading Suns" rulebook that provides an excellent treatment of a sci-fi 'verse with 3.0 rules. ... But I don't play it. It's probably worth shopping for a system that is designed around the setting you want to evoke.
Default 3.5 (not looking at modern here, cause I've got no experience with it) can't do gunfights worth an expletive. Default guns, those presented in the D&D DMG (and doing a quick google around, found here) ... don't do realistic damage. A .357 magnum, according to the wiki, does 2d12 damage. If you're getting shot for 2d12 damage, it certainly hurts, but there are better things to do than to seek cover.
The problem with the gun rules in 3.5 is... you'll be doing so much work houseruling them and finding good patches that you might as well find a better base system. Upping their lethalty is great, but then characters... don't feel like being heroes. Cause it hurts.
Cover in 3.5 is handled with a number of conditions, with very little to differentiate between things granting the same conditions. This is absolutely appropriate for the things being modeled in 3.5, especially considering that you generally don't want people using cover in a heroic game. It's just not fun.
The best cover-based game that I know of is Aces & Eights. Where you have a shot clock, you apply the cover, physically, to the clock, and then you see where you hit. It's also the most realistic. It would take a bit of effort to adapt the shot-clock to an online setting though.
To sum up, D&D is designed for heroic melee combat. Tactical cover based shooting is a stretch that isn't supported particularly well. While it's possible to do so, and I support @MadMAxJr's answer in that regard (especially regarding the massive damage), keeping track of the house rules will be difficult and frustrating.
I encourage you to post a new question (if you think my answer has merit) with details of your setting, soliciting systems. (I'm having a hard time of thinking of any to recommend that are non-generic off the top of my head.) Barring that, explore what GURPS has to offer, as that system can tend far more to the realistic than D&D.
3.5/PF abstracts combat details heavily, so you will have to fill in the gaps with some of your own rules. Gun/future weapon based combat may revolve heavily around the kinds of terrain available to the players. They are likely high damage weapons meant to harm and/or kill in one or two shots.
Here are some suggestions for fleshing out the rules:
Cover is going to be key. Most players aren't going to want to be shot. You may want to implement additional forms of cover to represent varying effectiveness of types of cover. A cluster of barrels and crates may only provide a +2 cover bonus or a pile of industrial steel beams may provide a +6 bonus.
Concealment will be another major factor. Smoke grenades, fog, and other conditions may need to apply. Characters relying on electronics for vision may have to treat targets as being concealed if they receive electronic interference.
If you're wanting to simulate some degree of realism, then heroes should not be expected to be able to sustain more than a few wounds before falling. Consider borrowing a rule for d20Modern that makes your CON score your massive damage threshold. If you sustain damage equal to exceeding your CON, you must make a DC 15 Fortitude save afterward or drop to 0 hp. There are other ways you can try to scale this, here are some suggestions from d20SRD.
People tend to forget that in the definition of Hit Points is a REPRESENTATION of the ability to "soak up" damage - in the form of: Dodging out of the way, Luck, "crap, that would have killed me" stress and the sheer presence of mind to keep going before you are mentally/physically exhausted (or unlucky) enough to die or go unconscious.
Lets face it if a giant hits you with a sword, realistically your going to be missing a limb and/or as good as dead, regardless of how much armor your wearing
The wounds/ Vitality set up is a good way to represent "deadlier" combat
You could just up the threat range on guns in general
Steal some crit tables from Rolemaster - lol
Like mxyzplk said, D20 Modern is somewhat compatible with Pathfinder. So you could steal the thing that (in my opinion) serve to make gunfights interesting in that system. (The bad kind of interesting for the person getting shot.)
Massive Damage A character's massive damage threshold is equal to the character's current Constitution score.
When a character takes massive damage that doesn't reduce his or her hit points to 0 or lower, the character must make a Fortitude save (DC 15). If the character fails the save, the character's hit point total is immediately reduced to -1. If the save succeeds, the character suffers no ill effect beyond the loss of hit points.
BAM. There's your reason for seeking out cover. Using the E6 rules only helps, of course.
Of course, with that rule change in place the 2d12 suggested by mxyzplk is a bit much. For inspiration from the source, you can take the weapon table from the D20 Modern Online SRD: http://www.scratchfactory.com/ModernSRD/EquipmentWeapons.php
My only other question then would be that, with the cover rules as written, there doesn't seem to be a way for a character to fire from around a corner. Ideas?
– Josh Nov 26 '11 at 03:45