What happen when an NPC unknowingly walks into an invisible PC?

Question

Two party members—one visible, one invisible—flanking an enemy archer. The invisible PC hasn't attacked the archer, and the archer is unaware of the invisible PC.

The archer wants to take a 5-foot step away from the visible PC so he can make a full attack against the visible PC without provoking attacks of opportunity, but the invisible PC occupies the square the archer tries to step into.

The GM worked around this problem this time, but we'd like to know what the rules say about this situation.

What happens when a creature tries to take a 5-ft. step into a square occupied by an invisible, undetected foe?

Answer 1

As per RAW:

Accidentally Ending Movement in an Illegal Space: Sometimes a character ends its movement while moving through a space where it’s not allowed to stop. When that happens, put your miniature in the last legal position you occupied, or the closest legal position, if there’s a legal position that’s closer.

Source: Special Movement Rules

So I think that 5ft movement will not happen. Your DM will probably justify that the archer will feel some kind of force pushing him back, and he will then know there is something in that square (total concealment for attacks still applies). Also, you're not entitled to AoO, because a 5ft step doesn't provoke them.

Even if RAW is kinda clear in this case, your DM may rule otherwise. One possible resolution for this action would be the archer initiating an accidental Overrun combat manuever, with some kind of penality.

Overrun

As a standard action, taken during your move or as part of a charge, you can attempt to overrun your target, moving through its square. You can only overrun an opponent who is no more than one size category larger than you. If you do not have the Improved Overrun feat, or a similar ability, initiating an overrun provokes an attack of opportunity from the target of your maneuver. If your overrun attempt fails, you stop in the space directly in front of the opponent, or the nearest open space in front of the creature if there are other creatures occupying that space.

When you attempt to overrun a target, it can choose to avoid you, allowing you to pass through its square without requiring an attack. If your target does not avoid you, make a combat maneuver check as normal. If your maneuver is successful, you move through the target's space. If your attack exceeds your opponent's CMD by 5 or more, you move through the target's space and the target is knocked prone. If the target has more than two legs, add +2 to the DC of the combat maneuver attack roll for each additional leg it has.

Source: Overrun

The archer would then roll CMB, and receive attacks of opportunity in the process. This may better reflect the situation.

EDIT: Added sources, as per comments request

Answer 2

The RAW answer is already covered, the 5' step would fail and they would know that something is there. That's no different to if you run into someone while running backwards and haven't noticed them.

However there is a lot of scope for "rule of cool" here. For example:

DM: The archer is 5' stepping into you, quick what do you do?

Player: I try and dodge out of the way

DM: Ok, acrobatics to try and avoid him (the player gets to move 5')

or

Player: I drop to the floor and try and trip him over me.

DM: Ok, roll a trip attempt, you get +4 since he's unaware but end up prone yourself.

etc.

In other words moments like this are a good opportunity to break from the literal rules and try for a memorable moment. It might come to nothing or it might be the best thing ever and lead to the next year being full of battle cries like "Don't trip over the elf" being shouted at enemy archers.