There's a part of empathy which is actually feeling what other people feel (as you understand it). You can feel empathy without any reasoned thought - you see someone experiencing something and you may just feel some version of what it's like to experience that yourself. This is something skeptics shouldn't have much trouble experiencing, as skeptics are not robots: they do experience emotion, like the rest of us (presumably... if any of us actually exist).
The other part of empathy is reasoning to try to understand enough about the other person to put yourself in their shoes (in order to either rationally understand where they're coming from, or to evoke the feeling mentioned above). This is what my answer will mostly focus on.
You seem to be human, other people seem to be human, so the most reasonable starting position is that other people think like you do. In that sense, you can try to imagine how you'd feel if you were in their situation.
To you apply skepticism to other people thinking like you do (which they often don't), you can see how other people act, and listen to what they say, and you can look at neuroscience that studies the structures and mechanisms of brains, and psychology, which studies how people think as more of a black box.
And then you can question your goal, and see how that maps to some action.
For me, there are certain mental states that I can't understand, but that doesn't mean they don't exist, it just means I can't understand them. So either I can try to squash them into some box I do understand, and treat them appropriately, which may result in a simpler and more comforting worldview for me, but may do a lot of harm to others. Or I can leave it at "I don't know", and listen to their experiences and feelings and what they want, and try to determine how to treat them in a way that would be best for their well-being, and for the well-being of society as a whole.
There may have been some skeptics who believe that we can't, with any degree of certainty, know anything about anything. I can't really speak for them.
Most skeptics (modern skeptics, at least / especially) may accept that we can't have complete and total certainty about anything (beyond "[I think therefore] I am"), but we can still have reasonable certainty about quite a few things.
Skepticism means questioning your beliefs. You still have beliefs, and you should still try to figure out what is most reasonable to believe, even if you're not certain (while also being willing to admit when you're not certain enough to make a judgement... but also not using this as an excuse to avoid a truth that would inconvenience you).
* I assume you aren't talking about solipsism (i.e. the question of whether other minds, and perceived reality as a whole, even exists at all).
