Being a resident in Poland or having a Polish wife does not grant you the right to work elsewhere in the EU. In fact, since your wife is a resident in her country of origin, EU law about spouses of EU citizen generally does not apply to you at all (there are some exceptions, e.g. if you lived together elsewhere in the EU in the past).
Unless you are also a Polish citizen, if you wife continues to live in Poland, you would therefore be treated there like any other Pakistani citizen in Germany and would need to apply for a regular work permit (it's not forbidden of course but the requirements are quite onerous and you have to qualify for a visa/permit based on your own situation, not your wife's citizenship). On the other hand, if your wife would find work in Germany, then EU law would kick in and you would have a right to move with her and also to work in the same country with minimal formalities.
Finally if you are in fact a Polish citizen and have an ID to prove it, having a passport is not required. EU citizens can work or travel in the whole EU without passport (at least if they come from a country that issues ID cards, otherwise they need a passport in practice but it's not mandatory per se).