Here's the problem. The translation you're using is not a very good translation. This is the verse you're referring to (in a translation that actually supports your argument):
In Genesis 4:12 (NIV), God tells Cain
When you work the ground, it will no longer yield its crops for you. You will be a restless wanderer on the earth.

The phrase here, "then in moaning and trembling you will be upon the earth". That is what is often translated as "restless wanderer" or "fugitive and vagabond"
Why? Well, let's dig in:
stenochōria Strongs 2532 dire calamity, extreme affliction
kaiv Strongs 2532 and etc
tremō Strongs 5141 tremble, to fear, be afraid
eimi Strongs 1510 to be, to exist, to happen, to be present
epi Strongs 1909 upon, on, at etc
ho Strong 3588 this, that, these
gē Strongs 1093 arable land, ground, the earth as a whole, a country, region, etc
So, the idea is with "[dire calamity/extreme affliction] and [trembling/fear] you [will exist/be present] upon [the earth/the land/the region]".
That doesn't really say anywhere that he will wander or that he will never stop wandering. However, if you combine these two concepts: fear and affliction, it's understandable that the translators could interpret that as someone who would wander.
If I was full of fear and troubled with distress, I would probably wander as well.
However, having said this, I have no idea why they chose that translation. It seems to be a poor translation of this phrase.
Given a decent understanding of the curse, it should be easier to see that building a city and being fearful and in distress do not conflict.