I'd say both. Technically speaking, sifu doesn't even mean teacher (that would be laoshi), but skillful master. If the first one laid the groundworks for you, the fundaments on which the second one builds, both have formed you, so both are your teachers.
It's no different from training, say, wing chun in one club and tai chi in another. You won't say the wing chun is your teacher, but the tai chi one isn't. It's unavoidable that wing chun and tai chi won't mix together in how you use it, if you train both, so even if you stop either one, the teachings from that one will still be in you.
The same goes for doing wing chun from two teachers, as long as they formed you up to the point you're carrying it in your other training.