If you have 2 children with someone out of wedlock and have been a happy family for 22 years and then one person becomes saved and the other is not, how do you please God without sinning?
Do you get married (before you share yourself with your partner again) and be unequally yoked?
Or end the relationship and break up the family?
Do not be unequally yoked with unbelievers. For what partnership can righteousness have with wickedness? Or what fellowship does light have with darkness?
2 Corinthians 6:14
Do not be unequally yoked with unbelievers. For a what partnership has righteousness with lawlessness? Or what fellowship has light with darkness?
The Bible contains several instances of approved marriages between a believer and a non-believer. (I use the term "approved" in the sense that God did not condemn it and the marriage had good results).
The most famous such marriage is found in the Book of Esther, where the Jewish heroine marries the pagan king Ahasuerus and thereby saves her people.
In the Book of Exodus, Moses marries the Midianite Zipporah (a tribe specifically forbidden for intermarriage with Israelites afterward).
In terms of the specific case mentioned in the OP, the couple are already married under common law. That being the case, the advice of 1 Cor. 7 applies:
If any brother has a wife who is an unbeliever, and she is willing to go on living with him, he should not divorce her; 13 and if any woman has a husband who is an unbeliever, and he is willing to go on living with her, she should not divorce her husband. 14 For the unbelieving husband is made holy through his wife, and the unbelieving wife is made holy through the brother. Otherwise your children would be unclean, whereas in fact they are holy.
Conclusion: The above instruction applies directly to the OP's case, where the couple is already married (although the marriage is not blessed by a minister) prior to one member of the couple becoming a Christian. Therefore the advice of 1 Cor. 7 applies: "If any brother has a wife who is an unbeliever, and she is willing to go on living with him, he should not divorce her."
There are two matters here that should be kept quite separate.
Married or not?
The idea of a legally registered marriage is relatively modern. In ancient times, most common marriages (especially among the poor where most could not afford the "party celebration") simply began when people agreed to begin living together. Thus, in the ancient Biblical sense, the OP is married to the partner.
This would also be effectively true in many legal jurisdictions and would be termed a defacto relationship essentially equivalent to marriage. Thus, the decision to marry the partner after a defacto relationship will not change much legally.
To divorce or not?
Paul specifically addresses the OP's question in 1 Cor 7:12-14
To the rest I say this (I, not the Lord): If a brother has an unbelieving wife and she is willing to live with him, he must not divorce her. And if a woman has an unbelieving husband and he is willing to live with her, she must not divorce him. For the unbelieving husband is sanctified through his believing wife, and the unbelieving wife is sanctified through her believing husband. Otherwise your children would be unclean, but now they are holy.
Note that the word "divorce" in the above translation is simply ἀφίημι (aphiemi) meaning to "send away" which would apply whether the couple is formally married or not.
I think Paul's advice is best.