In the absence of a court order, neither parent has superior rights to the children. Thus, if you have the children at some point, or have access to the children, without trespassing, then you have every right to have your children spend time with you.
Be aware, however, that some judges have issues with cohabitation and the matter could affect a custody decision if you ultimately end up in court due to an inablility to agree on a custody arrangement.
Lee S. Rosen
Board Certified Family Law Specialist
The Rosen Law Firm
4101 Lake Boone Trail, Suite 200
Raleigh, North Carolina 27607
The response posted above is based upon the limited factual information made available and is not intended as a full and complete response to the question. The only reliable manner to obtain complete and adequate legal advice is to consult with an attorney, fully explain your situation, and allow the attorney sufficient opportunity to research the applicable law and facts required to render an accurate opinion. The basic information provided above is intended as a public service but a full discussion with an attorney should be undertaken before taking any action.