Grandparent threatening to take me to court

This is an extremely long story. But, I will try to run through it as quick as possible. I have a daughter who I had out of wedlock when I was 16 years old. Never married the father. She was born in South Carolina. Biological father went to juvnile jail until daughter was 6 months old. Biological father paid $35.00 child support through clerk of court 4 times. No support order ever issued. I moved with my parents to West Georgia about 3 1/2 hours away from where her Biological father lived. Paternal Grandmother sent birthday cards and christmas cards. We took daughter to see her when we went back to town to visit family. Sometimes twice a year. But, mostly once a year. She never once made an effort to visit. Bought my daughter diapers one time. Biological father nowhere to be found and paternal grandmother refused info so that I could not enforce child support. We moved when daughter was 4 to South Georgia (about 6 hours from Paternal grandmother)and continued to bring daughter about once a year. Biologocal father gave daughter false promises. I married my husband and he was in the USAF. We got stationed in NC and we are still here. (5 hours from Paternal Grandmother. We have continued to bring daughter to see paternal grandmother every year at Thanksgiving. 2 years ago, I dropped daughter off at paternal grandmothers house for a few hour visit and I called to check on her and she wanted me to come over and stay with her for the rest of the visit. Told her paternal grandmother it was because she was bored. But, in reality it was because she was scared. Last Thanksgiving, we took her to see paternal grandmother again. Daughter wanted me to stay with her. So I did. Grandmother complained she didn’t get to see my daughter that much and I told her I do what I can. But, the road goes both ways. Find out later that my daughters biological cousins were supposedly molested by their stepfather. Not sure if it was true. No charges were ever made. Paternal grandmother finally wanted to come see daughter after 8 years of never attempting to visit. I arranged for grandmother to stay on base for an extremely cheap rate and when she said she was coming she said she was bringing her daughter (my daughters biological aunt)4 of her other grandchildren (the ones supposedly molested) And told me that the Aunts new boyfriend would also be coming. I told them that was fine. But, Hope would not be spending the night at the hotel room if he was to come. Explained that I didn’t know him and would never allow her to spend the night with strange men.They cussed me out…told me I was being ridiculous. But, ended up coming and I didn’t allow her to spend the night with the strange man. They still complained.They didn’t have to bring the man. But chose to anyway even after I didnt approve. Here recently they have wanted to come drive here and pick Hope up and take her back to South Carolina for a few days to a week…promise to bring her back. I say no because I don’t want to be 5 hours away if she gets scared and I would have to be in town if they were going to have her. I get cussed out for that. I still voice my concern about the molestation and the new boyfriend I know nothing about. Aunt admits she had been having an affair on the stepfather accused of molestation and found out she moved her kids in with this new man right after the molestation supposedly occured. Doesn’t seem to be very interested in her childrens mental health moving them into another strange mans house after they supposedly been through so much. A few weeks ago, I told paternal grandmother that we would not be coming to south carolina due to my husband having to work. She then asked my daughter if she could come get my daughter and take her back to South Carolina for Thanksgiving. Daughter didn’t know what to say so said she would like that. put Paternal Grandmother on the phone with me and Paternal Grandmother asked me. I told her daughter already had plans with my mother and even still I would not allow her to go out of state without me. Paternal grandmother got the Aunt to email me and tell me I was keeping Hope from them. I haven’t written them back. Paternal grandmother called and was rude to me saying “Let me speak to ****” (daughter) I just hung up. Paternal Grandmother and Aunt call on 3-way to my mothers house and talk to my sister. My sister told them my concerns and told them to just wait it out being nice to them. They keep emailing me…I don’t respond. Day after Thanksging Paternal Grandmother calls my mothers house again and talks to my father this time and threatens to get a lawyer and take me to court. I have not responded to these threats. Does this woman have rights to my child? Is this something I should be concerned about? I would like to cut off total contact now due to the molestation and the threats. Do I have to let her visit and would she be able to take my poor daughter out of state against my wishes? Please I need any advice you can give me. It would be greatly appreciated!! Also Father is still not in Childs life. Daughter has my maiden name and he never signed birth certificate. Thanks again!