Right now, you both have equal rights. If that’s really what you don’t want, the only way to stop it is to get a court order stating otherwise. If he is the one that ends up filing for custody, it makes it look like he is the one that wants it, instead of you. For example, right now, you can’t legally get a passport for your daughter without his permission (since he’s on the birth certificate and there is nothing saying who has legal custody). Also, there’s a difference between physical custody and legal custody. Physical custody is who she lives with and when (can be split in many ways) while legal custody is who gets to make legal decisions for her. If there is something that requires you both to agree (like a passport) you wouldn’t be able to do it right now because you both have the right to make legal decisions for her. Some school systems can be really picky about this too when it comes to registering your child for school.
You having possession of her right now gives you a bit more rights at the moment (because you are the custodial parent and he is not right now), but things could get really ugly. All he has to do is call the police and say you are refusing to let him see his daughter and things could get ugly fast (cause then he’d have it documented as you refusing to let him see his daughter and that would look bad).
I’m not trying to scare you, but I’ve seen people do pretty nasty things when it comes to custody. Usually its aimed more at hurting the other parent than worrying about what’s best for the child. I’ve been through it with my ex-husband and it was worth it to me. I’ve had sole physical and legal custody for quite a while now. Sure he could come back and try to get things changed, but I have a lot more rights than he does now, so it’ll be harder for him.