My ex wants to take me to court to get access.?

Let him go to court. He sounds like a smart guy, this means it will make it official, you will both have the opportunity to stipulate what you want. I don't see what the big deal is. He is a parent just as much as you are. What if something happens? Well he will be with a loving parent...