There are fat-free, hormone-free, antibiotic-free, free range/grass-fed and organic dairy products available, to eliminate the bulk of those worries. That leaves you with the lactose and calcium issues. If someone in your family has had lactose intolerance or kidney stones, you do need to be more concerned about those issues than if no one has those problems in your biological family history.
I drank tons of milk as a kid, and weaned myself down to lower fat options as I got older. Once hormone-free and antibiotic-free options became available, I started buying those. In fact, I didn’t even know such things were in the dairy supply until I saw the products that specified that they didn’t have them. Two widely available brands are Stoneyfield, which is organic; and Farmland which is antibiotic-free and hormone-free, and they have a process that adds other milk solids into their fat-free ‘Skim-Plus’ product line, which make it taste creamier than other skim products.
Oddly, when I was pregnant, I began craving orange juice and I drank so much of it then and very little milk, except in cereal. I drink much less of both of those now, and will brew tea (hot or iced) most of the time to add some flavor, because I don’t like the taste of water. I drink mostly herbal teas, since caffeine is a diuretic, and only rarely will I add a spoonful of local honey if I’m craving some sweetness. I think our tastes do change as we get older, and our body often tells us what we need. So if you don’t have a family history of issues with dairy or calcium, don’t quit milk altogether – just change over to the healthier versions and try to drink more water and/or other caffeine-free, unsweetened beverages.