Arguably, sex chromosomes don’t really determine sex (thanks for the clarification, @Jayne). Hormones are the major players in determining which sex organs develop, how (or even if) they function, and even differences in the brain.
Some people are born with an XY karyotype but for all intents and purposes appear to be female. This can be caused by genetic defects in androgen receptors leading to androgen insensitivity syndrome. (Androgens are the set of all hormones that influence the development of male features)
Or if someone is born with an XY karyotype and is missing a functional SRY gene, which is necessary for production of testis determining factor, this person would also appear to be female (this is called Swyer syndrome).
On the other hand, someone born with an XX karyotype and a functional SRY gene stuck in there will appear to be completely male.
So I think a lot of people question the concept of both sex and gender since there are cases that challenge pretty much any definition we choose to construct. But I also think that we have chosen to continue using this dichotomy because it has traditionally been very helpful. If I say “women” (e.g., this product was designed for women, or women are better at this task, or women learn better this way), there is no confusion about what I mean. It is only very rarely that this terminology becomes confusing and inaccurate.
You also ask if the only difference between men and women is that women can give birth. I would argue that there are a great number of differences between men and women, both biologically and culturally (and there comes a point where those two categories start to coalesce). But discussing those is another conversation altogether, so I will hold off for now.