It sounds like there is possibly an issue with him. The brothers might attack him to establish dominance…set the pecking order. But typically the females don’t do that. Have you had a vet check the cockerel out? Make sure there isn’t some disease that causes the other chickens to try drive him away.
Other than that, you could try segregating him for a while, give him some extra food to eat to help make him bigger. Once he gains some size and age, (and providing he is healthy) they will likely stop attacking him.
And if you ever have a rooster that attacks you or the other members of your family that might enter their pen, let me know. I have a cure for that too!