Since @cyndihugs is too lazy to summarize, so people don’t have to click on the link…
It appears no one really knows.. It seems to have been traced back into the 1880’s, used by baseball players, possibly referring to a particular player named ‘Charley’. A horse that was ‘retired’ was also called a ‘Charley’.
The latter seems the most probable origin, IMHO