Kind of depends what you’re switching from… eg. political science to public policy might not be a big deal because some of the classes might overlap, but political science to biology or engineering would mean you would have to catch up on a lot.
Personally, I feel junior year is too late, but that’s just me and it depends on a lot of factors. If you know you’re never going to be really happy in the direction that you’re currently going, then I would make the switch. It’s worth the extra work. If you think you might just be unsure or having cold feet, talk to your advisor and figure it out.