If you’re tired of wearing the white dress shirt with the khakis, it sounds like the heather grey polo might look better than the charcoal grey shirt. Just color-wise, a lighter grey would probably look better with khaki pants. Especially now that the weather is warmer, khakis and a polo are a very appropriate combination for a business casual environment.
You said you only have those three shirts and one pair of pants to wear to work right now. When you get around to shopping for work clothes, you should look for a few items that are more versatile so you can mix and match them around with each other. That will make it look like you have more outfits, even if you only have a few sets of clothes to begin with. If you don’t buy anything else, you should get a pair of black or navy slacks. You can wear them with all the shirts you have now. Next, start with one or two dress shirts that you can wear with both your khaki pants and your dark pants. You already have a white shirt, so maybe light blue or white with pinstripes would be good.
The basics of a business casual wardrobe are pretty boring, but once you have the building blocks, you can start changing things around to make your work clothes less stuffy. The idea isn’t to dress more casual, like wearing jeans or a t-shirt, but to be more playful and fashionable with your work clothes. The work clothes you have now are at the very casual end of business casual, and they’re also kind of boring- it sounds like something an employee at best buy or target might wear. Instead of doing that, gradually build up a wardrobe that is more professional and edgier.
If when you buy your next set of work pants, you get navy blue, then get a matching jacket to go with it. (It’s important to buy the pants and jacket together so they’ll have the same fit and will be made of the same fabric.) You can wear the navy blazer with khaki or grey pants and look really sharp. Adding a blazer to a business casual outfit means you can relax the standards a bit on what shirt you will wear. If your office has the right personality, you might be able to wear a cool t-shirt under it. If your style is more alternative, you could get slightly tapered pants and skinny ties and go for a post-punk look. Another thing I see a lot is men wearing conservative suits with very bold colored dress shirts- if done right it can show that you’re professional but creative.
Anyway, that was a really long tangent. I’d wear the polo tomorrow over the charcoal shirt, but they both probably look okay.