Another alternative could be a new category which has become increasingly more prolific in the last several years.
They’re usually advertised as “loungepants” and they’re quite loose and comfortable. Truth be told, they’re basically pajama pants but with the adddition of deep pockets tyically found in regular slacks and a button fly rather than no buttons.
They are as comfortable as sweats, but not so “grungy” looking and made of lighter material, typically a stretchy poly-blend or sometimes flannel.
The degree to which they will look as if you’ve just stumbled put of bed in your pajamas is more dependent upon choice of fabric and, most importantly, pattern.
You really don’t want something with a South Park cartoon print (I have two different ones like that) or outrageous plaid flannel or similar for outdoor wear. That’s best for indoor use only.
But they do have them in solid black or gray or even a subdued black-watch plaid, which should do the trick. I have about a half dozen or so for both indoor and outdoor wear. I’ve never gotten any strange looks or comments. They are very very comfortable.
As long as you don’t wear them to a job interview, you should be fine.
Or you can just continue wearing shorts and the h*ll with what anonymous strangers think or say about it. They don’t get a vote. You’re certainly not violating and public decency statutes, so it’s basically none of their business. Wear whatever you’re comfortable wearing.