Look up information specific to the waters. There are parasites and more commonly there are pollutants.
And yes, the bigger ones are more of a problem. The little ones ingest things like mercury and the big ones eat the little ones and they get a concentrated dose.
Where I live, they advise eating big fish like trout and salmon from Lake Michigan no more than once a month, and little fish like smelt once per week (here’s a chart) .
I don’t eat it at all, the fact that there is any advisory kills my appetite.
On the other hand, I do eat trout from Lake Superior and rivers in northern Michigan. And I’m going to Colorado soon and we eat trout there, too.