I took a basic physics course in high school, but did not learn anything. I took physics my first year in college and survived. I had a strong math background though, having already took my first calculus class in high school.
My college offered calculus and non-calculus based physics – only non-science majors took the latter. You need to know basic algebra and how to derive and integrate to understand basic physics.
Although all the sciences are connected and were in fact not segregated into different fields until the recent past (in relative terms), what you learn in high school biology and chemistry is so basic that the connection to physics may not be real clear. In other words, it will probably be very new to you.
I think I remember you saying you were going to study wildlife biology or ecology. In this case, I don’t think postponing physics for a future term would hurt you much, although I’m not sure. If you put it off, you should make sure that it isn’t a prerequisite for a number of other courses.
I recommend checking out MIT’s Open Courseware for basic physics. You can look at the topics covered, in as much or as little detail as you’d like, and decide for yourself. The book I used for college physics was by Paul A Tipler. Not great, but not bad either.