I just spoke to my roommate with your question, as she returned from a weekend trip to Toronto yesterday: Yes, you need your passport.
If you’re driving, as she did, it will take about an hour to cross the border because the US officers question everyone in the car and do some searching. They searched her trunk when she entered Canada and quizzed her and her friend extensively when she returned to New York State.