Generally speaking, you have the right to photograph/videotape anything deemed in public, that is to say, the person does not have a reasonable expectation of privacy. Shooting a couple in the park: legal. Shooting them in their house: illegal. Parade? Fine.
The property you are on plays into your rights too. If you are on private property (in a mall) you are subjected to the rules of the establishment, which many times prohibits photos/videos. If they ask you to stop, you should stop because you have no right to continue filming. They have the right to eject you from the premise. They do not have the right to touch or take your camera, nor to delete your photos. Standing up for your rights in such situations is at times tough, and it may be easier to delete the (or one) photo and turn the camera off to cause the situation to end.
What you do with the photo/video plays a lot of impact towards your rights. You cannot advertise someone as giving an opinion without their explicit consent (a model release sheet). So that photo of the couple in the park, you can’t use it to advertise your dating website by adding a speech bubble about how happy the couple is. In fact, I’d be hard pressed to use their image for anything commercial without written consent.
Generally speaking, if a person does not want to be filmed, you should comply and stop filming them unless you have a newsworthy or seriously valuable reason for filming them (a crime in progress, for example, a calamity, etc). Not doing so stands the risk of escalating the encounter, and many photographers have had their cameras smashed because they simply wouldn’t back down. People have rights too, just because you have a recording tool doesn’t mean their rights are ignored or somehow lessened. Of course, you can litigate regarding the broken equipment, but again, if one had just stopped filming, the entire sticky situation would not exist.