Based on my (limited) research, you cannot detect if botulism has caused problem in the food – it doesn’t change at all. The indication, according to http://wiki.answers.com/Q/What_are_the_signs_of_botulism_in_home_canned_foods, of botulism in your jam is that people get sick and/or die. Just because it goes bad doesn’t mean that it’s botulism. Most botulism cases in the US are found in Alaska (around 40%) and mostly in meat and fish products canned by the natives, not because they are natives, but probably because they can meat and fish! It seems that all fruit and vegetables contain botulism spores to some degree. Acidity seems to prevent them from giving off toxins.
There are mathematical rules to how long and at what temperatures food should be cooked to kill the Clostridium botulinum bacteria. See http://www.nzifst.org.nz/unitoperations/httrapps2.htm for detailed information. Botulism does not create mold on top, that is completely different and completely unrelated and is probably not going to kill anyone. It was common with jam (rhubarb mostly, but also blueberry) that my mom made that it would get a layer of mold on top – we scraped it off and ate the rest and it never affected anyone. She would cook it for hours and she’d put the jars in the oven for an hour or so before putting the jam in. She never sealed the jars with anything except just a screw-on lid and the jars were used over and over again for years.
While it is certainly extremely important to maintain good hygiene in the kitchen at all times and prevent any kind of cross-contamination, people should be careful to not get too paranoid. People have been making jams for hundreds of years with much less hygiene than we can even think of and enough of them made it through to make us:) The first jams came to Europe from the Middle East during the middle ages and in the late 17th century books were being published here in the US about how to make jams. Botulism wasn’t discovered for another two centuries and nobody had pressure cookers or cans, but somehow people made jam and survived:)
Happy jam making:)