Google’s spiders/robots that index images rely completely on alt tags. When an image is placed in a website the code looks like this:
<img src=“fluther.gif” alt=“lurve” />
Since the robots are unable to visually evaluate images they use the alt tag and other page content to determine an images subject mater. The reason you get so many unrelated images is because alt tags are often left blank and the robots fall back to other content on the page, which may or may not be directly related to the image.