I agree with the others about the Rosicrucian Museum; definitely worth a couple hours of your time. If you like the outdoors, Alum Rock Park is a nice place to hike. It’s a bit south of San Jose, but Almaden has some nice hiking trails, as @Jeruba already pointed out.
It’s not in San Jose, but if you enjoy California history, there are two missions fairly close by: Mission Santa Clara (in Santa Clara, natch), and Mission San Jose (which, despite its name, is actually in Fremont, just northeast of San Jose). If you have children, Happy Hollow is a good place to take them, to work off some energy. If they enjoy miniature train rides though, go a bit south to Los Gatos to Oak Meadow Park and ride Billy Jones Wildcat Railroad.
However, I agree with @Jeruba; there really isn’t that much to do in San Jose compared with San Francisco, and as @Jeruba points out, there are better places that are close by that you could go, such as Santa Cruz (the Boardwalk is always a good outing, especially since Playland has been gone for decades now), and Monterey (the Monterey Bay Aquarium, while pricey, is worth a visit, and the State Historic Park is a great way to learn about California history and old California). I spent part of my childhood just south of San Jose, and aside from the few things I and others have mentioned, there isn’t a whole lot to do. It’s basically miles and miles of suburbia surrounding downtowns here and there, that has eaten up some of the best farmland in the state (the Santa Clara Valley used to be acres and acres of fruit trees and farms…).
I’m rambling on, but you get the point. Do go to San Jose, just for the experience, but don’t spend a lot of time there. There’s far better places to go and things to do in this state of ours. :)