I’d be turned off by a writing style that was too casual for the subject matter, but that doesn’t mean it has to be dry and featureless. Steven Pinker, for example, has a vigorous, readable style, if a little self-important (sometimes to the point of being laughable). Mlodinov, Dennett, and Sagan all address an intelligent, educated audience that is not made up of academics and experts in their fields. The style says a lot about who you expect your audience to be, what you think of them, and how you want them to receive your work, so if you are clear about your intended audience, the appropriate style should follow.
The writing should not talk down to the reader, should not try to be hip and slangy, and should stay away from cuteness and coyness. But it’s fine to use contractions, casual figures of speech, and the second-person pronoun when you’re not writing a formal scholarly work.
I’m not one who favors a huge amount of white space on the page, such as keeping the left third of the page blank except for side heads and possibly illustrations. But I do like margins wide enough for notations if the book is nonfiction.
Personally, I prefer a serif font, fairly classic in style, or a modern adaptation of a classic font. San serif heads are okay. A lot depends on the type of book: textbook, popular science, travelogue, essays of opinion, political rant, analytic discourse? Different designs.
However your draft turns out, make sure it passes through the hands of a competent editor before you release it. Self-publishing removes a lot of obstacles to publication, but unfortunately it also removes some well-placed checkpoints that serve the interests of both writer and reader.