I try to reserve downvotes for things that I think have negative value. If I simply don't like something, I ignore it and move on; just because I don't like it doesn't mean that it lacks value for others. If I don't understand something, and it's important enough for me to act on, I'd rather have a discussion and learn.
I'll downvote posts that don't fit the purpose of a community, or which were clearly posted to start an arguement. I'll downvote comments that are insulting, inflammatory, or otherwise malicious.