“Off topic” is a legitimate reason to downvote a post or comment, even one made respectfully and in good faith.

I do sometimes wish more sites had adopted something like the system Slashdot used, with multiple categories of up or down votes (insightful, informative, off-topic, flamebait, etc.) which users could weight according to their own preferences. The simplistic “either up, down, or neutral” model is a rather blunt instrument.