While nominally derisive, there's often an element of affection or amusement in the term -- the old guy down the street who constantly complains at his own neighbor can be a codger, but the one who calls the police every night about the noise in your place like as not is not. It can also include elements of miserliness, depending on how it's used. The origin of this word is unknown, being 18th century slang (first citation from 1756) that made it into the everyday register of the language -- the most likely theory is that it's an alteration of cadger, meaning a beggar. The explanation that it's an abbreviation of coffin dodger is almost certainly post-hoc folk etymology.
On my next lap, when I passed the bench, the codger was shaking his cane at a squirrel that had taken his hat up into a tree.