If someone needs that "hold on" so to speak - I guess I pity them.
Many, many years ago I used to feel the same way. Now I understand that pity is not helpful, one bit. Respect is the way.
So, if I don't feel bad telling a normal guy "no thanks," then I don't feel bad telling a guy in a wheelchair, "no thanks."
It would be different if he were trying to solicit a donation to some sort of fund--unfortunately I'm still a sucker for that. It's a bit harder to resist.