She said, "You are to answer 'Yes, sir' or 'No, sir' to him. That's a WHITE man!"
Your Grandmother was brought up (same as my grandparents) at a time when they were taught to address their 'superiors' (don't read anything untoward in that) in a certain manner. My grandfather would always address anyone in a superior position as 'sir' and tip his hat.
On a positive note, think about how much joy you gave to this (probably lonely) man.