The problem of downsizing to a condo is the outrageous condo fees.
Even if I sell my larger house and pay cash for a condo, the condo fees are in perpetuity, and are likely to go up over time.
At $300 a month, that's $3600 a year. I could easily pay a neighborhood kid to mow my lawn and shovel snow for less than that.
And then I don't have my neighbors walking over top of me.