I say, if you want to hang out in San Diego and enjoy some lazy beach culture, that's fine. If you want to see interesting things, then 2 days at most will get you everything worth seeing.
If you're going into Mexico, I recently came back from Guanajuato. It was a beautiful city, very cramped though...