The people responsible for the poor are:
1. poor people themselves
2. the governments they live under.
Do you really believe this, honestly? So you never give to charities like Oxfam? The worst thing about being in that cult in the eighties was seeing the Ethiopian famine on TV but being told not to donate to famine relief. I felt really bad about that after leaving.
I think helping the poor is everyone's responsibility just as it would be if you knew them personally. Would you let kids with pot bellies and stick limbs starve if they lived in your street? It doesn't make any difference that we don't know them.
Perhaps I'm misunderstanding your post but I hope you do believe in famine relief, and making the world fairer for everyone. Some desperately poor people just need a leg up to work their way out of poverty.