Craig,
Perhaps one answer to your question (in addition to the obvious, i.e. that "physical" is the default, literal meaning of "suffering" unless a derived,figurative or metaphorical meaning is contextually implied) lies in the NT texts which explicitly parallel Jesus' sufferings with the Christians' sufferings. There are many of them in Paul (especially 2 Corinthians; Romans; Philippians) and the post-Pauline (Colossians), Acts and Hebrews. Perhaps the highest concentration is found in 1 Peter, where physical clues abound:
Slaves, accept the authority of your masters with all deference, not only those who are kind and gentle but also those who are harsh. For it is a credit to you if, being aware of God, you endure pain while suffering unjustly. If you endure when you are beaten for doing wrong, what credit is that? But if you endure when you do right and suffer for it, you have God's approval. For to this you have been called, because Christ also suffered for you, leaving you an example, so that you should follow in his steps.
"He committed no sin,
and no deceit was found in his mouth."
When he was abused, he did not return abuse; when he suffered, he did not threaten; but he entrusted himself to the one who judges justly. He himself bore our sins in his body on the cross, so that, free from sins, we might live for righteousness; by his wounds you have been healed. -- 2:17ff.
For it is better to suffer for doing good, if suffering should be God's will, than to suffer for doing evil. For Christ also suffered for sins once for all, the righteous for the unrighteous, in order to bring you to God. He was put to death in the flesh, but made alive in the spirit. -- 3:17f.
Since therefore Christ suffered in the flesh, arm yourselves also with the same intention (for whoever has suffered in the flesh has finished with sin), so as to live for the rest of your earthly life no longer by human desires but by the will of God. -- 4:1f.
Beloved, do not be surprised at the fiery ordeal that is taking place among you to test you, as though something strange were happening to you. But rejoice insofar as you are sharing Christ's sufferings, so that you may also be glad and shout for joy when his glory is revealed. If you are reviled for the name of Christ, you are blessed, because the spirit of glory, which is the Spirit of God, is resting on you. But let none of you sufferas a murderer, a thief, a criminal, or even as a mischief maker. Yet if any of you suffers as a Christian, do not consider it a disgrace, but glorify God because you bear this name. -- 4:12ff.
Edited to add the famous Oscar Wilde quote: "God, spare me physical pain; I can cope with moral suffering!"