Consistency of Good Men in All Outward Circumstances
Let them believe, on the contrary, that the apostles of our faith were neither puffed up when they were honored by men, nor cast down when they were despised. And certainly neither sort of temptation was wanting to those great men. For they were both cried up by the loud praises of believers, and cried down by the slanderous reports of their persecutors. But the apostles used all these things, as occasion served, and were not corrupted; and in the same way the saints of old used their wives with reference to the necessities of their own times, and were not in bondage to lust as they are who refuse to believe these things.
30. For if they had been under the influence of any such passion, they could never have restrained themselves from implacable hatred towards their sons, by whom they knew that their wives and concubines were solicited and debauched.
Source: Christian Doctrine (New Advent)