Tradition
St. Ambrose
On the Death of Satyrus
Book 2 › Section 24
24. Holy Joseph experienced the hatred of his brethren, the guile of those who envied him, the service of slavery, the mastership of merchantmen, the wantonness of his mistress, the ignorance of her husband, and the misery of prison.
Source: On the Death of Satyrus (New Advent)