Objection 1. It would seem that the angel guardian sometimes forsakes the man whom he is appointed to guard. For it is said (Jeremiah 51:9) in the person of the angels: "We would have cured Babylon, but she is not healed: let us forsake her." And (Isaiah 5:5) it is written: "I will take away the hedge"--that is, "the guardianship of the angels" [gloss]--"and it shall be wasted."
Objection 2. Further, God's guardianship excels that of the angels. But God forsakes man at times, according to Psalm 21:2: "O God, my God, look upon me: why hast Thou forsaken me?" Much rather therefore does an angel guardian forsake man.
Objection 3. Further, according to Damascene (De Fide Orth. ii, 3), "When the angels are here with us, they are not in heaven." But sometimes they are in heaven. Therefore sometimes they forsake us.
On the contrary, The demons are ever assailing us, according to 1 Peter 5:8: "Your adversary the devil, as a roaring lion, goeth about, seeking whom he may devour." Much more therefore do the good angels ever guard us.
I answer that, As appears above (Article 2), the guardianship of the angels is an effect of Divine providence in regard to man. Now it is evident that neither man, nor anything at all, is entirely withdrawn from the providence of God: for in as far as a thing participates being, so far is it subject to the providence that extends over all being. God indeed is said to forsake man, according to the ordering of His providence, but only in so far as He allows man to suffer some defect of punishment or of fault. In like manner it must be said that the angel guardian never forsakes a man entirely, but sometimes he leaves him in some particular, for instance by not preventing him from being subject to some trouble, or even from falling into sin, according to the ordering of Divine judgments. In this sense Babylon and the House of Israel are said to have been forsaken by the angels, because their angel guardians did not prevent them from being subject to tribulation.
From this the answers are clear to the first and second objections.
Reply to Objection 3. Although an angel may forsake a man sometimes locally, he does not for that reason forsake him as to the effect of his guardianship: for even when he is in heaven he knows what is happening to man; nor does he need time for his local motion, for he can be with man in an instant.
Source: Summa Theologica (New Advent)