9 Respecting vigils and the frequent keeping of night-watches in the basilicas of the martyrs, I have given a brief reply in another letter which, about two years ago, I wrote to the reverend presbyter Riparius. You argue that they ought to be abjured, lest we seem to be often keeping Easter, and appear not to observe the customary yearly vigils. If so, then sacrifices should not be offered to Christ on the Lord's day lest we frequently keep the Easter of our Lord's Resurrection, and introduce the custom of having many Easters instead of one.
We must not, however, impute to pious men the faults and errors of youths and worthless women such as are often detected at night. It is true that, even at the Easter vigils, something of the kind usually comes to light; but the faults of a few form no argument against religion in general, and such persons, without keeping vigil, can go wrong either in their own houses or in those of other people. The treachery of Judas did not annul the loyalty of the Apostles. And if others keep vigil badly, our vigils are not thereby to be stopped; nay, rather let those who sleep to gratify their lust be compelled to watch that they may preserve their chastity.
For if a thing once done be good, it cannot be bad if often done; and if there is some fault to be avoided, the blame lies not in its being done often, but in its being done at all. And so we should not watch at Easter-tide for fear that adulterers may satisfy their long pent-up desires, or that the wife may find an opportunity for sinning without having the key turned against her by her husband. The occasions which seldom recur are those which are most eagerly longed for.
Source: Against Vigilantius (New Advent)