Objection 1. It would seem that man could commit a venial sin in the state of innocence. Because on 1 Timothy 2:14, "Adam was not seduced," a gloss says: "Having had no experience of God's severity, it was possible for him to be so mistaken as to think that what he had done was a venial sin." But he would not have thought this unless he could have committed a venial sin. Therefore he could commit a venial sin without sinning mortally.
Objection 2. Further Augustine says (Gen. ad lit. xi, 5): "We must not suppose that the tempter would have overcome man, unless first of all there had arisen in man's soul a movement of vainglory which should have been checked." Now the vainglory which preceded man's defeat, which was accomplished through his falling into mortal sin, could be nothing more than a venial sin. In like manner, Augustine says (Gen. ad lit. xi, 5) that "man was allured by a certain desire of making the experiment, when he saw that the woman did not die when she had taken the forbidden fruit." Again there seems to have been a certain movement of unbelief in Eve, since she doubted what the Lord had said, as appears from her saying (Genesis 3:3): "Lest perhaps we die." Now these apparently were venial sins. Therefore man could commit a venial sin before he committed a mortal sin.
Objection 3. Further, mortal sin is more opposed to the integrity of the original state, than venial sin is. Now man could sin mortally notwithstanding the integrity of the original state. Therefore he could also sin venially.
On the contrary, Every sin deserves some punishment. But nothing penal was possible in the state of innocence, as Augustine declares (De Civ. Dei xiv, 10). Therefore he could commit a sin that would not deprive him of that state of integrity. But venial sin does not change man's state. Therefore he could not sin venially.
I answer that, It is generally admitted that man could not commit a venial sin in the state of innocence. This, however, is not to be understood as though on account of the perfection of his state, the sin which is venial for us would have been mortal for him, if he had committed it. Because the dignity of a person is circumstance that aggravates a sin, but it does not transfer it to another species, unless there be an additional deformity by reason of disobedience, or vow or the like, which does not apply to the question in point. Consequently what is venial in itself could not be changed into mortal by reason of the excellence of the original state. We must therefore understand this to mean that he could not sin venially, because it was impossible for him to commit a sin which was venial in itself, before losing the integrity of the original state by sinning mortally.
The reason for this is because venial sin occurs in us, either through the imperfection of the act, as in the case of sudden movements, in a genus of mortal sin or through some inordinateness in respect of things referred to the end, the due order of the end being safeguarded. Now each of these happens on account of some defect of order, by reason of the lower powers not being checked by the higher. Because the sudden rising of a movement of the sensuality in us is due to the sensuality not being perfectly subject to reason: and the sudden rising of a movement of reason itself is due, in us, to the fact that the execution of the act of reason is not subject to the act of deliberation which proceeds from a higher good, as stated above (Question 74, Article 10); and that the human mind be out of order as regards things directed to the end, the due order of the end being safeguarded, is due to the fact that the things referred to the end are not infallibly directed under the end, which holds the highest place, being the beginning, as it were, in matters concerning the appetite, as stated above (10, A1,2, ad 3; 72, 5). Now, in the state of innocence, as stated in the I, 95, 1, there was an unerring stability of order, so that the lower powers were always subjected to the higher, so long as man remained subject to God, as Augustine says (De Civ. Dei xiv, 13). Hence there can be no inordinateness in man, unless first of all the highest part of man were not subject to God, which constitutes a mortal sin. From this it is evident that, in the state of innocence, man could not commit a venial sin, before committing a mortal sin.
Reply to Objection 1. In the passage quoted, venial is not taken in the same sense as we take it now; but by venial sin we mean that which is easily forgiven.
Reply to Objection 2. This vainglory which preceded man's downfall, was his first mortal sin, for it is stated to have preceded his downfall into the outward act of sin. This vainglory was followed, in the man, by the desire to make and experiment, and in the woman, by doubt, for she gave way to vainglory, merely through hearing the serpent mention the precept, as though she refused to be held in check by the precept.
Reply to Objection 3. Mortal sin is opposed to the integrity of the original state in the fact of its destroying that state: this a venial sin cannot do. And because the integrity of the primitive state is incompatible with any inordinateness whatever, the result is that the first man could not sin venially, before committing a mortal sin.
Source: Summa Theologica (New Advent)