4 For many weep with the weeping of Babylon, because they rejoice also with the joy of Babylon. When men rejoice at gains and weep at losses, both are of Babylon. You ought to weep, but in the remembrance of Sion. If you weep in the remembrance of Sion, you ought to weep even when it is well with you in Babylon....
5. “On the willows in the midst thereof we hung up our instruments of music”. The citizens of Jerusalem have their “instruments of music,” God's Scriptures, God's commands, God's promises, meditation on the life to come; but while they are dwelling “in Babylon,” they “hang up their instruments.” Willows are unfruitful trees, and here so placed, that no good whatever can be understood of them: elsewhere perhaps there may. Here understand barren trees, growing by the waters of Babylon. These trees are watered by the waters of Babylon, and bring forth no fruit; just as there are men greedy, covetous, barren in good works, citizens of Babylon in such wise, that they are even trees of that region; they are fed there by these pleasures of transitory things, as though watered by “the waters of Babylon.” You seek fruit of them, and nowhere findest it....Therefore by deferring to apply the Scriptures to them, “we hang up our instruments of music upon the willows.” For we hold them not worthy to carry our instruments. We do not therefore insert our instruments into them and bind them to them, but defer to use them, and so hang them up. For the willows are the unfruitful trees of Babylon, fed by temporal pleasures, as by the “waters of Babylon.”
6. “For there they that led us captive demanded of us words of songs, and they that led us away, an hymn”. They demanded of us words of songs and an hymn, who led us captive....We are tempted by the delights of earthly things, and we struggle daily with the suggestions of unlawful pleasures; scarce do we breathe freely even in prayer: we understand that we are captives. But who led us captive? What men? What race? What king? If we are redeemed, we once were captives. Who has redeemed us? Christ. From whom has He redeemed us? From the devil. The devil then and his angels led us captive: and they would not lead us, unless we consented....
7. “Those” then “who have led us captive,” the devil and his angels, when have they spoken unto us: “Sing us one of the songs of Sion”? What answer we? Babylon bears you, Babylon contains you, Babylon nourishes you, Babylon speaks by your mouth, you know not to take in save what glitters for the present, you know not how to meditate on things of eternity, you take not in what you ask. “How shall we sing the Lord's song in a strange land?”. Truly, brethren, so it is. Begin to wish to preach the truth in such measure as you know it, and see how needful it is for you to endure such mockers, persecutors of the truth, full of falsehood. Reply to them, when they ask of you what they cannot take in, and say in full confidence of your holy song, “How shall we sing the Lord's song in a strange land!”
Source: The Enarrations, or Expositions, on the Psalms (New Advent)