This is a tough question, but I think a lot of the problem lies in our expectations. We think that the work of the Spirit inside us should feel a certain way or produce certain results, but these expectations are rarely accurate. Sure, in Acts, the giving of the Spirit (as an inaugural phenomenon) is experiential and involves tongues and fire, but after that, the disciples are still messing things up. Paul even rebukes Peter for being a hypocrite and he himself has a bust-up with Barnabas that sees them parting ways and not working together for a long time. The Holy Spirit doesn’t prevent even the apostles from making serious mistakes.
The problem is that we are most likely never to get any tangible confirmation that God is there—some supernatural event in which God manifests himself—and the work of the Spirit is hard to discern. I think that God’s presence in the world is hard to see because it is like a fish being in water. God is not “in” creation like the Greek gods were (he’s bigger than what he made and beyond it), but creation is, in a sense, in him. Perhaps we fail to see God because he is always visible, not because he’s never visible. We want God to appear, as if he is outside creation and should come into it to visit us, and God could obviously manifest himself in special ways if he were to choose. However, I think there actually just isn’t a time when he is absent. We shouldn’t associate God only with the supernatural. The “laws of nature” are God’s work. “In him we live, move, and have our being.” God is the animating force of our normal. (He is more than this, but not less.)
Similarly, the work of the Spirit is also hidden and, while it is internal to individuals, the goal of the Spirit’s work is to produce communities—the body of Christ, the church. And he is doing this by working with broken and rebellious people. Change is gradual, and I suppose we sometimes even regress.
In your email, you talk about knowing what your parents/church taught you to do and say, but now not being sure that you want to follow this path. Of course it is right for them to have given you such guidance, but this kind of obedience belongs to childhood, not maturity. Obedience is a good lesson that we need to learn all our lives, but we obey the authorities of our childhood out of fear and a sense of our own smallness and dependence. As we grow up, both in our psychology and our circumstances, we gain more and more independence and power. This forces us to consider our position in regard to our parents’ beliefs. We don’t have to believe what they believe anymore. At some point we have to decide whether we actually believe what we’ve always been told.
The funny thing is that Christianity is, in the end, a call to genuinely recognise our own smallness and dependence on God. The true heir of the kingdom is the child. But each thing in its turn. You have to leave literal childhood behind and become mature before you can recognise (and choose for yourself with open ears and eyes) that true life and strength lies in God alone.
So what you seem to be experiencing, and what all of the life of faith will entail, is the contest between strength and weakness. You’re growing in strength outwardly, which lets you leave the weakness of childhood behind. This makes the question of your need for a faith of weakness (i.e., biblical Christianity) all the more pressing. And yet you’re also becoming more aware of the inner weaknesses that you wish you were able to master but can’t. You want the Spirit to grant you mastery of your inner self, but the weakness persists.
You mentioned listening to Piper talk about the work of the Spirit from Romans 8 (being led by the Spirit and having our hearts changed, being made to desire heavenly things not worldly things, being eager to tell the world about Jesus), and you don’t see these things in you. But remember that the Bible has a lot else to say about it. Consider what Jesus says about the Comforter/Advocate in John 16:7-11:
“Nevertheless, I tell you the truth: it is to your advantage that I go away, for if I do not go away, the Helper will not come to you. But if I go, I will send him to you. And when he comes, he will convict the world concerning sin and righteousness and judgment: concerning sin, because they do not believe in me; concerning righteousness, because I go to the Father, and you will see me no longer; concerning judgment, because the ruler of this world is judged.”
In this text, the role of the Spirit is to “convict the world concerning sin and righteousness and judgment”, which is puzzling in itself, and Jesus’ explanation does not make it immediately clear what this means (at least not to me). I think these verses are saying that the Spirit continues Jesus’ work across the whole world and does so through his people.
The Spirit will “convict the world concerning sin, because they do not believe in me” suggests that the primary sin is to reject Jesus, and the Spirit’s work will be to turn hard hearts to faith.
To “convict the world concerning righteousness, because I go to the Father” probably means that Jesus had provided a visible the pattern of righteousness and the path by which to be made righteous, but he went away. The Spirit’s work is to continue to make Jesus seen, presumably by his followers passing on Jesus’ teaching and imitating his way of life.
To “convict the world concerning judgment, because the ruler of this world is judged” suggests that the Spirit enables us to apprehend the defeat of sin and death, and so to align ourselves with the one who has won the victory—i.e. we understand how the cross passed judgement on evil, and how it enables us to avoid sharing in evil and judgement.
So, yes, the work of the Spirit is to do those things that Piper mentions, but we shouldn’t only think about it on the level of the individual. John 16 is focused much more on the Spirit’s work in projecting Jesus across the world stage—what Jesus was for his disciples and Israel, the Spirit is for the church and the world. Remember that when Paul gets to talking about the fruits and gifts of the Spirit, these things are also directed towards building the church, not primarily the individual. You are not meant to be a one-man army. You will struggle with things that others don’t. You have gifts that others don’t. The point is that we grow in Christ and image Christ together.
1 John has a lot to say about assurance, and the ground of assurance that John talks about is interesting.
“This is the message we have heard from him and proclaim to you, that God is light, and in him is no darkness at all. If we say we have fellowship with him while we walk in darkness, we lie and do not practice the truth. But if we walk in the light, as he is in the light, we have fellowship with one another, and the blood of Jesus his Son cleanses us from all sin. If we say we have no sin, we deceive ourselves, and the truth is not in us. If we confess our sins, he is faithful and just to forgive us our sins and to cleanse us from all unrighteousness. If we say we have not sinned, we make him a liar, and his word is not in us.” (1 Jn 1:5-10)
John starts by telling us that we must obey, walk in the light, etc. This might make you think that assurance depends on us ridding ourselves of sin and being perfect. But with the next breath he tells us that we have always been and will go on being sinners—and, he adds, if you think that’s not how it is, you are not walking in the truth. The ground of our assurance is not in our own moral goodness, but in “the blood of Jesus his Son [which] cleanses us from all sin”.
In case we’re confused, the next passage does the same thing in reverse order. 1 Jn 2:1-6 tells us that when we sin, we have an advocate who speaks on our behalf—Jesus, who died to remove sin—but it also reminds us that knowing Jesus means imitating Jesus. We remodel our way of living.
“My little children, I am writing these things to you so that you may not sin. But if anyone does sin, we have an advocate with the Father, Jesus Christ the righteous. He is the propitiation for our sins, and not for ours only but also for the sins of the whole world. And by this we know that we have come to know him, if we keep his commandments. Whoever says “I know him” but does not keep his commandments is a liar, and the truth is not in him, but whoever keeps his word, in him truly the love of God is perfected. By this we may know that we are in him: whoever says he abides in him ought to walk in the same way in which he walked.”
1 Jn 2:15-17 is also useful:
“Do not love the world or the things in the world. If anyone loves the world, the love of the Father is not in him. For all that is in the world—the desires of the flesh and the desires of the eyes and pride of life—is not from the Father but is from the world. And the world is passing away along with its desires, but whoever does the will of God abides forever.”
John talks about the realignment of our loves. He contrasts love of the world with love of God. Love of the world and its desires are worthless because these are “passing away”. I think it’s fair to say that “passing away” is not only a reminder of the future death and re-creation of the world (the new heavens and the new earth at the End), but also that the pattern of this world is marred by its past (the fall) and is associated with death in every way.
We naturally do love the world because our desires are powerful, but these loves are bound up with sin and death (because this world is tied to sin and death too). This is ultimately self-love and there is no future in this way. It is what perpetuates the grip of sin and death.
By contrast, loving God expresses itself in brotherly love too (as John goes on to say repeatedly). Love that is directed towards the other is the pathway to life and to the undoing of sin and death. The work of the Spirit is to unravel our natural desires and to help us to have better loves. And this is again something that occurs in community—we learn to love and help one another. It’s not something that happens easily or instantly simply by having prayed the sinner’s prayer.
In short, Christian assurance is neither based on you conquering pride, selfishness, sexual sin, etc. nor OK with those things. The Spirit within you means that you are made holy in status, once and for all (you are made fit for God’s presence and service), and that you are labouring with your fellow disciples to perceive your own sin and to align yourself with that holiness that you have been given.