You know, I've got a fretless banjo and I have never been able to play it as well as I can a banjo with frets.
The banjo is not an easy instrument. You pretty much have to use all of your fingers at once and each of them is doing something different.
In the midst of all that activity the frets help you to keep track of your hand as it moves around the neck, or at least it helps me. I think there would just be too much going on without the frets. Your mind would have to concentrate on more than the music.
I know that some people love playing this way but not me. Give me frets any day.