Some people are allergic to sisal or other kinds of poufs, so make sure you pick a pouf that isn't too tough on your skin, and that doesn't irritate it.
Exfoliating is good, but it's not good to use a pouf that does a lot of exfoliation every day, unless your skin is very, very flaky. Even then, caution is advised.
Be especially careful when using a facial pouf. When the Buf Puf came out, I went overboard with it on my face and my skin got red and irritated and I had to deal with it for about three days. I'd recommend a nylon pouf for the face. It's difficult to scrub too hard with one of those.