I had the same issue with the first sulfur soap I bought. It was terribly drying and gave me flaky skin. Oily, acne-prone skin doesn't mean the skin is hydrated, so I think we still need a cleanser with some moisturizers in it.
I then bought a different brand sulfur soap with a low sulfur %, it's 2% and has a non-comodogenic oil in it. It's enough to prevent bacterial acne but still keeps my face moisturized.
Some people also expect sulfur soap to clear up acne scars. Well, that doesn't happen but it does prevent future breakouts and scars.