Yep, always stay with it if you're roasting almonds, pecans -- any kind of nut. They burn in a hurry and without warning. They go from nicely toasted to black and crispy while you watch. So keep your eyes on them.
Don't have your skillet too hot, either. If it is, it can burn the nuts on the outside and leave the insides raw. Start with a moderate heat and increase it if necessary. It's better to sneak up on roasting nuts. You're roasting -- not sauteeing.