Yes, "she's a beautiful dancer" would normally be interpreted to mean she dances well, just like a "small businessman" is normally interpreted to mean he owns a small business. There's enough ambiguity in the construction that it could theoretically be interpreted the other way around, though, so that's the source of a common form of joke -- "He's a small businessman; he's only four feet tall" is funny because it forces you to go back and reinterpret the first phrase.
The ambiguity happens because "beautiful" is a word that can apply to either a person or a performance, and in generally we read it as referring forward to the next noun rather than backward to the subeject of the sentence.
One thing you can do to fix this is to add an additional descriptor between beautiful and dancer that makes it clear that you're listing out aspects of the woman rather than describing the nature of her dance: "She's a beautiful blonde dancer" unambiguously means she's pretty, not that she dances especially well. (Similarly, "he's a small, fat, balding businessman" no longer seems to have anything to do with what kind of business he runs.)