You say that it is a ""curve of neutral stability", so you would use the first of these - hyphenate.
The point of joining "neutral" and "stability", in "neutral-stability", is to apply the compound adjective of neutral stability to the curve. The point of not joining them is to apply "neutral" to a stability curve, that is, apply "neutral" to the application of "stability" to "curve".
This is a general rule that you can often use to increase clarity. But it is often not followed, and there are no doubt exceptions where it might even reduce clarity. A lot can depend on whether the compound adjective really make sense - e.g., whether there is such a quality as "neutral stability".