Are there any statistics to see if a categorical variable produces good segments within a scatter plot?

Question

Within a scatterplot of two continuous variables. I am looking for a statistical method to determine if a 3rd variable that has a limited number of categories/groups (3-8) is able to produce visually well-defined segments within the original scatter plot.

The following scatterplot would be a example of a good segmentation:

It is clear from this scatter plot that the different scatters are well grouped and are distinct from the other.

Compare that with this one:

This one doesn't have any distinct groups. It may be true that there is some significant correlation/association for all three, i.e., the greater the population, the greater the area, but there are no clear "segments" here.

Initially I have had the idea of measuring the average distance from centroid, and calculate the percentage of dots of a group that are not intersecting with other groups excluding outliers (e.g. IQR method). This could work, but how would I get statistics from it that would signify a good vs bad segmentation? And before I reinvent the wheel, I wanted to check if there are any known/used statistics already.

Or perhaps a test for unimodality of the residuals (dip test)? If it is not unimodal I could try to get the modes? Anyhow, any pointers to robust methods appreciated.

Addendum: I've applied Christian's Hennig's answer using a Calinski-Harabasz Index to dervive the stats. For reference, I published the results here (PDF). The plots are ordered by p values with the Null Hypothesis of independence. p<0.05 indicates a stronger evidence of visual well defined k clusters, which can be nicely seen within the plots.

Answer 1

This is an interesting and large question and no answer is likely to seem complete.

You can take the question further graphically and you can take it further numerically. Existing methods do help and so I see little or no call to invent methods ad hoc.

Graphics

Your first plot already includes ellipses fitted somehow and indeed the extent to which those ellipses do or do not overlap gives a graphical handle on the question.

A once fashionable and in my view unduly neglected method plots convex hulls for each group or category, or convex hulls of points not on the convex hull, and so on -- offering compromises between inclusiveness and robustness or resistance of summary. See e.g. https://www.statalist.org/forums/forum/general-stata-discussion/general/1517556-convex-hulls-on-scatter-plots for some simple examples.

A plot like your second is likely to be seem confusing to all. Different methods include plotting groups separately in a series of small multiples or (sometimes best of all) plotting each group separately but with a backdrop of all the other points. This method has been dubbed that of front-and-back plots. See e.g. https://journals.sagepub.com/doi/pdf/10.1177/1536867X211025838

Numerics

The importance of the categorical variable as a extra predictor in regression or similar models is usually best assessed by declaring as it as a factor variable to your software and fitting more complicated models in which each group may have a different intercept, or a different slope, or both. The measure of whether groups differ is how far they make different predictions of the outcome variable.

Answer 2

You need two steps:

1. Some way of modelling the distribution of the different categories
2. Comparing the distributions of the different categories.

There are many different ways to model distributions and to compare the difference between distributions.

A classical example would be MANOVA which models the mean and covariance matrix of the different distributions (and assumes equal covariance of the different distributions) and compares the variance within the groups and between the groups as a measure of the difference between the groups.

If the covariance for the different groups differs then you could use a quadratic classification model and use some performance measure of the model in predicting the right classes as a measure for the difference between categories.

For more fancy distributions you can use more fancy classification schemes. With a nearest neighbors algorithm you could approximate some sort of divergence measure (if I search on google with keywords 'nearest neighbours compute divergence' then I get several suggestions).

Answer 3

In cluster analysis, the Silhouette coefficient (SC; or Average Silhouette Width) is a distance-based statistic that measures the quality of a clustering, i.e., to what extent the objects are closer to other objects in the same class than to the closest class to which they don't belong.

This can also be computed for situations as yours in which there is a given grouping; for these data probably the Euclidean distance makes sense.

One qualification is that clusterings found by a cluster analysis method (for with the Silhouette was originally meant) tend to be better separated than data from underlying groupings that have a fairly large variation. Therefore I'd recommend to contrast the SC obtained for your categories (which may look disappointingly low for people who know typical values in cluster analysis) with a permutation test approach, i.e., simulate 1000 (say) data sets where you randomly reshuffle the group labels, compute the SC for all of these, and have a look to what extent (measured in terms of standard deviations of the permutation results, say) the SC in your data is "significantly" larger.

The webpage also mentions a Simplified Silhouette that comes with less computational effort.

Sleeping over this, I realised that I should also mention another classical cluster validity index, the Calinski-Harabasz index (CH), In R here. It can once more be calibrated (or a statistical test be run) using the permutation principle. More than the SC, this is based on the standard statistics characterising the Gaussian distribution, namely mean vector and sums of squares, so will be appropriate for within-group distributions that are not too far from the Gaussian. It is based on (multivariate) Analysis of Variance logic. In fact, as @Stephan Kolassa correctly noted, both the SC and the CH will reward classes with large within-class homogeneity, whereas (potentially nonlinear) classes with larger within-class variation may not be assessed as good.

Answer 4

I interpreted

visually well-defined segments

as "separable in some (natural) space parametrization". I assume that for you, this is not a case of visually well-defined segments:

Further, your first image seems to suggest a GMM-type geometry, which is a natural choice. Since you already know the categorical part of your data $D$ (acting here as class assignment $k_i\in\{1...K\}$), you also have the (MLE) Gaussian Mixture Model fit $\hat{M}_D$ (no EM algorithm needed). Now you could compute a goodness of fit $T_{\hat{M}_D}$ of your model $\hat{M}_D$ by summing all pairwise Kullback-Leibler divergences $\text{KL}(\mathcal{N(\mu_i,\Sigma_i)}\,||\,\mathcal{N}(\mu_j,\Sigma_j)),\,\,(1\leq i, j \leq K)$ of Gaussians (which is never $\infty$ due to infinite support).

Two datasets $D_1$, $D_2$ with the same number of "classes" $K$ should be comparable by $T_{\hat{M}_{D_1}}$ and $T_{\hat{M}_{D_2}}$, where higher values of this "test statistic" (I have reasons not to call it that) indicate better visual separation.

EDIT: Normalization w.r.t. $K$ could be achieved by taking the average or maximum over the KL divergences (instead of summing). If you have a lot of datasets, you could also compare these aggregations against your (empirical) distribution of $T_{\hat{M}_{D_i}}|D_i$ to arrive at an absolute threshold/measure, similar to a p-value.