Having mulled over this a bit, I have come to the following conclusion(s). I should note that I have no sources and am aware of no prior research to back up my claims; but I hope they bear themselves out.
There are two levels of sound change at play here, as others have said: cluster reduction and nasal assimilation.
[Note: For the sake of simplicity, I ignore dialectal variation and use generic, simplified RP for my IPA, excepting the non-RP pronunciation of some clusters.]
Cluster reduction in compounds and sandhi
First, the cluster /nd/ in handbag is reduced to just /n/. Whether a word-final /nd/ cluster is or can be reduced depends on a few things.
Morpheme boundaries
If there is a morpheme boundary between the /n/ and the /d/, the ability to reduce the cluster is significantly reduced. Thus the past tense forms penned and mined are not usually reduced in any context, even before a stop (see below).
On the other hand, if there is no morpheme boundary between the /nd/ cluster and the following element (i.e., the /nd/ is not word-final), then the ability to reduce is significantly increased. There are only a few words where non-final /nd/ is followed by anything other than a separate morpheme (like finder with the morpheme -er), but one such word is sandwich, in which /nd/ is indeed usually reduced—albeit sometimes quite irregularly to an [m], which to me makes no sense whatsoever and must be regarded as spurious and unpredictable.
Quality of preceding element
If /nd/ occurs after a phonemically long vowel or a diphthong, the ability to reduce the cluster is significantly increased. Thus mind, blind, and found can both be optionally reduced in any context, even at the end of an utterance.
If the /nd/ cluster ends an unstressed syllable, particularly one with a reduced vowel (usually [ə] or [ɪ/ɨ]), the ability to reduce the cluster is likewise increased, so that second or errand can be optionally reduced in any context. Something like imagined is less likely to be reduced, because of the morpheme boundary.
Quality of the following element
If none of the two first triggers is present, so we’re dealing with a phonemically short vowel followed by a tautomorphemic /nd/, then the ability of a given /nd/ cluster to be reduced depends on the element that follows it.
Going through the alphabet and finding/making up word pairs where the first ends in /nd/, I find the following general principles apply:
/nd/ is fairly consistently reduced to /n/ before /b d g p t k tʃ ʤ/
handbag, blonde girl, mend cars, wind chime, etc. all with reduction
/nd/ can optionally (but quite commonly) be reduced to /n/ before /l r m n f s ʃ θ/
landfill [lan(d)fɪɫ], landline [lan(d)ɫaɪn], handsome [han(t)səm], friendship [fɹʷɛn(d)ʃɪp], windmill [wɪn(d)mɪɫ], grand theatre [ɡɹʷan(d) θiːətə], handwriting [han(d)rʷaɪtɪŋ]
/nd/ is usually not reduced before vowels and /j w h v z (ʒ)/
handover [handəʊvə], grandeur [ɡɹʷandjʊə/ɡɹʷanʤʊə])
(There are some exceptions to 1. above, such as where proper names are involved: /nd/ is usually reduced in blind girl, but never (?) in Bond girl; and the generic noun sandwich is often reduced, as per above, but the place name Sandwich is usually not. Similarly, ad-hoc or uncommon compoundings are less likely to suffer reduction: landline is quite commonly reduced, but wandlore much less so. There are probably other exceptions, too.)
In other words, the reduction is:
- ubiquitous before stops (including affricates)
- optional (but common) before non-vocoid sonorants and unvoiced fricatives
- possible (but less common) before vocoid sonorants and voiced fricatives (including /h/)
This seems to more or less follow a principle of sonority: the more sonorant the following consonant is, the more likely it is to block the reduction of /nd/ to /n/. /h/ is the exception here, but it is quite common for /h/ to be voiced [ɦ] in many positions in English, and it is typologically also quite common for /h/ to pair with consonants of high sonority.
If either of the first two triggers mentioned above is present, the following element still plays a role, but the overall ability to reduce the cluster is reduced or increased relatively. So mind can be reduced even before vocoid sonorants and voiced fricatives quite readily, but it is even more likely (almost certain, even) to be reduced before a stop; whereas mined does have the possibility of being reduced before a stop, but it’s far less likely—and before vocoid sonorants and voiced fricatives, I would say that reduction would be almost entirely absent and impossible.
Parallel clusters
Let’s compare this cluster with some similar ones. /nd/ consists of a nasal (which I’ll indicate as an archephoneme as /ɴ/) followed by a voiced stop, so similar clusters are:
- /ɴ/ + /b/ => ‹mb›
- /ɴ/ + /g/ => ‹ng›
Both these clusters appear on the surface with two different pronunciations: [m] and [mb] for /ɴb/, and [ŋ] and [ŋɡ] for /ɴg/. These alternations historically came some time during the first century of Modern English (around 1600 or so), as a result of ng-coalescence and mb-coalescence, which reduced them in word-final position, but not word-internally. Note that all the cases we’re discussing here, including handbag, are word-final, they are just frequently compounds.
Whereas mb-coalescence took place in all of English, ng-coalescence skipped a few dialects. In such dialects, song and singer are [sɔŋɡ] and [sɪŋɡə], but a songbook is not [*sɔŋɡbʊk], but [sɔŋbʊk]. A songwriter is often, but not always, a [sɔŋɡɹʷaɪtə]. Without being any kind of expert on these dialects, my hypothesis would be that the same general sonority principle as unearthed above is at play, and /ng/ thus matches /nd/ quite nicely.
That basically leaves us with a description of an ongoing historical change of word-final clusters, frozen in different ‘stages’:
Productive reduction common to all of the Anglosphere
= /nd/ gets reduced according to a sonority principle
Reduction still productive in some places, no longer productive in other places
= /ng/ split in twain, matching productive /nd/ in some dialects, but non-productive /mb/ in most
Reduction no longer productive, generalised the same way everywhere
= /mb/ always reduced everywhere
Nasal assimilation
Compared to all that waffle above, this should be quite simple.
English has three nasal phonemes /n m ŋ/, and (at least) four nasal surface phones [n m ŋ ɱ], the latter being a labiodental nasal. In other words, there are three phones that have a direct phonemic correspondent, and one that does not.
There are two types of nasal assimilation to consider here:
- Full assimilation, which assimilates to any nasal phone.
- Partial assimilation, which assimilates only to non-phonemic phones but cannot assimilate to a different phoneme.
There is, as far as I know, no real reason why the distribution should be the way it is in English, but it is as follows:
- /n/ can undergo full assimilation: it can become [m ŋ ɱ]
- /m/ can undergo partial assimilation: it can only become [ɱ]
- /ŋ/ does not assimilate
Note that even in full assimilation, assimilation to a following nasal is much less common and mandatory than assimilation to a following stop or fricative. In practice—since /m/ can only undergo partial assimilation and /ŋ/ cannot occur syllable-initially—this means that /nm/ tends not to be assimilated to [mm] nearly as much as /np/ is assimilated to [mp], for example. Something like one more can be assimilated as [ˈwʌmˈmɔː] (possibly because of the stress distribution), but gunman is not likely to be assimilated as [ˈɡʌmːən].
Of course, in handbag, the reduction of /nd/ to /n/ has already taken place at a deeper level, and the highly productive and very near-surface process of nasal assimilation to a following stop occurs almost mandatorily in all contexts.
This means that the relevant phonemic input here is /hanbag/, which is then quite regularly (fully) assimilated to [hambaˑɡ].
