TLDR
The W in 'two' and 'sword' is silent because of a sound change that took place somewhere between Old English & Middle English. The change applied to words in which the W was preceded by [s, t] and followed by a back vowel like [ɔ o ɑ u] etc.
'Swore' and 'sworn' also lost their W's at one point, but were later on restored by analogy with swear.
The silent W in 'sword' and 'two'
Between Old English and Middle English, there was a sound change through which a /w/ was lost in the environment of a preceding [s] or [t] (and sometimes [h]) and a following back vowel ([ɒ ɔ o ɑ u] etc). So 'two' was twā and its pronunciation was /twɑː/ in Early Middle English, it became /twoː/ and then it lost the /w/ and became /toː/ (the vowel /oː/ was shifted to /uː/ because of the Great Vowel Shift).
'Sword' was pronounced /sword/, it became /sord/ (or /sɔrd/). The conjunction/adverb 'so' used to be pronounced with a /w/, but it lost its /w/ due to the same reason. The change also affected 'who' as it was hwa before.
However, the change didn't occur when the /w/ was followed by a front vowel; for instance, 'twin' and 'swift' didn't lose their /w/'s because the w's in those words were followed by a front vowel /ɪ ~ i/.
Now 'swore' and 'swollen' deviate from this rule. They do have back vowels after the /w/ and there is an /s/ before the /w/, yet the /w/ is preserved. That's because of Analogical change.
Analogical change
Before explaining 'swore', I'd like to elucidate 'analogy' (or analogical change). It's ‘a type of language change in which some forms are deliberately changed merely to make them look more like other forms’ [Trask]. Larry Trask, a phenomenal linguist, gives a remarkable insight into 'analogy' in his book Trask's Historical Linguistics. I'll just explain it the way Trask has explained with another example.
Suppose I tell you that frumicate is a rare and obscure English verb meaning to put on airs. I'm fairly certain you've never heard it before (because you haven't, have you?). What do you suppose its past tense is? You've never heard its past tense before, but I'm pretty sure you'll say frumicated (agree?). How do you know its past tense is frumicated?
‘You do it by invoking analogy’—that is, you assume that the required past tense is formed according to a very common pattern that is already familiar to you from large numbers of other English verbs. In this case, the pattern for forming past tense is so widespread and regular that it actually constitutes a rule of English grammar for regular verbs. [Adapted from Trask]
Trask gives another example:
The past tense of 'catch' was formerly catched, but as a result of this analogy, it has become caught (which is now the standard past tense of 'catch').
The /w/ in 'swore' and analogy with 'swear'
The same change (loss of /w/) should've happened with 'swore' and 'swollen', but these are nonetheless pronounced with a /w/ today.
It's possible though that the analogy of these forms—which always retained their /w/s because they had front vowels—prevented the regular sound change from affecting 'swore' and 'swollen'.
Or they did actually lose their /w/s at one point, but were later restored by the analogy with swear and swell. [see Analogy and levelling — Trask's Historical Linguistics pp101-102]
Wikipedia says the same: /w/ in swore is due to analogy with swear. Another Wikipedia article says that ‘[a]n example of analogical maintenance would be the perseverance of /w/ in swollen by analogy with the present tense swell (contrast with sword, where the /w/ is lost by regular sound change)’.
The change didn't apply to swear and swell because the /w/ in those words was followed by a front vowel and the sound change didn't affect words in which the /w/ was followed by front vowels.
