Is there any rule for determining whether a verb beginning with ε- will augment to η- vs ει-, or must all verbs' behaviors be memorized?

Question

For instance, the verb ἐλευθερῶ augments to ἠλευθέρουν in the past, whereas the verb ἔχω augments to εἶχον (not ἦχον as might have been predicted).

Answer 1

You'll basically have to memorise them, yes, though there are patterns. Both the η- in ἠλευθέρουν and the ει- in εἶχον represent a contraction of ε + ε, but the former is much older than the latter.

The ε + ε > η contraction dates to a time (prehistoric, as far as Attic is concerned) when η was still meaningfully the long version of ε, which is also when most verb forms were established and therefore the most common outcome of an augmented verb starting with ε-.
At some point before the Classical period (in Attic), η was lowered to [ɛː] and ε was raised to [e] and that connection was severed, with a new long vowel ει [eː] now serving as the long version of ε (a similar thing happened for ο/ω/ου). Any contraction of ε + ε that happened after this happened will yield ει instead of η. You'll probably see this in verbs coined late (no examples come to mind just now), but it's also very common in verbs that used to start with a digamma (ϝ [w]) or a σ followed by a vowel, both of which also disappeared in Attic relatively late:

• ἕπομαι 'follow' → εἱπόμην (from PIE *sekʷ-)
• ἔχω 'have' → εἶχον (from PIE *seǵʰ-)
• ἑστιάω 'host' → εἱστίων (earlier ϝεστια-)
• ἐθίζω 'grow accustomed' → εἴθιζον (earlier ϝετιδ-)
• cf. also ὠθέω 'push' → ἐώθουν rather than ˣὤθουν (earlier ϝωθ-)

A disappearing initial ϝ or (especially) σ will often leave a rough breathing, but not, as you can see, all the time. Still, if you do see a verb start with rough breathing, that's a good clue.

Sometimes you'll also see these "new" contractions show up by analogy in verbs where they shouldn't:

• ἐάω 'permit' → εἴων for no etymological reason

Or conversely, the old contractions in verbs where you'd expect new ones:

• ἐργάζομαι 'work' → ἠργαζόμην (earlier ϝεργαδ-); expected εἰργαζόμην also attested

There are other irregularities involving the augment as well. Some verbs get a double one:

• ἁλίσκομαι 'be seized' → aor. ἑᾱ́λων (next to expected impf. ἡλισκόμην)
• ὁράω 'see' → ἑώρων
• βούλομαι 'want' → ἠβουλόμην (probably analogy with (ἐ)θέλω → ἤθελον)
• μέλλω 'be about to' → ἤμελλον

And, because it's Attic, sometimes you get quantitative metathesis:

• ἑορτάζω 'celebrate' → ἑώρταζον (< *ἡόρταζον/εἱόρταζον)

So the short of it is that yes, you'll have to study your principal parts.