There are no rules, it is a matter of whether there exist sounds that are likely to be transcribed by such combinations of kana + small kana.
A relevant remark in a Wikipedia article is
主に外来語に使われる「イェ」「ウィ」「ファ」「ティ」等は、拗音と同じく2文字で1モーラの音を表す。これらのモーラが日本語の音韻体系で占める位置づけについては諸説あるが、点字(日本点字表記法)ではこれらを拗音として扱う。
So those mentioned here are common enough, but a combination like アゥ is hard to imagine as one-mora sound.
For the particular ones you mention, first note that ェ is more /je/, not fully a vowel. So in theory, イェ/キェ/シェ/チェ/ニェ/ヒェ/ミェ/リェ are palatalized エ/ケ/セ/テ/ネ/ヘ/メ/レ, but as real world examples, you mostly don't see キェ/ニェ/ヒェ/ミェ/リェ.
- イェ: Yale university is commonly イェ―ル大学 (/je/)
- シェ: shell is シェル (/ʃe/)
- チェ: Czech is チェコ (/tʃe/)
The IPA may not be completely accurate, but it should give some idea for the sounds. For the others, they may well be used in transcribing words in Slavic languages. (Russian Нет sounds ニェット.)