I would also like to test training on simple files that are
specifically made to represent rules in music theory (basic chords,
intervals, etc.)
Chords and intervals aren't really musical 'rules', they're (a way of describing) structures that can exist independently of any rules for their generation or analysis. Of course any particular terminology may have its own rules, but that is conceptually different to musical rules governing 'what sounds are actually played', which sounds like ultimately what you care about.
Anyway, if you want to make some simple midi files consisting of (say) just one chord for test purposes, those should be trivial to generate if you can't find them online.
Alternatively, are there any simple songs that exemplify theory more
than others?
At the level of fairly simple rules, you will find that each song exemplifies rules relating to that particular style of music. So if you feed in some Latin jazz pieces, you'll find those tend to follow a certain set of rules, while a set of Baroque organ pieces will have a different set of tendencies (i.e. some of their 'rules' would be different). What you'd probably want to do would be to feed the system with the kind of music that you want it to produce.
Of course it's possible that there are deeper rules at a greater level of abstraction that hold for a wide range of genres. These would probably be discoverable by feeding in a range of different styles of works. How a neural net would cope with that would be interesting - would it end up 'seeing' music as divided into genres, with groups of rules that can be used together?
It might be an interesting experiment to train your net on simpler pieces (nursery rhymes?) before giving it more complex pieces, and compare that approach with throwing more complex stuff at it from the start.