Are there specific rules / conventions at play when creating demonyms? Or are they merely formed organically over time - the most popular winning out?
There are many suffixes to choose from, but I cannot find concrete guidelines as to which is proper to use in which instance.
Generally, you have to go with what the locals call themselves. The names of foreign countries and their associated demonyms are often transliterated or reinterpreted versions of the original—or altogether fabricated because the original is unknown. As with any term in any language, the most popular wins.
These five suffixes seem by far the most common:
-(a)n (mostly for countries and continents) as in Mexican and African
-ian as in Bostonian and Oregonian
-ite (mostly for cities) as in Vancouverite
-er as in Londoner and New Englander
-ese as in Chinese and Japanese
Other suffixes listed in the Wikipedia article such as -ard as in Spaniard and -onian as in Newportonian are rarer and sound a little more old-fashioned. If you need to coin a new demonym, such as for a novel, then I would use one of the more common suffixes and leave it at that. People from Earth are often called Earthlings, but all of the other Solar planetary demonyms end in -an, including other names for Earthfolk such as Terrans, Tellurians, and in Futurama, Earthicans.
Speaking of -folk, it’s best left to things with a folksy sort of feel—Tolkien’s Shirefolk, for instance, parallels townsfolk, gentlefolk, menfolk, that sort of thing.
As you have observed, there are a lot of ways to make demonyms in English. While we can draw some generalizations for which kinds of place names (toponyms) get which kinds of demonyms, there are no hard and fast rules. The various suffixes come from different linguistic sources, though, and that has some bearing on which ones are used where. We can observe some trends:
The Latin-derived suffixes -an and -ian are by far the most common choice for English demonyms. One reason is that English names for non-English-speaking places are often Latin-derived: Italian, Croatian, Arabian, Indian, American (the name predates America's settlement by English speakers). -an is also common for Spanish and Portuguese-speaking places: Mexican, Paraguayan, Mozambican. The simple suffix has become the go-to for most newer demonyms, even if the place names are not Latin at all: Hawaiian, New Jerseyan, Chicagoan. Variations include Torontonian and Panamanian. For toponyms that end in O or U, if the suffix is true to its Latin roots it becomes -vian: Oslovian, Peruvian.
Some places have Latin-based demonyms even though the English toponym is quite different from its Latin equivalent: Guernsey → Sarnian, Newcastle → Novocastrian, Halifax → Haligonian. And of course Norway → Norwegian, which spawned the analogous demonyms Galloway → Galwegian and Glasgow → Glaswegian.
-ite comes from Greek. Few Greek demonyms have survived into modern English, but the suffix lives on in the public consciousness due to its use for nations and tribes in the Old Testament. It has become another common choice for recently coined demonyms: Manhattanite, Wisconsinite, Tokyoite, Delhite, Vancouverite.
Many places in Germany have identical demonyms in German and English using the suffix -er, such as Berliner and my favorites: Hamburger and Frankfurter. -er is Germanic in origin and is used with many toponyms of pure English origin: Londoner, New Yorker, Marylander, Aucklander; and of other Germanic languages: Amsterdammer, Stockholmer.
-ish is also Germanic in origin but is less versatile because it's a adjectival suffix; you don't refer to someone as an Irish, a Spanish, a Turkish, or a Swedish. Some places have one demonym for the people collectively, and another for an individual: Irishman, Spaniard, Turk, Swede.
Suffix-less demonyms like Turk, Swede, Dane, Czech, Pole, and Afghan derive from their local languages. In fact, these are not demonyms in the strict sense because they do not derive from place names; rather, they are the opposite phenomenon, wherein the place name derives from the name of an ethnic group or tribe (thanks, @phoog).
-ese is Romance in origin. It comes from Latin -ensis, which is commonly seen in the scientific names of species. It is common in demonyms for Romance-speaking places and places that became connected with Europe as a result of explorations by Romance speakers, particularly ones in East Asia and French-speaking places in Africa. These demonyms also are more often used as adjectives or collective nouns, rather than referring to individuals (Japanese art, the Japanese, not a Japanese). When the toponym ends in -o and isn't Romance in origin, the suffix becomes -lese (Congolese, Togolese).
Several places in the Middle East and South Asia use the suffix -i, which probably has sources in several unrelated languages: Israeli, Iraqi, Emirati, Azerbaijani, Nepali.
And of course there will always be outliers with their own unique linguistic history: French, German, Dutch, Swiss, Greek, Icelandic, Argentine, Cypriot, Montenegrin, Edinburgoynian.
To answer your question, no, there are not concrete guidelines about which suffixes to use for which places, except to look it up. But when in doubt, if you choose -an people will get your meaning.
Lots of examples from https://en.wikipedia.org/wiki/Demonym
