Keep in mind that "register" is a piece of linguistics jargon, so the correct translation depends on who you are writing or talking to.
You can use レジスター when you know the reader of your text is a linguist.
If your target audience is ordinary people who are interested in languages (like most people on this site), it's safer to introduce the term 言語使用域 (or 使用域 for short) first, which is a common and self-explanatory translation for this.
If you're talking to the general public, it's usually better to avoid using such a difficult concept in the first place. For example, you almost never encounter this term in language education programs on TV. I have posted thousands of answers on this site, many of which are related to literary register (example), but I generally don't use the term register or レジスター because many people simply don't understand it.
レジストル seems to be the French pronunciation of "register," but I don't think it's used outside contexts of French.
If you still think レジストル is the right term for you after reading this, yes, 文芸レジストル (or 文学レジストル) should be a correct choice, although it's likely to be understood only by French experts. Otherwise, you will need to come up with a correct expression on a case-by-case basis, taking into account the knowledge level of your readers. For example, if I really wanted to say "literary register" on a site like this without further elaboration, I might use 文学の言語使用域.
By the way, I found several articles that mention "the six (or five) registers of the French language", but I haven't seen such prescribed classifications with respect to Japanese.