I am not a lawyer, and I’m certainly not your lawyer, who has reviewed all the facts of the case and the relevant case law and so on and so forth. This is meant for educational purposes only.
Without licensing anything
Game rules and mechanics and so on are not covered by any of the various intellectual property protections available in most countries. (Most countries in the world have signed the Berne Convention, which means broadly speaking copyright works more-or-less the same way in each.) (It’s theoretically possible to get a patent on a game mechanic, but I’m not aware of any that have ever been sought, much less granted, for an RPG. I suspect it would be a tough sell.)
However, while the mechanics and rules themselves aren’t protected, the text (and images and tables and so on) explaining them is covered by copyright, and owners can also claim various things as trademark (and Wizards of the Coast certainly does for “Dungeons & Dragons”). What this means is that, in theory, you can expand upon those rules with your own material, but you have to be very careful you aren’t copying any text, images, logos, and so on, nor deriving your text etc. from those belonging to Wizards (i.e. taking their text and changing words doesn’t save you). Furthermore, your product must be very clearly not be confuse-able with an official D&D product. This has been done—but by someone whose day job was literally as an intellectual property lawyer who had successfully sued Wizards of the Coast in the past. If you aren’t, or don’t have such a person on the payroll, you could get yourself in a lot of trouble.
In practice, even if you are very certain that you are on the right side of the law here, there’s always the risk of being unable to defend yourself because of the likely-vast discrepancy in resources you have available vs. those available to the Wizards of the Coast legal department. It is an unfortunate reality that those with money can often deplete the defense funds of those who do not even if they are wrong, because people can’t afford to prove it. That very much applies here.
Under the Open Game License
The solution to all the problems above is to license the appropriate material—a license is literally something that says you are allowed to use material that isn’t yours and you couldn’t otherwise use. Using material under a license is generally much, much safer than going it alone, because the specific rules you have to follow are explicitly laid out in the license.
And in the case of D&D, much of the material you would like to use is already available under license—the Open Game License. Conveniently, as the name suggests, this license is fairly “open”—it lets you do quite a lot of things. Write new classes, subclasses, items, feats, and so on? Absolutely—tons of OGL products do those things. That’s the point.
In fact, you likely can even go well beyond that—Pathfinder’s first edition was literally built off of the parts of the “v.3.5 revised edition” of D&D that were released under the OGL. (Strictly speaking, no part of 3.5e was ever released under the OGL—rather, the “d20 System” was released under the OGL. But the “d20 System” was literally just a copy-and-paste of most of D&D 3.5e’s core rules.) Paizo tweaked a great many things about those rules—how and when you got feats, how skills worked, how combat maneuvers were rolled, what various classes and spells actually did, and so on. It’s clearly based on the same content, but also a lot of details have been changed—Pathfinder absolutely would have been a violation of copyright to publish without the OGL.
And Pathfinder, for a time, was actually eating D&D’s lunch. Paizo is a huge competitor for Wizards of the Coast. There is no way they would have allowed Paizo to do what they did if they had any rights to stop it.
There are a few caveats, however.
You have to read and implement the OGL carefully.
You can only use those things that are covered by the OGL. A lot of 5e isn’t—and anything not covered is going to be extremely dangerous to even mention.
You have to avoid any Product Identity at all costs—the OGL requires you to not reference Product Identity even in those ways that you could have without a license. This includes settings like the Forgotten Realms, or any of the people, places, gods, and so on found within them.
Your own rules and mechanics will most likely need to be declared open game content as well. That is, if you use Wizards’ work, you are agreeing to allow anyone else to use yours in the same way. You can make characters, places, settings, names, etc. etc. Product Identity, but you’ll have a hard time doing so with feats, classes, and so on. (If you really want to protect something, I advise discussing it with a lawyer, because I am not 100% sure on this point.)
Double-check the copy of the OGL that is attached to the material you want to use. There have been a few different versions of the licenses, and while as far as I know, nothing crucial has changed between them, but you will want to be sure. I don’t think they have, but after Pathfinder it’s plausible that they may have tightened up the license to prevent something like that from happening again with 5e. That may affect your goals.
About the DM’s Guild
This is somewhat tangential to your question, and I don’t know much in the way of details about the license for the DM’s Guild, but my understanding is you likely want nothing to do with it. The license is more restrictive and likely would run afoul of your goals here. I only mention it because it might otherwise be a useful marketplace for your product, and because DM’s Guild material can use a selected, limited amount of Product Identity which could conceivably be tempting to you. I don’t believe it will work out for you, however.
About monetization
Contrary to popular belief, whether or not your product is free has very, very little to do with whether or not it counts as copyright infringement. (Being free can help a claim of fair use, but it would have to be just one part of a larger defense—fair use definitely does not apply to what you describe, whether it’s free or not.) So it doesn’t really matter if you are trying to sell this product or not—you still cannot use unlicensed copyrighted materials, either way.
The OGL does not prohibit you from selling materials made based on open game content. It’s possible to write licenses that do prohibit commercial use—and then it would very much matter whether you were trying to sell your product—but the OGL does not. So it also doesn’t matter with respect to the OGL whether or not your product costs money.
