There's no lore for it
I could not find any lore that would allow one to keep a devil from returning to the hells upon death.
However, there are three spells that could be tried to achieve this. Magic circle is pretty solid as to rules as written, and the fiend trapped in the circle is also a well-established trope. The other two are mostly depending on DM buy-in. With that being the case, if you want this in the campaign, it probably would be best to negotiate a way with the DM of how it could be made to work with a special quest/ritual/item/spell, or, if you are the DM, come up with such.
The alternative obviously is that there is no way to stop a devil from reforming in hell.
Here are the details on the possible options with spells:
1. Inverted Magic Circle
Magic Circle says:
The creature can’t willingly enter the cylinder by nonmagical means. If the creature tries to use teleportation or interplanar travel to do so, it must first succeed on a Charisma saving throw. [...] When you cast this spell, you can elect to cause its magic to operate in the reverse direction, preventing a creature of the specified type from leaving the cylinder
The devil tries to leave the circle by interplanar travel, so it will get a Charisma saving throw. If it fails, the circle holds it back.
It is not spelled out if turning to smoke and ichor and instantly returning to its home layer are all one thing, or are a sequence of effects, so the DM must decide this. If it is all one thing, then the circle preventing planar travel should prevent the whole process. If it are separate steps, the devil would dissolve into smoke or ichor, but could not return, and would remain as a cloud of smoke or puddle of ichor in the circle, for as long as the circle remains active. Either way, it would prevent escape.
2. Disintegrate
Disintegrate says:
The target is disintegrated if this damage leaves it with 0 hit points.
A disintegrated creature and everything it is wearing and carrying, except magic items, are reduced to a pile of fine gray dust.
If you condsider being turned to dust as a transformation rather than dying (like when you get turned to stone), then depending on the timing of effects you could stop the devil from returning to hell, because it never dies. If you think being turned to dust is dying in dictionary sense of "stopping to live", the dust can turn to smoke or ichor and the devil will return to hell. Which of the two it is will be up to the DM, as the rules for dying never define what it exactly means.
3. Wish
Of course, no "can you" answer is complete without mentioning off-label use of wish with the DM approving such use. A cleric's divine intervention combined with DM approval might work, too.