What you might be looking for is the empty die trick for rejecting "impossible" outcomes (i.e. results that you'll always reroll until they don't happen).
For example, here's how to model a mechanic where you roll 3d12 and reroll them all if the result is over 12:
function: test ROLL:s if total is MAX:n or less {
SUM: 0 + ROLL
if SUM > MAX { result: d{} } \ ignore rolls that sum to more than MAX! \
\ now do something with ROLL (and/or SUM) here... \
result: SUM
}
output [test 3d12 if total is 12 or less]
The magic happens on the line if SUM > MAX { result: d{} }; if this condition is true, the function stops and returns the "empty die" d{}, which AnyDice will completely ignore when collecting the results. The end result is exactly as if you were to reroll any rolls that match the condition until they no longer do.
In the rest of the function, you can then calculate whatever result you want based on the input ROLL and return it. Note that the return value needs to be a number (or a die); if we tried to return a sequence (such as ROLL itself), AnyDice would just automatically sum it. In the example program above I just return the sum, but one possible alternative (if you wanted to examine the values of the individual dice) would be to encode the sequence ROLL as a base-10 or base-100 number, e.g. like this:
result: 10000 * 1@ROLL + 100 * 2@ROLL + 3@ROLL
(Here's a more generic helper function to do this if you want one.)
I used an normal d12 for the example above, but obviously you could also use a custom biased die if you wanted. And you could also implement something like your "roll one die at a time and reroll last if over 12" mechanic, it would just be more complicated and/or tedious.
Basically, you'd need to have a series of function calling each other or a single function calling itself recursively to model the step-by-step rolling, something like this:
function: test BASE:n plus ROLL:n plus N:n times DIE:d max MAX:n {
if BASE + ROLL + N > MAX { result: d{} }
if N = 0 { result: BASE + ROLL }
result: [test BASE + ROLL plus DIE plus N-1 times DIE max MAX]
}
output [test 0 plus d12 plus 2 times d12 max 12]
Note that, in this example, I'm rerolling the most recent die if the roll plus the base total so far plus the number of remaining dice to roll exceeds the maximum. That's because we know that the remaining dice will always roll at least 1 each anyway. So, for example, if we rolled an 11 on the first d12 out of three when the maximum was 12, we'd reroll it since we'd know that the following two rolls would have to each increase the total by at least 1, making it at least 11 + 1 + 1 = 13.
