It's not (yet) exactly what you're asking for, but here's an AnyDice program that returns the highest roll in a dice pool, excluding any dice rolled in another pool:
function: highest of DICE:d excluding EXCL:s {
NEW: {}
loop I over {1@DICE} {
if I = EXCL { I: 0 }
NEW: {NEW, I}
}
POOL: (#DICE) d NEW
result: 1@POOL
}
output [highest of 3d6 excluding 4d6]
It uses some pretty tricky AnyDice syntax, so let me explain a few of the details:
When you pass a die to an AnyDice function expecting a sequence (like EXCL:s above), AnyDice will run the function for every possible roll of the dice, and sum the results (weighted by the probability of each roll). This is basically a brute force approach, and it can easily time out for large dice pools, but it's very versatile when it works. This is how I'm handling the exclusion mechanic here.
#DICE gives the number of dice in the pool DICE, and {1@DICE} gives the possible values of a single die in the pool. That is, if DICE = 3d6, then #DICE = 3 and {1@DICE} = {1, 2, 3, 4, 5, 6}.
When you compare a number to a sequence (as in I = EXCL), the result will be true (non-zero) if the comparison is true for any element of the sequence. That is, if I = EXCL basically checks whether I belongs to the sequence EXCL.
So what the loop inside the function does is construct a custom die NEW which looks just like one of the dice in DICE, but with any sides matching EXCL replaced by 0. Then POOL: (#DICE) d NEW creates a new custom dice pool consisting of as many of those custom dice as there were in the original pool DICE, and the final line returns the highest result in that pool.
Getting a bit closer, here's a version that adds the bonus for multiple equal highest dice:
function: highest of ROLL:s with bonus {
MAX: 1@ROLL
BONUS: MAX > 0 & (MAX = ROLL) > 1
result: 10 * MAX + BONUS
}
function: highest of DICE:d excluding EXCL:s {
NEW: {}
loop I over {1@DICE} {
if I = EXCL { I: 0 }
NEW: {NEW, I}
}
POOL: (#DICE) d NEW
result: [highest of POOL with bonus]
}
output [highest of 3d6 excluding 4d6]
Since AnyDice doesn't support fractional numbers, I'm multiplying the result of the roll by 10, so that, say, a result of 5.1 (= 5 + 0.1 bonus) is represented as 51.
The code above is exactly the same as the first program above, except that I've replace 1@POOL with a call to a helper function that adds the bonus if the highest roll occurs multiple times (and is not zero). The helper function uses the same trick of taking the roll in as a sequence with :s, so that it gets automatically iterated over every possible outcome of the roll.
Finally, here's a program that calculates the number of non-excluded dice in pool B:
function: number of DICE:d excluding EXCL:s {
NEW: {}
loop I over {1@DICE} {
NEW: {NEW, !(I = EXCL)}
}
result: (#DICE) d NEW
}
output [number of 2d6 excluding 4d6]
It's the same as the first program, except that now we're labeling the sides of our custom dice with 0 if they're in the exclusion list, and 1 otherwise, and then taking the sum of the roll as the result.
The remaining problem is to combine these programs, to account for the fact that both pools A and B depend on pool C simultaneously. The problem is that AnyDice doesn't really handle two-dimensional output well, but we can use the same trick as we used to handle the 0.1 bonus above, multiplying the result of pool A further by 10 before adding the result of pool B, like this:
function: highest of ROLL:s with bonus {
MAX: 1@ROLL
BONUS: MAX > 0 & (MAX = ROLL) > 1
result: 10 * MAX + BONUS
}
function: highest of A:d and count of B:d excluding EXCL:s {
ANEW: {}
loop I over {1@A} {
if I = EXCL { I: 0 }
ANEW: {ANEW, I}
}
BNEW: {}
loop I over {1@B} {
BNEW: {BNEW, !(I = EXCL)}
}
result: 10 * [highest of (#A) d ANEW with bonus] + (#B) d BNEW
}
output [highest of 3d6 and count of 2d6 excluding 4d6] named "A.A, B"
Here, the output consists of a three-digit number PQR, where P is the highest non-excluded roll in pool A, Q is 1 if the highest non-excluded roll in pool A is multiple, and R is the number of non-excluded rolls in pool B.
Oh, wait! You said you wanted the highest roll in pools A and B together, not just the highest roll in A? Well, I can do that, too, although it does require a little bit of reorganization, since the highest roll is no longer independent of the number of non-excluded rolls in pool B:
function: highest of A:s and B:s with bonus and count {
MAX: [highest of 1@A and 1@B]
BONUS: MAX > 0 & (MAX = A) + (MAX = B) > 1
COUNT: B > 0
result: 100 * MAX + 10 * BONUS + COUNT
}
function: SEQ:s excluding EXCL:s {
NEW: {}
loop I over SEQ {
if I = EXCL { I: 0 }
NEW: {NEW, I}
}
result: NEW
}
function: highest of A:d and B:d with bonus and count excluding EXCL:s {
AX: (#A) d [{1@A} excluding EXCL]
BX: (#B) d [{1@B} excluding EXCL]
result: [highest of AX and BX with bonus and count]
}
output [highest of 3d6 and 2d6 with bonus and count excluding 4d6]
