This program should do it.
function: attack ROLL:n plus BONUS:n vs AC:n hit HIT:d miss MISS:d crit CRIT:d on CRIT_RANGE:n {
if ROLL = 1 { result: MISS }
if ROLL >= CRIT_RANGE { result: CRIT }
if ROLL + BONUS >= AC { result: HIT }
result: MISS
}
function: attack ROLL:n plus BONUS:n vs AC:n for DMG:d crit CRIT_DMG:d on CRIT_RANGE:n {
result: [attack ROLL plus BONUS vs AC hit DMG miss 0 crit DMG + CRIT_DMG on CRIT_RANGE]
}
function: plussneakattack BASE_DAMAGE:n plus SA:d {
if BASE_DAMAGE = 0 { result: 0 }
result: BASE_DAMAGE + SA
}
AC: 17
MYCRIT: 20
ADVANTAGE: [highest 1 of 2d20]
ABONUS: 2 + 3
SNEAK_ATTACK: 1d6
DBONUS: 3
DAMAGE: 1d8
HITDMG: DAMAGE + DBONUS
CRITDMG: DAMAGE + SNEAK_ATTACK
RAPIER_ATTACK: [attack ADVANTAGE plus ABONUS vs AC for HITDMG crit CRITDMG on MYCRIT]
RAPIER_SNEAK: [plussneakattack RAPIER_ATTACK plus SNEAK_ATTACK]
DAMAGE: 1d6
HITDMG: DAMAGE + DBONUS
CRITDMG: DAMAGE + SNEAK_ATTACK
HANDCROSSBOW_ATTACK: [attack ADVANTAGE plus ABONUS vs AC for HITDMG crit CRITDMG on MYCRIT]
HANDCROSSBOW_SNEAK: [plussneakattack HANDCROSSBOW_ATTACK plus SNEAK_ATTACK]
TOTAL_ATTACK: HANDCROSSBOW_ATTACK + RAPIER_ATTACK
TOTAL_SNEAK: [plussneakattack TOTAL_ATTACK plus SNEAK_ATTACK]
output RAPIER_ATTACK named "Rapier"
output RAPIER_SNEAK named "Rapier with Sneak"
output HANDCROSSBOW_ATTACK named "HandCrossbow"
output HANDCROSSBOW_SNEAK named "HandCrossbow with Sneak"
output TOTAL_ATTACK named "Full Attack Without Sneak"
output TOTAL_SNEAK named "Full Attack with Sneak"
The plussneakattack function takes advantage of the fact that a miss will do 0 damage and a hit will do non-zero damage: if the input damage is 0, it returns 0 back out; if the input damage is non-zero, it adds the sneak attack damaged passed in.
The one caveat as-written is that critting on both the rapier and the crossbow will apply sneak attack a total of three times, since CRITDMG includes sneak attack damage. My understanding is that 5e doesn't work that way, but it'd be easy enough to change CRITDMG if needed.
Any number of attacks (to AnyDice's limits, of course) can be added to TOTAL_ATTACK, then add the plussneakattack call just once at the end to get the "with sneak damage exactly once (plus crits)" for the whole round.
The above works for the question as-asked, which assumes that crits always add sneak attack damage. I don't believe that's correct for 5e RAW, but it may be at some tables.
I believe that this program calculates sneak attack more correctly by pulling it out into a virtual additional attack when determining the round's damage. Again, though, any number of attack can be added together:
function: attack ROLL:n plus BONUS:n vs AC:n hit HIT:d miss MISS:d crit CRIT:d on CRIT_RANGE:n {
if ROLL = 1 { result: MISS }
if ROLL >= CRIT_RANGE { result: CRIT }
if ROLL + BONUS >= AC { result: HIT }
result: MISS
}
function: attack ROLL:n plus BONUS:n vs AC:n for DMG:d crit CRIT_DMG:d on CRIT_RANGE:n {
result: [attack ROLL plus BONUS vs AC hit DMG miss 0 crit DMG + CRIT_DMG on CRIT_RANGE]
}
AC: 17
MYCRIT: 20
ADVANTAGE: [highest 1 of 2d20]
ABONUS: 2 + 3
SNEAK_ATTACK: 1d6
DBONUS: 3
DAMAGE: 1d8
HITDMG: DAMAGE + DBONUS
CRITDMG: DAMAGE
RAPIER_ATTACK: [attack ADVANTAGE plus ABONUS vs AC for HITDMG crit CRITDMG on MYCRIT]
DAMAGE: 1d8 + SNEAK_ATTACK
HITDMG: DAMAGE + DBONUS
CRITDMG: DAMAGE
RAPIER_SNEAK: [attack ADVANTAGE plus ABONUS vs AC for HITDMG crit CRITDMG on MYCRIT]
DAMAGE: 1d6
HITDMG: DAMAGE + DBONUS
CRITDMG: DAMAGE
HANDCROSSBOW_ATTACK: [attack ADVANTAGE plus ABONUS vs AC for HITDMG crit CRITDMG on MYCRIT]
DAMAGE: 1d6 + SNEAK_ATTACK
HITDMG: DAMAGE + DBONUS
CRITDMG: DAMAGE
HANDCROSSBOW_SNEAK: [attack ADVANTAGE plus ABONUS vs AC for HITDMG crit CRITDMG on MYCRIT]
DAMAGE: SNEAK_ATTACK
HITDMG: DAMAGE
CRITDMG: DAMAGE
ROUND_SNEAK: [attack ADVANTAGE plus ABONUS vs AC for HITDMG crit CRITDMG on MYCRIT]
TOTAL_ATTACK: HANDCROSSBOW_ATTACK + RAPIER_ATTACK
TOTAL_SNEAK: TOTAL_ATTACK + ROUND_SNEAK
output RAPIER_ATTACK named "Rapier"
output RAPIER_SNEAK named "Rapier with Sneak"
output HANDCROSSBOW_ATTACK named "HandCrossbow"
output HANDCROSSBOW_SNEAK named "HandCrossbow with Sneak"
output TOTAL_ATTACK named "Full Attack Without Sneak"
output ROUND_SNEAK named "Round's Sneak Attack"
output TOTAL_SNEAK named "Full Attack with Sneak"