I found the cause of this behavior to be indirectly documented in man keyboard:
DESCRIPTION
The keyboard file describes the properties of the keyboard. It is read by setupcon(1) in order to configure the keyboard on the console. In Debian systems the default keyboard layout is described in /etc/default/keyboard and it is shared between X and the console.
The contents of /etc/default/keyboard are generated on system install and gave a hint to the actual problem (I added the ctrl:nocaps option later):
XKBMODEL="pc105"
XKBLAYOUT="se"
XKBVARIANT=""
XKBOPTIONS="ctrl:nocaps"
The above together indicates that X will set the xkb options at some point during startup (probably after .xsession and the like) which causes any xmodmap settings set during .xsession to be lost.
So there is the cause, the solution is hinted at from man setupcon:
The keyboard configuration is specified in ~/.keyboard or /etc/default/keyboard. The font configuration is specified in ~/.console-setup or /etc/default/console-setup.
Checking man console-setup yields:
The file console-setup specifies the encoding and the font to be used by setupcon(1) in order to setup the console. It can be used also to specify the keyboard layout but it is not recommended to do so, use keyboard(5) instead.
So /etc/default/keyboard is used to set keyboard settings for TTY and X. While /etc/default/console-setup is can be used to setup keyboard (not really recommended but works) and font for console only.
So to make this all work I moved /etc/default/keyboard to /etc/default/console-setup and added the following to my .xsession:
#!/bin/bash
# The below assumes bash features, rewrite if you use other shells.
source /etc/default/console-setup
XKBPARMS=""
if [[ "$XKBLAYOUT" ]]; then
XKBPARMS="-layout $XKBLAYOUT"
fi
if [[ "$XKBMODEL" ]]; then
XKBPARMS+=" -model $XKBMODEL"
fi
if [[ "$XKBVARIANT" ]]; then
XKBPARMS+=" -variant $XKBVARIANT"
fi
if [[ "$XKBOPTIONS" ]]; then
XKBPARMS+=" -option $XKBOPTIONS"
fi
if [[ "$XKBPARMS" ]]; then
setxkbmap $XKBPARMS
fi
xmodmap ~/.Xmodmap
Now xmodmap works correctly and I have the correct keymap and options in both the TTY and in X.
xkbbut it seemed way too complicated for the equivalentxmodmap. Specifically I remap rightCtrlto becomeMod3instead of functioning as a control modifier. This becomes my mod key for i3 which is easier to use on my hands than theWinwhich breaks my fingers andAltconflicts with just about every other keybinding I use. But thanks for the comment, I'm sure it will be useful to someone who doesn't have a weird key rebinding schema. :) – Emily L. Nov 12 '17 at 11:14