5

I'm trying to use Vim as a hex editor.

I am using the %!xxd / %!xxd -r method, and it worked fine until I tried a some bytes that didn't work as expected :

00000000: 1337 6161 6161 6161 6161 6161 6161 6161  .7aaaaaaaaaaaaaa
00000010: 6161 6161 6161 6161 6161 6161 6161 62ff  aaaaaaaaaaaaaabR
00000020: 3713 0000 0a                             7....

With this in the Vim buffer, running %!xxd -r and then %!xxd will produce :

00000000: 1337 6161 6161 6161 6161 6161 6161 6161  .7aaaaaaaaaaaaaa
00000010: 6161 6161 6161 6161 6161 6161 6161 62c3  aaaaaaaaaaaaaab.
00000020: bf37 1300 000a                           .7....

Notice the difference at the end of the second line.

Now you might think that xxd is not used correctly, however running this : cat bytes | xxd -r | xxd will yield :

00000000: 1337 6161 6161 6161 6161 6161 6161 6161  .7aaaaaaaaaaaaaa
00000010: 6161 6161 6161 6161 6161 6161 6161 62ff  aaaaaaaaaaaaaab.
00000020: 3713 0000 0a                             7....

which is what I was expecting to see in Vim.

Do you have any idea of what I am doing wrong?

D. Ben Knoble
  • 26,070
  • 3
  • 29
  • 65
aguilbau
  • 53
  • 1
  • 5
  • Combinaison = combination? (Et bienvenue à [vi.se]!) – D. Ben Knoble Apr 04 '20 at 14:14
  • merci :D I'll change that, good catch. – aguilbau Apr 04 '20 at 14:20
  • Interesting! It's turning \u00ff into \xc3\xbf, which is exactly the UTF-8 encoding of that Unicode character. So encoding is the source of the problem here. The solution might be using vim -b to enable binary mode (see :help 'binary'), but not sure if that will break anything on the xxd view (since that's textual and not really binary...) – filbranden Apr 04 '20 at 15:22

2 Answers2

3

This will happen if you start editing the xxd-dumped version of the binary file, in which case Vim will assume a UTF-8 encoding (which you can confirm with :set fenc? returning fileencoding=utf-8).

When you turn the file back into binary form, the character still looks correct (using ga on top of it shows <ÿ> 255, Hex 00ff, Oct 377, Digr y:), but that's actuall not considering the encoding, so if you were to save this file now, the UTF-8 encoding will kick in and will turn \u00ff into the bytes \xc3\xbf, which are the UTF-8 encoded version of this Unicode character.

One possible workaround is to use a different file encoding such as latin1, which is a 8-bit encoding and shouldn't expand characters above 127.

A better solution is to use vim -b to enable the 'binary' option. (You can also use :set binary from inside Vim.) Enabling 'binary' will disable a file encoding completely (:set fenc? returns fileencoding=) and should preserve 8-bit characters as they are.

It turns out this works OK with the xxd-dumped output since xxd only uses 7-bit ASCII characters in its output, so using 'binary' doesn't really hurt it too much.

(From a purist's point of view, an ideal solution here would use two separate buffes in Vim, one for the xxd-dumped output and a separate one with 'binary' set for the binary encoded output of xxd. A plug-in facilitating binary editing would probably help manage the separate buffers and set the right options on all of them.)

filbranden
  • 28,785
  • 3
  • 26
  • 71
1

Here's what I have in my ~/.vimrc to allow quite comfortable and issue-free editing of binary files, which are narrowed down to *.bin and *.hex files in the vimcode below, but that can be adjusted as needed, of course:

" Use xxd(1) to make editing binary files actually usable, by filtering
" the entire buffer through xxd(1) as an external command, and save/restore
" the view position while saving the buffer, to stay at the same position
"
" NOTE: Thanks to the file's dual nature, restoring the last saved position
"       of the cursor upon opening a file this way is effectively broken
"
augroup vimrc-xxd-binary-edit
  autocmd!
  autocmd BufReadPre  *.bin,*.hex set binary
  autocmd BufReadPost *.bin,*.hex
    \ if &binary
    \ |   execute "silent %!xxd -autoskip -cols 32"
    \ |   set filetype=xxd
    \ |   redraw
    \ | endif
  autocmd BufWritePre *.bin,*.hex
    \ if &binary
    \ |   let s:winview = winsaveview()
    \ |   execute "silent %!xxd -revert -autoskip -cols 32"
    \ | endif
  autocmd BufWritePost *.bin,*.hex
    \ if &binary
    \ |   execute "silent %!xxd -autoskip -cols 32"
    \ |   set nomodified
    \ |   call winrestview(s:winview)
    \ |   redraw
    \ | endif
augroup END

As noted in the code comments above, you may also want to disable restoring of the cursor position upon opening a file when &filetype ==# "xxd". See also :h restore-cursor for the usual way the cursor position is restored.

See also my patches that contain both the vimcode visible above and more related vimcode, which have been accepted upstream by the vim project in commits 6a500661a9 and 81b8bf5b4a.

dsimic
  • 217
  • 1
  • 7
  • Although that link is to another answer on this site, it's still good practice to put enough detail in this answer that it stands alone (and helps avoid link rot). – D. Ben Knoble Aug 11 '23 at 15:31
  • Wouldn't it be then seen as copying and pasting the same aswer multiple times? I already had a few answers that were the same in different questions, for the sake of completeness, and a couple of them were deleted by the admins. I even received a warning that I might be banned because of my misbehavior. Knowing that, I hope you understand my concerns and confusion. – dsimic Aug 11 '23 at 21:30
  • 1
    Your concern makes sense, though I'm not aware of any of your answers being deleted that way on this site. If you find yourself putting the same answers in many places, I suggest that either the questions are duplicates or that the answers need adjustment for differing context of the question. Regardless: the answer as it stands is essentially a comment or incomplete. Including enough detail in this answer that it stands on its own would solve that problem. – D. Ben Knoble Aug 21 '23 at 12:51
  • My answers were deleted in these two questions: https://stackoverflow.com/questions/25447324/how-to-use-cut-with-multiple-character-delimiter-in-unix and https://stackoverflow.com/questions/7142735/how-to-specify-more-spaces-for-the-delimiter-using-cut . Those questions are duplicates of another question, but how can I flag them as duplicates when I have very little points? I tried editing those questions to include a note about them being duplicates, so someone can pick that up and actually flag the questions, but those pending edits seem to have been deleted silently. – dsimic Aug 21 '23 at 13:44
  • I'll see to rewrite this answer a bit later today so it becomes a valid answer on its own. By the way, I appreciate your suggestions! – dsimic Aug 21 '23 at 13:47
  • 1
    Looking better now? :) – dsimic Aug 21 '23 at 16:00
  • Ah, [so] is its own beast :) I'm not sure what the best thing to do is there. Looks great! – D. Ben Knoble Aug 21 '23 at 16:32
  • I'm glad you like it. :) Based on my limited experience so far, Stack Overflow seems to operate in its own way, so to speak. – dsimic Aug 21 '23 at 17:41
  • @dsimic if you disagree with anything that happens on any SE site, you can raise the topic on meta (as any mod would tell you, right?). No point doing so here in the comments. Trying to do things you don't have privileges to tends to infuriate the more senior users. Just don't. If those dupes have been there for 12 years they will be around until you'll have enough rep... – Friedrich Aug 21 '23 at 21:24
  • I see, thank you very much for your guidance. I'm still new here and I'm just still learning how the things work here, simply because all communites work in a slightly different way. – dsimic Aug 22 '23 at 02:22