Note that the error says local:5037, not tcp:5037. ADB is actually trying to connect to an unix socket at /tmp/5037, which fails.
In my case this was because I ran adb as root before. Thus, the socket was owned by root, forbidding my user from removing it. Relevant excerpt from strace adb fork-server server:
socket(PF_LOCAL, SOCK_STREAM, 0) = 8
unlink("/tmp/5037") = -1 EPERM (Operation not permitted)
setsockopt(8, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
bind(8, {sa_family=AF_LOCAL, sun_path="/tmp/5037"}, 12) = -1 EADDRINUSE (Address already in use)
close(8) = 0