2

I have a centos server that recently needed a fsck repair on it's main hard drive. After fixing a bunch of inode issues (booting from a system rescue disk), the box booted clean. Most of the services are running fine, however the sshd service immediately core dumps when I try to start it.

Running gdb /usr/sbin/sshd core.xxx, the last line before it dumps the core is:

Reading symbols from /lib/libnss_files.so.2 ... (no debugging available)
Loaded symbols for /lib/libnss_files.so.2
Core was generated by `/usr/sbin/sshd'.
Program terminated with signal 11, Segmentation fault
#0 0x00a0dffc in PEM_read_bio () from /lib/libcrpto.so.6

I've tried reinstalling sshd by (yum reinstall openssh), but no luck.

***** Update ***** I was able to get it working. In case it helps anyone else, here was the critical clue:

rpm -Va openssl prints out:

prelink: /lib/libcrypto.so.0.9.8e: prelinked file was modified
S.?...... /lib/libcrypto.so.0.9.8e

Obviously something is messed up with libcrypto (the gdb core dump also pointed to libcrypto). Looking at /lib/libcrypto* I figured out what version it was running. There is no way that I know of to have yum force reinstall the dependencies, so in the end, I had to copy the libcrypto.so.0.9.8e from another working server (using wget because scp did not work). After replacing that file, all was well.

Mircea Vutcovici
  • 18,319
John P
  • 1,689

2 Answers2

3

This is a CentOS system. You were not reinstalling the entire suite. The relevant packages for OpenSSH are: openssh, openssh-server and openssh-clients. The sshd binary is part of the openssh-servers package.

Try yum reinstall openssh openssh-server openssh-clients and see if that works.

In general, to check the consistency of the packages on the system, you can use rpm verify.

rpm -vV packagename for an individual package.

or

rpm -vVa to check consistency of all installed packages.

ewwhite
  • 198,150
  • Besides the binaries, that are easy to check and recover, he has to check the consistency of the data that is stored on the same fs (e.g. database files, web sites etc.). – Mircea Vutcovici Apr 23 '12 at 02:20
  • The "yum reinstall" did not reinstall the libraries - I had to manually update the so file – John P Apr 23 '12 at 13:08
2

I think that one of the libraries used by sshd or may be pamlib are corrupted. Best is to check the signatures of all packages, then verify your data/configuration integrity. If you can build the system from scratch and use a fresh backup.

Mircea Vutcovici
  • 18,319
  • You're right - turned out to be libcrypto. For bonus points, is there any way to have rpm/yum check the signatures of all installed packages? Just in case something else is corrupted that I just have not triggered yet. – John P Apr 23 '12 at 02:07