0

There were 3 disk drives on the server, but one failed with input/output error and is not recoverable.

When trying to boot with the remaining 2 drives, I get:

Welcome to emergency mode!

After logging in, type "journalctl -xb" to view system logs, "systemctl reboot" to reboot, "systemctl default" or ^D to try again to boot into dafault mode.

Give root password for maintainence (or type control-D to continue):

I have tried fsck, but it says nothing besides the drives being clean.

How could I proceed, in order to have a bootable system again, without using "format and reinstall"? Or, at least, be sure that this system is not recoverable?

Are you using Ubuntu? Sure!

  • cat /etc/issue : Ubuntu 16.04.6 LTS \n \l;

Is it a RAID? Human owner said: "no".

  • cat /etc/mdadm.conf : No such file or directory;
  • cat /proc/mdstat : Personalities: [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]. unused devices: <none>.

Have you deleted the bad 3rd one from /etc/fstab?

I don't think there is anything there to be deleted there, as cat /etc/fstab only lists volumes of the devices sda and sdb, the same devices listed by lsblk. So there is no 3rd drive on /etc/fstab.

The only curious thing is that dev/sda1, listed as <mount point> /, has this <option>: errors=remount-ro. And the suggestion to use touch /forcefsck doesn't solve anything, because this file does not exist, as comproved by nano /forcefsck.

As it asks, have you run journalctl -xb?

I did not, because most of my work experience is with Windows, and any suggestion on a crash screen there can, and should, be completely ignored, as they are not helpful. But I found out that the journalctl -xb is very helpful, and even interesting to read.

But I ran now and found these 3 lines in RED color:

  1. line 636 Feb 17 07:08:04 ██████ kernel: ERST: Can not request [mem 0xd7e6e000-0xd7e6ffff] for ERST.
  2. line 1249 Feb 17 07:09:34 ██████ systemd[1]: Timed out waiting for device dev-disk-by\x2duuid-82271ee0\x2dc055\x2d497a\x2db52f\x2d566c8e456f29.device .
  3. line 1879 Feb 17 07:09:36 ██████ iscsid[1580]: iSCSI daemon with pid=1581 started!

I also found out these errors that seems to be interesting:

line 1248 Feb 17 07:09:34 ██████ systemd\[1\]: dev-disk-by\x2duuid-82271ee0\x2dc055\x2d497a\x2db52f\x2d566c8e456f29.device: Job dev-disk-by\x2duuid-82271ee0\x2dc055\x2d497a\x2db52f\x2d566c8e456f29.device/started timed out.
line 1249 Feb 17 07:09:34 ██████ systemd\[1\]: Timed out waiting for device dev-disk-by\x2duuid-82271ee0\x2dc055\x2d497a\x2db52f\x2d566c8e456f29.device .
line 1250 -- Subject: Unit dev-disk-by\x2duuid-82271ee0\x2dc055\x2d497a\x2db52f\x2d566c8e456f29.device has failed
    [...]
line 1256 -- The result is timeout.
line 1257 Feb 17 07:09:34 ██████ systemd\[1\]: Dependency failed for File System Check on /dev/disk/by-uuid/82271ee0-c055-497a-b52f-566c8e456f29.
    [...]
line 1265 Feb 17 07:09:34 ██████ systemd\[1\]: Dependency failed for /data.
    [...]
line 1273 Feb 17 07:09:34 ██████ systemd\[1\]: Dependency failed for Local File Systems.
    [...]
line 1281 Feb 17 07:09:34 ██████ systemd\[1\]: local-fs.target: Job local-fs.target/start failed with result 'dependency'.
line 1282 Feb 17 07:09:34 ██████ systemd\[1\]: local-fs.target: Triggering OnFailure= dependencies.
line 1283 Feb 17 07:09:34 ██████ systemd\[1\]: data.mount: Job data.mount/start failed with result 'dependency'.
line 1284 Feb 17 07:09:34 ██████ systemd\[1\]: systemd-fsck@dev-disk-by\x2duuid-82271ee0\x2dc055\x2d497a\x2db52f\x2d566c8e456f29.service: Job systemd-fsck@dev-disk-by\x2duuid-82271ee0\x2dc055\x2d497a\x2db52f\x2d566c8e456f29.service/start failed with result 'dependency'. 
line 1285 Feb 17 07:09:34 ██████ systemd\[1\]: dev-disk-by\x2duuid-82271ee0\x2dc055\x2d497a\x2db52f\x2d566c8e456f29.device: Job dev-disk-by\x2duuid-82271ee0\x2dc055\x2d497a\x2db52f\x2d566c8e456f29.device/start failed with result 'timeout'.

Post not suitable for "Ask Ubuntu", because, as said there, "16.04 is EOL and therefore off topic here".

BsAxUbx5KoQDEpCAqSffwGy554PSah
  • 153
  • 2
    If you now have only 2, have you deleted the bad 3rd one from /etc/fstab? And as it asks, have you run journalctl -xb? – Bib Feb 16 '23 at 22:26
  • @Bib Thank you for trying to help me. I edited my question text to reply to your first question. As for the second one, journalctl -xb has 1922 lines. Am I supposed to read every line, or search for something more specific? – BsAxUbx5KoQDEpCAqSffwGy554PSah Feb 17 '23 at 11:51
  • 1
    You should review every line and find out why it's bombing out, sort out the wheet from the chaff. Start from the bottom and go up. Sorry, but when this sort of thing happens, there is no magic wand. – Bib Feb 17 '23 at 12:00
  • @Bib Did, updated my question text up there. Found errors. – BsAxUbx5KoQDEpCAqSffwGy554PSah Feb 17 '23 at 13:04
  • Going by the logs, I assume, /dev/disk/by-uuid/82271ee0-c055-497a-b52f-566c8e456f29 which is mounted on /data is borked. If it's iscsi, I would be looking to see if you can contact its server correctly and to whether it has issues. – Bib Feb 17 '23 at 13:13
  • @Bib I just need the server to boot up correctly, leaving the cursed emergency mode. I don't need the /data if it is not essencial to the operating system. So I commented it out with nano /etc/fstab, and the server correctly booted up. What I don't understand is why it was not being reached, if cat /etc/fstab said # /data was on /dev/sdb5 during installation. It ceased to be local? – BsAxUbx5KoQDEpCAqSffwGy554PSah Feb 17 '23 at 13:31
  • Who knows... You would need to review, modify, test and see how things went. If it works though, then it works... – Bib Feb 17 '23 at 13:58
  • @Bib Would you like to fomally provide the answer to the original question, by quoting the solution I have found, that is, commenting the line on /etc/fstab? – BsAxUbx5KoQDEpCAqSffwGy554PSah Feb 17 '23 at 14:57

1 Answers1

3

Linux startup will bail out for a number of reasons resulting in emergency mode.

One possibility is that /etc/fstab has a mount configured which either no longer exists or is corrupted in some way.

In this instance, /dev/disk/by-uuid/82271ee0-c055-497a-b52f-566c8e456f29 which is mounted on /data is not functioning correctly.

Therefore, /etc/fstab should be edited with the offending line either deleted or commented out.

This should result in the server being brought up as normal from which any additional investigation can begin.

Bib
  • 2,380
  • Thank you very much! I would never guess about looking at /etc/fstab if you did not mention it in the comments. About "any additional investigation", I will just carry on if I'm able. I don't need to do a report about the disk failure, just need to configure the server to be used for something (else). – BsAxUbx5KoQDEpCAqSffwGy554PSah Feb 17 '23 at 19:56