0

Summary:

  • Had a directory at ~/src/database containing ~250 GB of files.
  • Executed mv ~/src/database ~/
  • Instead of expected result (database now at ~/), I ended up with a database directory in both the source and destination, but with only a small fraction of the original files (i.e. the database was apparently corrupted/lost during mv somehow).

The lost files are not an issue---I have multiple backups. The issue is that even after doing rm -rf ~/database; rm -rf ~/src/database my harddrive is now short ~500 GB (coincidentally, the size of 2 copies of the database).

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme1n1p2  1.9T  1.1T  701G  61% /

However, a complete breakdown of the drive in baobab shows that I'm only using ~600 GB with all files accounted for on the root volume. This is consistent with what I knew beforehand and also the failed mv operation. The math is roughly correct: 1.1 TB - 500 GB = 600 GB, which is exactly what baobab says I'm using.

In the past I have had similar issues, and the solution was to remove the parent directory, which somehow reclaimed the space. That isn't working here.

  1. I already tried removing the parent of the source, didn't work.
  2. Parent of the destination is my home dir, so not easy to rm that.

Question:

How can I reclaim this space? Is there a way to force the OS to reclaim stranded nodes from the disk?

$ findmnt -T ~/src
TARGET SOURCE         FSTYPE OPTIONS
/      /dev/nvme1n1p2 ext4   rw,relatime,discard
$ findmnt -T ~/
TARGET SOURCE         FSTYPE OPTIONS
/      /dev/nvme1n1p2 ext4   rw,relatime,discard
EntangledLoops
  • 143
  • (1) mv is equivalent to cp+rm if from one filesystem to another (or in some rare circumstances (example)). The question suggests there is just one filesystem involved, so in theory there was no need for cp. Is this really so? What is the output of findmnt -T ~/src/? findmnt -T ~/? What is the type of the filesystem? Are you sure it's healthy? (2) B (byte) is 8 times larger than b (bit). GB is 8 times larger than Gb. It's very uncommon to measure disk usage in Gb or Tb. I don't know where 1.1 Tb came from. Check your units. – Kamil Maciorowski Jun 04 '23 at 17:15
  • Not sure what you're talking about exactly. Yes, mv is basically cp+rm, and it appears the cp portion failed and we never reached rm. This is just a standard directory move, same as if you used a UI tool to drag a directory from one place to another on the same drive. The units involved here are all Gigabytes and Terabytes. I updated the b -> B if you want to be pedantic, but doesn't change the issue. Not sure if filesystem is healthy. Can you suggest a way to check that? – EntangledLoops Jun 04 '23 at 17:20
  • In some circumstances mv is cp+rm. Within a single filesystem it's (almost always) not, there is no cp to begin with, there is only rename. I think it's important to know how many filesystems were involved, hence my requests for findmnt. – Kamil Maciorowski Jun 04 '23 at 17:24
  • There is one filesystem involved. There is only one harddrive involved. (Updated post.) – EntangledLoops Jun 04 '23 at 17:44
  • OK, so ext4. If the linked example does not apply then I guess there is (and there was) no need for cp "inside" mv. The stranger "the size of 2 copies of the database" is. The tool to check ext4 is fsck.ext4 aka e2fsck (the filesystem must not be mounted; or check this question). – Kamil Maciorowski Jun 04 '23 at 17:55
  • To make fsck easy, boot from USB (or other drive), e.g., a Live Ubuntu or other Linux. The Disks utility (on Live Ubuntu, also) is also a convenient GUI for partition check and repair, again requiring the drive under repair not be mounted. – DrMoishe Pippik Jun 04 '23 at 18:00
  • Silly question - given the name, database, is this an odd storage location of the files of an actual database application and you tried to move it while the process utilising said database was still running? – tink Jun 04 '23 at 21:31
  • @tink It's a special kind of database (rocksdb). It was not in use. This is a large folder of many small-ish files (~50 Mb each) optimized for quick access during neural net training. – EntangledLoops Jun 05 '23 at 00:52
  • I'm thinking the drive is failing. I'm running a neural net training process for long periods of time (weeks). The entire system is watercooled except for the NVME, which is running super hot. The manufacturer sticker is peeling off due to the heat. Combine that with thousands of read/write operations per minute over weeks....it may be just hardware failure in part of the drive, or overheating causing the drive to disable some regions. – EntangledLoops Jun 05 '23 at 00:55

0 Answers0