How to open tar gz file created using tar piped to gzip

Question

I wanted to generate a tar gz file with the highest compression level, and I followed the solution from https://superuser.com/a/546509/461740:

tar cv /path/to/directory | gzip -9 > file.tar.gz

But now, I cannot uncompress the file, I have the following error:

tar: This does not look like a tar archive
tar: Skipping to next header
tar: Archive contains ‘23241_at',16’ where numeric off_t value expected

I googled this error, in most cases it is caused by a "double" gzip compression, I tried the solutions proposed for this problem, but it doesn't seem to correspond to my problem.

Any idea how to fix that?

Edit: of note, when I call the command file on the uncompressed file (that is supposed to be a tar file), I got the result: data.

Edit: result of head -c512 file.tar | hexdump -C:

00000000  73 71 6c 5f 64 75 6d 70  2f 0a 73 71 6c 5f 64 75  |sql_dump/.sql_du|
00000010  6d 70 2f 62 67 65 65 46  6f 72 65 69 67 6e 4b 65  |mp/bgeeForeignKe|
00000020  79 2e 73 71 6c 0a 73 71  6c 5f 64 75 6d 70 2f 00  |y.sql.sql_dump/.|
00000030  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000080  00 00 00 00 00 00 00 00  00 00 30 30 30 30 37 35  |..........000075|
00000090  35 00 30 30 30 37 36 34  33 00 30 30 31 31 33 30  |5.0007643.001130|
000000a0  30 00 30 30 30 30 30 30  30 30 30 30 30 00 31 32  |0.00000000000.12|
000000b0  35 33 35 31 36 30 34 34  34 00 30 31 32 37 30 33  |535160444.012703|
000000c0  00 20 35 00 00 00 00 00  00 00 00 00 00 00 00 00  |. 5.............|
000000d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000120  00 00 00 00 00 00 00 75  73 74 61 72 20 20 00 66  |.......ustar  .f|
00000130  62 61 73 74 69 61 6e 00  00 00 00 00 00 00 00 00  |bastian.........|
00000140  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 75  |...............u|
00000150  6e 69 6c 64 65 65 00 00  00 00 00 00 00 00 00 00  |nildee..........|
00000160  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000200

Following @kasperd command to try to repair my file, tail -c+39 file.tar > repaired.tar, hexadecimal dump of the first 512 bytes:

00000000  73 71 6c 5f 64 75 6d 70  2f 00 00 00 00 00 00 00  |sql_dump/.......|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000060  00 00 00 00 30 30 30 30  37 35 35 00 30 30 30 37  |....0000755.0007|
00000070  36 34 33 00 30 30 31 31  33 30 30 00 30 30 30 30  |643.0011300.0000|
00000080  30 30 30 30 30 30 30 00  31 32 35 33 35 31 36 30  |0000000.12535160|
00000090  34 34 34 00 30 31 32 37  30 33 00 20 35 00 00 00  |444.012703. 5...|
000000a0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000100  00 75 73 74 61 72 20 20  00 66 62 61 73 74 69 61  |.ustar  .fbastia|
00000110  6e 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |n...............|
00000120  00 00 00 00 00 00 00 00  00 75 6e 69 6c 64 65 65  |.........unildee|
00000130  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000200

Result of tar -tvf on "repaired" tar:

drwxr-xr-x username  0 2015-06-08 02:26 sql_dump/
-rw-r--r-- username 19123 2015-06-08 02:25 sql_dump/bgeeForeignKey.sql
tar: Skipping to next header
tar: Archive contains `688_x_at\',1' where numeric off_t value expected
tar: Archive contains `y\',\'not ' where numeric mode_t value expected
tar: Archive contains `725,\'ENSG' where numeric time_t value expected
tar: Archive contains `excluded' where numeric uid_t value expected
tar: Archive contains `\'),(\'208' where numeric gid_t value expected
tar: Archive contains `excluded' where numeric uid_t value expected
tar: Archive contains `\'),(\'208' where numeric gid_t value expected
?rwsrwsrwt 4294967295/4294967295 18446744073709551615 1970-01-01 00:59 ty','not excluded'),('208686_s_at',13725,'ENSG00000204256',7.73,'present',15097863,NULL,'high qualit unknown file type `\''
tar: Skipping to next header
tar: Archive contains `ent\'),(31801' where numeric off_t value expected
tar: Archive contains `no data\'' where numeric mode_t value expected
tar: Archive contains `347,\'ENSG' where numeric time_t value expected
tar: Archive contains `,\'no dat' where numeric uid_t value expected
tar: Archive contains `a\',\'desc' where numeric gid_t value expected
tar: Archive contains `,\'no dat' where numeric uid_t value expected
tar: Archive contains `a\',\'desc' where numeric gid_t value expected
?rwsrwsrwt 4294967295/4294967295 18446744073709551615 1970-01-01 00:59 ,'descent'),(31801346,'ENSG00000104375','UBERON:0007625','HsapDv:0000095','no data','poor quality',' unknown file type `\''
tar: Skipping to next header
tar: Exiting with failure status due to previous errors

Output of dd if=repaired.tar skip=39 count=2 | hexdump -C

2+0 records in
2+0 records out
1024 bytes (1.0 kB) copied00000000  64 64 20 66 6f 72 65 69  67 6e 20 6b 65 79 20 28  |dd foreign key (|
00000010  64 69 66 66 65 72 65 6e  74 69 61 6c 45 78 70 72  |differentialExpr|
00000020  65 73 73 69 6f 6e 49 64  29 20 72 65 66 65 72 65  |essionId) refere|
00000030  6e 63 65 73 20 64 69 66  66 65 72 65 6e 74 69 61  |nces differentia|
00000040  6c 45 78 70 72 65 73 73  69 6f 6e 28 64 69 66 66  |lExpression(diff|
, 6.9324e-05 s, 14.8 MB/s
00000050  65 72 65 6e 74 69 61 6c  45 78 70 72 65 73 73 69  |erentialExpressi|
00000060  6f 6e 49 64 29 20 6f 6e  20 64 65 6c 65 74 65 20  |onId) on delete |
00000070  73 65 74 20 6e 75 6c 6c  3b 0a 2f 2a 21 34 30 30  |set null;./*!400|
00000080  30 30 20 41 4c 54 45 52  20 54 41 42 4c 45 20 60  |00 ALTER TABLE `|
00000090  64 65 61 52 4e 41 53 65  71 53 75 6d 6d 61 72 79  |deaRNASeqSummary|
000000a0  60 20 45 4e 41 42 4c 45  20 4b 45 59 53 20 2a 2f  |` ENABLE KEYS */|
000000b0  3b 0a 0a 00 00 00 00 00  00 00 00 00 00 00 00 00  |;...............|
000000c0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000200  73 71 6c 5f 64 75 6d 70  2f 52 45 41 44 4d 45 2e  |sql_dump/README.|
00000210  74 78 74 0a 73 71 6c 5f  64 75 6d 70 2f 62 67 65  |txt.sql_dump/bge|
00000220  65 49 6e 64 65 78 2e 73  71 6c 0a 73 71 6c 5f 64  |eIndex.sql.sql_d|
00000230  75 6d 70 2f 64 75 6d 70  5f 62 67 65 65 5f 76 31  |ump/dump_bgee_v1|
00000240  33 5f 31 2e 73 71 6c 0a  73 71 6c 5f 64 75 6d 70  |3_1.sql.sql_dump|
00000250  2f 52 45 41 44 4d 45 2e  74 78 74 00 00 00 00 00  |/README.txt.....|
00000260  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000002a0  00 00 00 00 00 00 00 00  00 00 00 00 30 30 30 30  |............0000|
000002b0  36 34 34 00 30 30 30 37  36 34 33 00 30 30 31 31  |644.0007643.0011|
000002c0  33 30 30 00 30 30 30 30  30 30 30 32 30 36 34 00  |300.00000002064.|
000002d0  31 32 35 33 35 31 36 30  31 37 34 00 30 31 34 34  |12535160174.0144|
000002e0  30 33 00 20 30 00 00 00  00 00 00 00 00 00 00 00  |03. 0...........|
000002f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000340  00 00 00 00 00 00 00 00  00 75 73 74 61 72 20 20  |.........ustar  |
00000350  00 66 62 61 73 74 69 61  6e 00 00 00 00 00 00 00  |.fbastian.......|
00000360  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000370  00 75 6e 69 6c 64 65 65  00 00 00 00 00 00 00 00  |.unildee........|
00000380  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000400

I tried gzip -d my_file.tar.gz, gunzip my_file.tar.gz, and zcat my_file.tar.gz, but I still have the tar error afterwards. — FBB, Jun 23 '15 at 13:04
Please add the version of tar tar --version and tell if there is some NFS or AFS directory — Hastur, Jun 23 '15 at 13:24
@gronostaj: you mean, gunzipping again the tar.gz? Or gunzipping the uncompressed tar? — FBB, Jun 23 '15 at 13:29
I suppose GNU tar. Resume: you uncompress the file and when asked the system said to you that is a gzipped file. Then the problem is not there. When you do file file.tar it should answer you POSIX tar archive (GNU). If it answers data means that it misses the magic seed and/or is corrupted the tar. — Hastur, Jun 23 '15 at 13:42
Sorry, I meant gunzip -f my_file.tar.gz and then file my_file.tar — gronostaj, Jun 23 '15 at 13:43
Yep, so, file my_file.tar says "data". I am currently trying the command from Nafscript's update. @Hastur: if the tar is corrupted, is there any way of trying a rescue? — FBB, Jun 23 '15 at 13:49
The first 512 bytes of repaired.tar are perfectly fine. It is a valid header describing a directory. (And since it is a directory, no data blocks is needed after it). So the next 512 bytes of the file has to be another header. If every header in the tar archive are corrupted in the same way, then it is going to be a lot of work to repair. But for now, we don't know what other corruption you have, just that there is other corruption. What does tar -tvf reparied.tar say? — kasperd, Jun 25 '15 at 07:08
Try extracting just sql_dump/bgeeForeignKey.sql from the archive. Are the contents of that file intact? Is the file name correct? — kasperd, Jun 25 '15 at 22:06
@kasperd sorry, but how can I extract a single file from the archive? — FBB, Jun 26 '15 at 10:14
@kasperd the file is correctly extracted, however it gives the same error as with the tar -tvf command, but without the lines starting with ?rwsrwsrwt, and without the redundant lines. I will try with the other files... — FBB, Jun 26 '15 at 12:00
In that case it sounds like the next corruption is within the next header. This should show the last block of the first file, and the header of the next: dd if=repaired.tar skip=39 count=2 | hexdump -C — kasperd, Jun 26 '15 at 15:01
@FBB I am starting to see a picture here. What was the exact tar version used to produce the file? — kasperd, Jun 26 '15 at 15:24
@FBB If my suspicion is right, you won't be able to extract your current tar file. But you can make it work, if you start over with a slight modification to the command used to produce the tar file in the first place. — kasperd, Jun 26 '15 at 15:34
@FBB I found machines with GNU tar versions 1.22 and 1.26. Neither version would behave like you describe, so it sounds unlikely that 1.23 would. I am suspecting your archive was not produced using the exact command you mention in your question. Could you test tar cv /path/to/directory | tar -t | wc on the system where the tar file was produced? — kasperd, Jun 29 '15 at 10:52

Nafscript · Answer 1 · 2015-06-23T13:30:26.990

0

tar -xvfz file.tar.gz

-x extract
-z gunzip
-v verbose

Update

Also try this:

gunzip < file.tar.gz | tar xvf -

edited Jun 23 '15 at 13:30

answered Jun 23 '15 at 12:46

Nafscript

192

It doesn't work. – FBB Jun 23 '15 at 13:08
The second command didn't seem to work :/ – FBB Jun 23 '15 at 14:26
Have you tried to compress and extract another file with suggested commands? Seems your archive is corrupted – Nafscript Jun 23 '15 at 15:09
I just did, and it worked perfectly :/ – FBB Jun 24 '15 at 13:25

Hastur · Answer 2 · 2015-06-23T12:54:30.207

0

If all is ok it should be enough

tar -zxvf file.tar.gz

the z option is needed for the gzip compression.
On tar (GNU tar) 1.27.1 it works too without

tar -xvf Myfile.tar.gz

It there is no special reason to do in the way you used, you can directly create a tarred gzipped file with

tar -zcvf Myfile.tar.gz  /path/to/directory

or with a different compression program as

tar -jcvf Myfile.tar.bz  /path/to/directory

To check which kind of file you have to dial with use the command file

file *.tar.*

Myfile.tar.gz: gzip compressed data, from Unix, last modified:...
Myfile.tar.bz: bzip2 compressed data, block size = 900k

edited Jun 23 '15 at 12:54

answered Jun 23 '15 at 12:48

Hastur

18,942

Of course I tried these standard commands, it doesn't work. – FBB Jun 23 '15 at 13:08
Do you mean you tryed tar -zcvf Myfile.tar.gz /path/to/directory and after tar -zxvf file.tar.gz and it doesn't work? What about file Myfile.tar.gz? – Hastur Jun 23 '15 at 13:18
I tried these commands, yes. file Myfile.tar.gz says "gzip compressed data" – FBB Jun 23 '15 at 13:31

kasperd · Answer 3 · 2015-06-24T13:44:43.243

There is nothing wrong with the command you say was used to create the archive. You can list the contents and verify the checksum of the archive using this command:

tar -tzf file.tar.gz

From your question it sounds like the file you have now is not really a .tar.gz file. Something unintended must have happened to the file, and the possibilities I can come up with are:

You somehow mistyped the command to produce the archive in the first place.
Either the tar command or the gzip command in your PATH when you ran the command were not the real versions but rather something else producing a bad output.
You had an environment variable overriding some library used by the programs with a broken version.
A hardware error caused the data to be corrupted.
Some program has been used to modify the file between the time you created it and the time you try to extract it.

In order to figure out what has really gone wrong with this file and how it could possibly be recovered, we need to look closer on the actual contents of the file.

In the hexdump provided in your updated question I notice that many of the fields are not at the proper alignment within the header. Comparing with a good tar file, I see that in yours the fields are located 38 bytes too far inside the file.

Moreover I notice that the first directory inside your archive has a very long name which even has a newline in the 38th byte.

So what I conclude is that the contents of your file is not just a tar archive. It is one line of text followed by a tar archive. The erroneous line reads

sql_dump/.sql_dump/bgeeForeignKey.sql

I don't know how you managed to get that line to be stored inside your .tar.gz file. But if we skip it the first directory has a much more reasonable name sql_dump/. If that is the only corruption, you will be able to recover the data by this one command:

tail -c+39 file.tar > repaired.tar

I can open the tar using vi, and read the content of my original files, but it displays the same error message as when trying to untar, at the beginning; when I call head on it, it displays weird information related to the original files in the tar, e.g.: dir/ dir/file1 dir/0000755000764300113000000000000012535160444012703 5ustar usernamedir/file10000644000764300113000000004526312535160414016320 0ustar username-- 1st line of file1. — FBB, Jun 23 '15 at 14:26
@FBB The output of head on a binary file won't help identifying the problem. But a hexadecimal dump of the first 512 bytes would tell us something. Try head -c512 file.tar | hexdump -C. — kasperd, Jun 23 '15 at 15:27
Sadly, it didn't work :/ I updated my question with the first 512 bytes of the "repaired" file. — FBB, Jun 25 '15 at 03:58
I updated my question with the result of tar -tvf on the "repaired" file. — FBB, Jun 25 '15 at 14:34

score 0 · Answer 4 · edited Mar 12 '23 at 19:51

0

You forgot minus to redirect tar output to pipe:

$ tar cf - /path/to/directory | gzip -9 > file.tar.gz

edited Mar 12 '23 at 19:51

Community

1

answered Jun 23 '15 at 14:04

Dmitry Dmitriev

1

Interesting, what is the expected behavior of my command without the minus? – FBB Jun 23 '15 at 14:21

How to open tar gz file created using tar piped to gzip

4 Answers4