9

I have about 12 GB of image tiles made up of about 2 million files. I'd like to zip these up to make transferring them to a server simpler. I just plan on storing the files in the zip files for transferring, no compression. Helm is present on the web server and can handle unzipping files.

I'd like to point a program at all these files in one go and get it to zip them up into files of approx 1 GB each, but each zip file needs to be independent of the others.

I have 7-zip installed with supports splitting across volumes, but these volumes are dependent upon one another to be unzipped.

Anyone have any suggestions? Thanks in advance!

Journeyman Geek
  • 129,178
Gavin
  • 315
  • 1
  • 3
  • 9

7 Answers7

4

The freeware on Windows called "Spinzip" should do the work for your purpose ! ;) http://skwire.dcmembers.com/wb/pages/software/spinzip.php

It is based on IZARCC (automatically included in Spinzip). You have to check but the full original path may be kept in the zipped files !

See ya

Erb
  • 405
3

I'm not aware of a program that can do that, since if you are making one zip in multi-volumes they will all be related. Your best bet may be to make 12 folders and put a GB in each one, then zip the folders individually.

JNK
  • 8,306
  • 28
  • 31
  • Any nice methods of automatically spitting the files in to approx 1 GB chunks while preserving folder structure files live under? Bit tedious to do manually with 2 million files. – Gavin Jul 27 '10 at 22:43
  • Try this program on cnet: http://download.cnet.com/File-Splitter-Deluxe/3000-2248_4-10026918.html?tag=contentMain;contentBody;1d

    I havent used it myself but it should fit the bill for a one time operation.

    – JNK Jul 28 '10 at 01:05
  • Just read the File Splitter Deluxe blurb - seems it needs a small program present to rejoin files. I need to load independent zip files up to server and use Helm interface to extract them from zips, so won't work in this case by sounds of it. Cheers for the suggestion though. – Gavin Jul 28 '10 at 09:40
3

In the end I created a quick python script to split the files in to sub directories for me before zipping each individually.

In case it's useful to anyone else, here's my script:

import os
import csv
import shutil

def SplitFilesIntoGroups(dirsrc, dirdest, bytesperdir):
    dirno = 1
    isdircreated = False
    bytesprocessed = 0

    for file in os.listdir(dirsrc):
        filebytes = os.path.getsize(dirsrc+'\\'+file)

        #start new dir?
        if bytesprocessed+filebytes > bytesperdir:
            dirno += 1
            bytesprocessed = 0
            isdircreated = False

        #create dir?
        if isdircreated == False:
            os.makedirs(dirdest+'\\'+str(dirno))
            isdircreated = True

        #copy file
        shutil.copy2(dirsrc+'\\'+file, dirdest+'\\'+str(dirno)+'\\'+file)
        bytesprocessed += filebytes

def Main():
    dirsrc='C:\\Files'
    dirdest='C:\\Grouped Files'

    #1,024,000,000 = approx 1gb
    #512,000,000 = approx 500mb
    SplitFilesIntoGroups(dirsrc, dirdest, 512000000) 

if __name__ == "__main__":
    Main()
Gavin
  • 315
  • 1
  • 3
  • 9