41

I'm trying to write a bash script that will process a list of files whose names are stored one per line in an input file, something the likes of

find . -type f -mtime +15 > /tmp/filelist.txt
for F in $(cat /tmp/filelist.txt) ; do
  ...
done;

My problem is that filenames in filelist.txt may contain spaces, so the snipped above will expand the line

my text file.txt

to three different filenames, my, text and file.txt. How can I fix that?

agnul
  • 11,900
  • 14
  • 63
  • 83
  • 1
    You know, directory entries can also contain newlines, as well as spaces and other wacky characters. The only things a directory entry (file name) can't have are the "/" and the "" characters. (slash and null). – chris Nov 06 '09 at 17:51
  • Yep, but the files I'm working on are created by windows users on my LAN accessing a samba share, so there's a limit to filename weirdness – agnul Nov 06 '09 at 18:23
  • You can also create an array from a file cf.: https://stackoverflow.com/questions/30988586/creating-an-array-from-a-text-file-in-bash See also: http://mywiki.wooledge.org/BashFAQ/005#Loading_lines_from_a_file_or_stream – Roland Mar 05 '20 at 09:08

7 Answers7

54

Use read:

while read F  ; do
        echo $F
done </tmp/filelist.txt

Alternatively use IFS to change how the shell separates your list:

OLDIFS=$IFS
IFS="
"
for F in $(cat /tmp/filelist.txt) ; do
  echo $F
done
IFS=$OLDIFS

Alternatively (as suggested by @tangens), convert the body of your loop into a separate script, then use find's -exec option to run if for each file found directly.

Douglas Leeder
  • 50,599
  • 9
  • 90
  • 136
  • 1
    Although I voted it up, I found this script fail on paths that contain non-ASCII characters. – kakyo Nov 10 '13 at 19:11
7

You can do this without a temporary file using process substitution:

while read F
do
  ...
done < <(find . -type f -mtime +15)
Dennis Williamson
  • 324,833
  • 88
  • 366
  • 429
  • Neat! Didn't know about process substitution. – agnul Nov 06 '09 at 19:09
  • 2
    Note that the process substitution feature is a bash extension, and is not even available with bash in sh-compatibility mode. You must start your script with `#!/bin/bash` for it to work. BTW, I also recommend using `while IFS="" read -r F` to avoid possible problems with whitespace at the beginning or end of filenames, and backslashes at the end (although if the files are coming from Windows, backslashes are probably not possible). – Gordon Davisson Nov 06 '09 at 20:19
5

use while read

echo $FILE | while read line
do
echo $line
done

You can do redirect instead of echo

DVK
  • 123,561
  • 31
  • 206
  • 320
2

You could use the -exec parameter of find and use the file names directly:

find . -type f -mtime +15 -exec <your command here> {} \;

The {} is a placeholder for the file name.

tangens
  • 37,881
  • 18
  • 117
  • 136
  • The problem is that I'm not running a single command and I would end up trying all types of weird combinations to find the right way to quote and escape the stuff after -exec – agnul Nov 06 '09 at 19:22
2

pipe your find command straight to while read loop

find . -type f -mtime +15 | while read -r line
do
   printf "do something with $line\n"
done
ghostdog74
  • 307,646
  • 55
  • 250
  • 337
0

I'm not a bash expert by any means ( I usually write my script in ruby or python to be cross-platform), but I would use a regex expration to escape spaces in each line before you process it.

For Bash Regex: http://www.linuxjournal.com/node/1006996

In a similar situation in Ruby ( processing a csv file, and cleaning up each line before using it):

File.foreach(csv_file_name) do |line| 
    clean_line = line.gsub(/( )/, '\ ') 
    #this finds the space in your file name and escapes it    
    #do more stuff here
end
konung
  • 6,767
  • 6
  • 52
  • 76
-1

I believe you can skip the temporary file entirely and just directly iterate over the results of find, i.e.:

for F in $(find . -type f -mtime +15) ; do
  ...
done;

No guarantees that my syntax is correct but I'm pretty sure the concept works.

Edit: If you really do have to process the file with a list of filenames and can't simply combine the commands as I did above, then you can change the value of the IFS variable--it stands for Internal Field Separator--to change how bash determines fields. By default it is set to whitespace, so a newline, space, or tab will begin a new field. If you set it to contain only a newline, then you can iterate over the file just as you did before.

qid
  • 1,853
  • 10
  • 15