Unable to move correct files to the mvfiles

Question

I am trying to grep for the delimiters (comma or pipe or semicolon) in the multiple files.

If the file contains any of these delimiters then it is fine. I want to move files that don't contain any of these delimiters to the mvfiles directory. The script is currently moving all the files even if the delimiters exist in the files.

filename=$(find /opt/interfaces/sample_check -type f \( -name "*message.txt*" -or -name "*comma2*" -or -name "*comma3*" \))
pathname=/opt/interfaces/sample_check

echo $filename
echo $pathname

if `head -1 $filename | grep -o [';']`; then

    echo "Found"
else
    mv $filename /opt/interfaces/sample_check/mvfiles
fi

Answer 1

Try adjusting your logic a little. Also, include all your delimiters in your grep pattern, and tweak where you put your quotes for it.

pathname=/opt/interfaces/sample_check
find "$pathname" -type f \( -name "*message.txt*" -or -name "*comma[23]*" \) |
  while read -r filename
  do if sed -n '1d; 2{ /[,;|]/q0 }; q1' "$filename"
     then echo "Delimited: $filename"
     else echo "Moving ==>> $filename"
          mv "$filename" /opt/interfaces/sample_check/mvfiles/
     fi
  done

Since we only want to decide based on delimiters in line 2 of the file, let's use sed instead.

sed -n '1d; 2{/[,;|]/q0 }; q1' 

sed -n says print nothing unless requested - we don't need any output.

1d deletes the first line (we're not editing, just abandoning any further processing on this line so it skips the rest of the program.)

2{...} says do these commands only on line 2. /[,;|]/q0 says if the line has any comma, semicolon, or pipe, then quit with a zeo exit code to indicate success.

q1 says if it gets here, quit with exit code of 1.

These trigger the branching of the if. :)

Answer 2

I would avoid putting all file names into one variable because you would get problems if there are filenames that contain spaces. Instead I propose to read the file names line by line

pathname=/opt/interfaces/sample_check
echo $pathname

find "$pathname" -type f \( -name "*message.txt*" -or -name "*comma2*" -or -name "*comma3*" \) | while read filename
do

    echo $filename

    if head -1 "$filename" | grep '[;]' >/dev/null; then

        echo "Found"
    else
        mv "$filename" "$pathname/mvfiles/."
    fi
done

There are surely more options to solve the problem.

You can also use grep -q '[;]' instead of grep '[;]' >/dev/null on modern systems, but on old systems option -q might not work.

Note: I used .../mvfiles/. assuming mvfiles is an existing directory. This avoids overwriting or creating a file of the same name in case the directory should not exist. Additionally I used quotes to avoid problems with file names containing spaces.

Answer 3

Try using grep -L to list files that don't match.

find $pathname -type f \( -name "*message.txt*" -or -name "*comma2*" -or -name "*comma3*" \) | xargs egrep -L ";|,|\|" | xargs  -IX mv  X $pathname/mvfiles

In the above example, I use egrep because of the pipe'd OR conditions. That is, we want to specify multiple regexs to match. If the file contains any of ; , | , the filename will not be output by egrep. This leaves only files that don't match being passed to xargs. In the mac version of xargs, you can specify a replacement string with the -I parameter. For every filename output by egreg, xargs will call mv <filename> $pathname/mvfiles.

As part of a follow up question, I was asked how to only review the second line of the file. Here's a bit of code to do just that:

awk ' FNR == 2 && /[;|,]/ { print FILENAME } ' * 

The above awk will display the current filename (FILENAME) when the file record number (FNR) is 2 (the second line in each file) and the input string matches the regex [;|,].

To inject that bit of code in my answer above, you can do this:

find $pathname -type f \( -name "*message.txt*" -or -name "*comma2*" -or -name "*comma3*" \) | xargs awk ' FNR == 2 && /[;|,]/ { print FILENAME } '   | xargs  -IX mv  X $pathname/mvfiles

So above, I replaced the 'xargs egrep' with 'xargs awk'. Also, I removed the * from the end of awk command because that's basically xargs default function - to take all of the input on stdin and provide it as input to the command ( in this case, awk ).

Since we are using awk, we can actually avoid the last use of xargs to move the files. Instead, we can build the commands and pipe them to bash (or some shell).

Here's a version where we pipe commands to bash:

find $pathname -type f \( -name "*message.txt*" -or -name "*comma2*" -or -name "*comma3*" \) | xargs awk ' FNR == 2 && /[;|,]/ { print "mv " FILENAME  " '$pathname'/mvfiles" } '   | bash

If you want to just debug the above statement without taking actions, you can remove the bash command at the end to leave this debug version:

Debug version (doesn't move files - only prints mv commands):

find $pathname -type f \( -name "*message.txt*" -or -name "*comma2*" -or -name "*comma3*" \) | xargs awk ' FNR == 2 && /[;|,]/ { print "mv " FILENAME  " '$pathname'/mvfiles" } '  

Answer 4

From what I understand, you want to loop through a list of filenames that matches your search then check their content to see if it contains either a pipe, comma or semicolon. If that's the case you can use this.

# pathname is your directory you want files to search in
pathname=/opt/interfaces/sample_check

# here you can add different filenames that you want to match. I.e. only take files in the loop which contain filename1 in the name, or starts with filename2, or starts with filename3.
find $pathname -type f \( -name "*filename1*" -or -name "filename2*" -or -name "filename3*" \) -print0 |
while IFS= read -r -d '' file; do
    if grep -e '[|;,]' "$file"
      then
        echo "Found in $file."
    else
        echo "Not found in $file. Moving to directory xxx."
        mv "$file" /opt/interfaces/sample_check/mvfiles
    fi
done