27

I have generated a wordlist.txt of 11 GB by crunch-3.6. When I try to open the file with Vi or gedit, I run into problems because of the file size. How can I view this file?

Nathan Smith
  • 257
vais sai
  • 273
  • 8
    I don't think it's a duplicate. Although its closely related and some answers might be helpful, this one is about viewing and not editing. The accepted answer is not helpful for viewing. – pLumo Nov 09 '17 at 09:07
  • 1
    I agree. Viewing a file and editing a file are not the same thing, and the distinction only becomes more important when the file is very large. – Eliah Kagan Nov 14 '17 at 15:58

2 Answers2

48

Don't use a text editor for viewing text.

There are better tools:

View files with less (Scroll with Space, End, Home, PageUp, PageDown; Search with "/something" ; Leave with q).

From less manual:

Less does not have to read the entire input file before starting, so with large input files it starts up faster than text editors like vi (1).

Usage:

less wordlist.txt

Consider the use of less -n:

-n or --line-numbers:

Suppresses line numbers. The default (to use line numbers) may cause less to run more slowly in some cases, especially with a very large input file. Suppressing line numbers with the -n option will avoid this problem.

(thanks for suggesting -n option @pipe)

Use grep to get only the lines you're interested in:

# Show all Lines beginning with A:
grep "^A:"  wordlist.txt

# Show all Lines ending with x and use less for better viewing
grep "x$"  wordlist.txt | less

Use head or tail to get the first or last n lines

head wordlist.txt
tail -n 200 wordlist.txt

For editing text, refer to this question.

pLumo
  • 26,947
  • 4
    Note that more will load the whole file in memory, unlike less. If it's that big, I'd advise against using it. The best option would definitely be to use grep. – Nepho Nov 09 '17 at 13:06
  • I removed more as it does not really add any benefit compared to less – pLumo Nov 09 '17 at 13:17
  • 1
    My comment might actually be false. I started checking both more and less source code, and both seem to use either open or fopen, which doesn't change anything regarding the way the file is loaded, as far as I know – Nepho Nov 09 '17 at 13:17
  • 2
    Agreed. more is a very dated utility, if less is available I don't think of any reason why you'd use more. – Nepho Nov 09 '17 at 13:18
  • 1
    @Nepho, more has one huge advantage over less: it doesn't support the LESSOPEN input preprocessor. If you're trying to view the raw text of a man page or something, it's much faster to type more my_docs.man than to dig through the less documentation to figure out how to suppress the preprocessor. – Mark Nov 09 '17 at 23:34
  • 1
    I suggest invoking less using something like less -nS for huge files, to speed things up a little. – pipe Nov 10 '17 at 12:09
  • thanks@ pipe. I added -n, but didn't find information on why -S should provide better performance. – pLumo Nov 10 '17 at 12:22
11

Often, just "grep" is enough to find what you need.

If you need more "context" around a particular line, then use "grep -n" to find the line numbers of the lines of interest, then use sed to print out a "chunk" of the file around that line: 

$ grep -n 'word' file 
123:A line with with word in it 

$ sed -n '120,125p' file 
A line 
Another line
The line before
A line with with word in it 
The line after
Something else
Phill W.
  • 211