text mining: read text files in Python

Question

When I use R, I can read many text documents which are contained in a file folder for one time. However, I just started to learn Python. When I use command: file = open('c:/txt/Romney', 'r'),trying to open all text files contained in that Romney folder, I found out I have to read inside text files one by one, I cannot read all for one time just like I do in R. Any suggestion?

What do you mean by "all at once"? Do you want to concatenate them? Or iterate over them one-by-one? — Patrick Collins, Jun 16 '14 at 20:49
I'm curious how R does it. That behavior sounds like R trying to guess what you might want to do — MxLDevs, Jun 16 '14 at 20:56
Posting the R code you are trying to emulate/duplicate in Python would be helpful. — hrbrmstr, Jun 17 '14 at 00:31

score 5 · Answer 1 · edited May 23 '17 at 11:56

In a language like Python, you'll need to use a for loop to read the contents of each file, one at a time.

(Related: How to list all files of a directory in Python)

from os import listdir
from os.path import isfile, join

path = "C:/txt/Romney"
files = [ f for f in listdir(path) if isfile(join(path,f)) ]

for file in files:
    with open file as f:
        text = f.read()
        do_something_with(text)

score 2 · Answer 2 · answered Jun 16 '14 at 22:20

2

In addition to Dave Yarwood's answer, if what you actually wanted to do was concatenate the files, you could do it with:

from os import listdir
from os.path import isfile, join
from itertools import chain

path = "C:/txt/Romney"
files = [open(f) for f in listdir(path) if isfile(join(path,f))]

for line in chain(*files):
    do_something_with(line)

(just for fun, because I've never used itertools.chain to string together files before)

answered Jun 16 '14 at 22:20

Patrick Collins

9,805
4
26
66

Shouldn't you then do `for file in files: file.close()`? – Dave Yarwood Jun 17 '14 at 13:41
And wouldn't it be `[f.open() for f in ...` ? – Dave Yarwood Jun 17 '14 at 13:42
@DaveYarwood The file is automatically closed when the object gets garbage collected. And `listdir` just returns a list of strings, not file handles. Strings don't have an `open` method. – Patrick Collins Jun 17 '14 at 21:47
Ah! I didn't know that. Thanks! – Dave Yarwood Jun 17 '14 at 21:52

text mining: read text files in Python

2 Answers2