How do I read line by line a text file in ruby (hosting it on s3)?

Question

I know I've done this before and found a simple set of code, but I cannot remember or find it :(.

I have a text file of records I want to import into my Rails 3 application.

Each line represents a record. Potentially it may be tab delimited for the attributes, but am fine with just a single value as well.

How do I do this?

score 48 · Answer 1 · edited May 23 '17 at 12:25

48

File.open("my/file/path", "r").each_line do |line|
  # name: "Angela"    job: "Writer"    ...
  data = line.split(/\t/)
  name, job = data.map{|d| d.split(": ")[1] }.flatten
end

Related topic

What are all the common ways to read a file in Ruby?

edited May 23 '17 at 12:25

Community

1
1

answered Apr 27 '11 at 19:06

fl00r

81,403
31
214
233

3

This doesn't address the problem of the file being on s3 – Patm Jun 13 '13 at 20:29
1

@Patm, oh I see :) But at the moment all three answerers were answering the topic wasn't about S3. It was just `How do I read line by line a text file in ruby` – fl00r Jun 14 '13 at 18:47
Is there a way to do so when the file sits in s3? – Satchel Oct 04 '14 at 04:43
@Angela File.open("insert url here") – Tejas Manohar Nov 19 '14 at 21:16
1

Does this close the file properly? – Joshua Pinter Jul 02 '15 at 15:04
@fl00r What makes you think that this code properly closes the file? See: `a = File.open('path', 'r'); a.each_line do |line| puts line end; puts a.closed?` prints `false`! – hagello Jan 04 '18 at 12:46
@fl00r According to the link you are quoting, an explicit `close` is necessary. – hagello Jan 04 '18 at 12:48

score 21 · Answer 2 · answered Apr 27 '11 at 19:03

21

You want IO.foreach:

IO.foreach('foo.txt') do |line|
  # process the line of text here
end

Alternatively, if it really is tab-delimited, you might want to use the CSV library:

File.open('foo.txt') do |f|
  CSV.foreach(f, col_sep:"\t") do |csv_row|
    # All parsed for you
  end
end

answered Apr 27 '11 at 19:03

Phrogz

284,740
104
634
722

Does IO.foreach provide an iterator? – Zach Smith Sep 16 '17 at 10:48

score 4 · Answer 3 · answered Apr 27 '11 at 19:03

4

  IO.foreach("input.txt") do |line| 
    out.puts line
    # You might be able to use split or something to get attributes
    atts = line.split
  end

answered Apr 27 '11 at 19:03

Upgradingdave

12,596
10
61
72

score 1 · Answer 4 · answered Oct 01 '14 at 02:46

1

Have you tried using OpenURI (http://ruby-doc.org/stdlib-2.1.2/libdoc/open-uri/rdoc/OpenURI.html)? You would have to make your files accessible from S3.

Or try using de aws-sdk gem (http://aws.amazon.com/sdk-for-ruby).

answered Oct 01 '14 at 02:46

guapolo

2,303
2
19
19

the files would have permission , but I guess it still needs to be able to use the answer at the top using the File class to go through it and I wasn't clear it could. – Satchel Oct 04 '14 at 04:43

score 1 · Answer 5 · answered Dec 31 '14 at 15:17

You can use OpenURI to read remote or local files.

Assuming that your model has an attachment named file:

# If object is stored in amazon S3, access it through url
file_path = record.file.respond_to?(:s3_object) ? record.file.url : record.file.path
open(file_path) do |file|
  file.each_line do |line|
    # In your case, you can split items using tabs
    line.split("\t").each do |item|
      # Process item
    end
  end
end

How do I read line by line a text file in ruby (hosting it on s3)?

5 Answers5