Creating an array from a text file in Bash

Question

A script takes a URL, parses it for the required fields, and redirects its output to be saved in a file, file.txt. The output is saved on a new line each time a field has been found.

file.txt

A Cat
A Dog
A Mouse 
etc...

I want to take file.txt and create an array from it in a new script, where every line gets to be its own string variable in the array. So far I have tried:

#!/bin/bash

filename=file.txt
declare -a myArray
myArray=(`cat "$filename"`)

for (( i = 0 ; i < 9 ; i++))
do
  echo "Element [$i]: ${myArray[$i]}"
done

When I run this script, whitespace results in words getting split and instead of getting

Desired output

Element [0]: A Cat 
Element [1]: A Dog 
etc...

I end up getting this:

Actual output

Element [0]: A 
Element [1]: Cat 
Element [2]: A
Element [3]: Dog 
etc...

How can I adjust the loop below such that the entire string on each line will correspond one-to-one with each variable in the array?

This is what [Bash FAQ 001](http://mywiki.wooledge.org/BashFAQ/001) is all about. Also [this section](http://mywiki.wooledge.org/BashFAQ/005?highlight=%28readarray%29#Loading_lines_from_a_file_or_stream) of the array topic in [Bash FAQ 005](http://mywiki.wooledge.org/BashFAQ/005). — Etan Reisner, Jun 22 '15 at 19:51
I would link this as a duplicate of https://stackoverflow.com/questions/11393817/bash-read-lines-in-file-into-an-array, but the accepted answer there is awful. — Charles Duffy, Jun 22 '15 at 20:01
Etan, thank you so much for such a fast and accurate reply! I had tried to search my question in the forums, but did not think to look for the FAQ on stackoverflow. The mapfile command addressed my needs exactly! Thanks again :) Answer in [section 2.1](http://mywiki.wooledge.org/BashFAQ/005?highlight=%28readarray%29#Loading_lines_from_a_file_or_stream). — user2856414, Jun 22 '15 at 20:01
(Set up the link in the opposite direction, since we have a better accepted answer here than we have there). — Charles Duffy, Jun 22 '15 at 20:08

score 136 · Accepted Answer · edited Apr 22 '18 at 00:49

136

Use the mapfile command:

mapfile -t myArray < file.txt

The error is using for -- the idiomatic way to loop over lines of a file is:

while IFS= read -r line; do echo ">>$line<<"; done < file.txt

See BashFAQ/005 for more details.

edited Apr 22 '18 at 00:49

codeforester

34,080
14
96
122

answered Jun 22 '15 at 19:56

glenn jackman

223,850
36
205
328

7

Since this is being promoted as the canonical q&a, you could also include what is mentioned in the link: `while IFS= read -r; do lines+=("$REPLY"); done – fedorqui Apr 27 '16 at 11:48
10

mapfile does not exist in bash versions prior to 4.x – ericslaw Mar 30 '17 at 20:04
15

Bash 4 is about 5 years old now. Upgrade. – glenn jackman Mar 31 '17 at 01:24
7

Despite bash 4 being released in 2009, @ericslaw's comment remains relevant because many machines still ship with bash 3.x (and will not upgrade, so long as bash is released under GPLv3). If you're interested in portability, it's an important thing to note – De Novo Jan 28 '19 at 21:45
1

Sure, an OS may ship with older bash, but individuals can upgrade their own installations, or install bash 4 separately (with homebrew or the like). – glenn jackman Jan 28 '19 at 21:56
1

macos mojave bash reports as 3.2.57 (wow really apple?). I suspect work wont let me touch too many things on the laptop though :( – ericslaw Jan 29 '19 at 03:03
1

You should be able to use whatever tools you need to do your job. – glenn jackman Jan 29 '19 at 03:34
16

the issue isn't that a developer can't install an upgraded version, it's that a developer should be aware that a script using `mapfile` will not run as expected on many machines without additional steps. @ericslaw macs will continue to ship with bash 3.2.57 for the foreseeable future. More recent versions use a license that would require apple to share or allow things they don't want to share or allow. – De Novo Jan 29 '19 at 06:35

codeforester · Answer 2 · 2020-01-23T03:25:33.427

37

mapfile and readarray (which are synonymous) are available in Bash version 4 and above. If you have an older version of Bash, you can use a loop to read the file into an array:

arr=()
while IFS= read -r line; do
  arr+=("$line")
done < file

In case the file has an incomplete (missing newline) last line, you could use this alternative:

arr=()
while IFS= read -r line || [[ "$line" ]]; do
  arr+=("$line")
done < file

Need alternative to readarray/mapfile for script on older version of Bash

edited Jan 23 '20 at 03:25

answered Sep 14 '17 at 18:15

codeforester

34,080
14
96
122

I find it that I have to put parentheses around `IFS= read -r line || [[ "$line" ]]` for it to work. Otherwise, it works great! – Tatiana Racheva Jan 23 '20 at 02:50
1

@TatianaRacheva: isn't it that the semicolon that was missing before `do`? – codeforester Jan 23 '20 at 03:26

Jahid · Answer 3 · 2015-12-21T14:39:32.773

10

You can do this too:

oldIFS="$IFS"
IFS=$'\n' arr=($(<file))
IFS="$oldIFS"
echo "${arr[1]}" # It will print `A Dog`.

Note:

Filename expansion still occurs. For example, if there's a line with a literal * it will expand to all the files in current folder. So use it only if your file is free of this kind of scenario.

edited Dec 21 '15 at 14:39

answered Jun 22 '15 at 21:57

Jahid

19,822
8
86
102

Is there any way to set `IFS` only temporarily (so that it recovers its original value after this command), while still persisting the assignment to `arr`? – Hugues Dec 18 '15 at 20:22
1

Note that filename expansion still occurs; e.g. `IFS=$'\n' arr=($(echo 'a 1'; echo '*'; echo 'b 2')); printf "%s\n" "${arr[@]}"` – Hugues Dec 18 '15 at 21:48
@Hugues : yap, filename expansion still occurs. I will add that bit of info..thnks.. – Jahid Dec 18 '15 at 22:08
Sorry, I disagree. `IFS=... command` does not change `IFS` in the current shell. However, `IFS=... other_variable=...` (without any command) does change both `IFS` and `other_variable` in the current shell. – Hugues Dec 20 '15 at 19:39
@Hugues : You are right again, sorry about that... Fixed it with save-and-reset way. – Jahid Dec 21 '15 at 14:41
1

Thanks! This works; it's unfortunate that there is no simpler way as I like the `arr=` notation (compared to `mapfile`/`readarray`). – Hugues Dec 21 '15 at 16:43

score 7 · Answer 4 · answered Dec 14 '17 at 12:42

Use mapfile or read -a

Always check your code using shellcheck. It will often give you the correct answer. In this case SC2207 covers reading a file that either has space separated or newline separated values into an array.

Don't do this

array=( $(mycommand) )

Files with values separated by newlines

mapfile -t array < <(mycommand)

Files with values separated by spaces

IFS=" " read -r -a array <<< "$(mycommand)"

The shellcheck page will give you the rationale why this is considered best practice.

score 4 · Answer 5 · answered Jun 26 '17 at 20:04

4

You can simply read each line from the file and assign it to an array.

#!/bin/bash
i=0
while read line 
do
        arr[$i]="$line"
        i=$((i+1))
done < file.txt

answered Jun 26 '17 at 20:04

Prateek Joshi

3,743
3
38
50

1

How do you access the array? – hola Mar 30 '18 at 14:08

dosentmatter · Answer 6 · 2017-12-18T01:04:05.490

This answer says to use

mapfile -t myArray < file.txt

I made a shim for mapfile if you want to use mapfile on bash < 4.x for whatever reason. It uses the existing mapfile command if you are on bash >= 4.x

Currently, only options -d and -t work. But that should be enough for that command above. I've only tested on macOS. On macOS Sierra 10.12.6, the system bash is 3.2.57(1)-release. So the shim can come in handy. You can also just update your bash with homebrew, build bash yourself, etc.

It uses this technique to set variables up one call stack.