I’m going to try to post a few howto articles for some of the simpler tasks. Many of these will be things that I use daily.
It is often necessary to read a file with bash, and act upon the entire line. There are many different ways to do this, but I’ll outline two of the simpler methods, both suitable for stacking on a single command line.
For this exercise I’ll assume the file is a list of files that we need to execute a command on.
# cat file.lst |while read line; do echo "${line}"; done /tmp/file1.txt /tmp/file with space.txt #
All we’ve managed to do here is cat the file.
Alternately we could move the cat to the end of the while loop as follows:
# while read line; do echo "${line}"; done < <(cat file.lst) /tmp/file1.txt /tmp/file with space.txt #
So we have a list of txt files. If we need to rename them all to .text we could use the following command.
# cat file.lst |while read line; do newname=$(echo ${line}|sed 's/txt$/text/'); mv -v "${line}" "${newname}"; done `/tmp/file1.txt' -> `/tmp/file1.text' `/tmp/file with space.txt' -> `/tmp/file with space.text' #
I prefer to use a -v when using a loop to perform tasks such as this. Adding the -i argument (inquire) will also ask before overwriting files. Another tip when doing something potentially destructive is to put an echo before the mv so you echo the commands instead of executing them.
You also aren’t limited to getting the list of files from a file. ls, find, and many other commands can be used to obtain the list. For example:
# find /tmp -name '*.txt' | while read line; do echo "${line}"; done /tmp/file1.txt /tmp/file with space.txt #
That’s it for tonight. If anyone has questions, requests, or suggestions, please post comments, or send me an email. My address is anton at this domain.
Thanks Anton – that helped me with a problem
You’re very welcome! It’s nice to know someone’s getting some good out of my posts.
Yep, thanks. Was useful.
Thanks Anton.
“bash read a file line by line” on google gets your page first – and it does what it says!
Thanks again.
Help me
Thanks Anton
Thanks… Helped me…
Thank you Anton,
This helped me a lot!
Thanks Anton, I am in need of this
[...] Thanks to the following link: http://anton.lr2.com/archives/2005/03/23/read-a-file-with-bash/ [...]
This is very useful. Thanx
Thanks, this pointed me in the right direction for what I was trying to do.
The problem with using cat though, is any variables assigned within the while loop are local to the subprocess that bash will execute cat in, meaning that:
VALUE=”"
cat /path/to/file | while read line; do
[ "${line%%:*}" == "Keyword" ] && VALUE=”${line#*:}”
done
echo $VALUE
doesn’t work as expected (assuming there is a line ‘Keyword: something [...]‘ in /path/to/file), as VALUE is assigned local to the while loop, as bash spawned a subprocess for cat.
A solution (there’s probably others):
VALUE=”"
exec 9
A solution (there’s probably others):
VALUE=”"
exec 9
Sorry about the multiple posts.. ‘Submit Comment’ doesn’t substitute “<” for “<”..
A solution (there’s probably others):
VALUE=”"
exec 9< /path/to/file # assign file descriptor 9 to file
while read -u 9 line; do # read from file descriptor 9
[ "${line%%:*}" == "Keyword" ] && VALUE=”${line#*:}”
done
exec 9<&- # free file descriptor 9
echo $VALUE
VALUE will be assigned local to the script, and will still be set to the value of ${line#*:} after the while loop has exited. Plus, the file is read using bash builtins only
Switch the loop around like this:
Now the while loop happens in the current process and VAR is available when the loop finishes.
This is very helpful to my problem, especially using 9 descriptor.
thanks,
Xiuping
Thanks, concise and correct info.
Thanks for the info. Good stuff, this. Sloved my problem.
while read line; do
echo $line
done < file.lst
Redirecting with a < is fine if you have a file. In most cases I am using commands to generate a list.
while read line; do
echo $line
done < <(find . name '*mp3')
Or more commonly:
find . -name '*mp3'|while read line; do echo "$line" ; done
Of course, usually the list is generated by some command, but the “cat” command is unnecesary here.
A lot of the simple line processing jobs can more easily be done by use of xargs.
For example:
cat file.lst | xargs echo
Instead of:
cat file.lst |while read line; do echo “${line}”; done
Yeah, I know this page was probably just an intro on how to read line by line from a shell script, but I wanted to point out that in many cases that is not necessary.
The purpose is to show how to read a file line by line with bash and I would epxect you change the echo to a number of useful commands.
As for using xargs:
# cat file.lst
file with spaces.txt
file2.txt
#cat file.lst | xargs echo
file with spaces.txt file2.txt
#cat file.lst | while read line ; do echo $line ; done
file with spaces.txt
file2.txt
#cat file.lst | xargs rm -v
rm: cannot remove `file’: No such file or directory
rm: cannot remove `with’: No such file or directory
rm: cannot remove `spaces.txt’: No such file or directory
removed `file2.txt’
#cat file.lst|while read line ; do rm -v “$line” ; done
removed `file with spaces.txt’
removed `file2.txt’
There are differences and reasons not to use xargs, even for simple things like echo and rm.
Hi ,
Please see this example.
The input file contents are as follows:
Lets name it as test3
ABC ABC ABC ??? + —- ##
ABC ABC ABC ??? + —- ##
ABC ABC ABC ??? + —- ##
Now I am doing a while loop to read this ,like below
cat test3|while read line
do
echo $line
done
The expected output is just like the above , however i am getting the below output
ABC ABC ABC bin dev etc lib mnt net opt svm tmp usr var vol xfn + —- ##ABC ABC ABC bin dev etc lib mnt net opt svm tmp usr var vol xfn + —- ##
ABC ABC ABC bin dev etc lib mnt net opt svm tmp usr var vol xfn + —- ##
Could you please let me know why is this so ?
Few things about this problem
1. This problem happens only when I execute this script as a cron job , Manual execution gives the proper output
2. This works fine If I remove the “???” characters from the file test3. that is ., I am getting this problem only when the input file has the character “???”
Thanks ,
Dilip
BASH is expanding the ??? to all 3 letter files (and directories) in the current directory. Crontab runs from / so that’s what it’s matching. When I run it in my home dir I see bin, log, and src dirs.
This is the same mechanism that allows rm ??? to remove all 3 letter files.
Wrap the $line in “s to prevent bash from doing the replacement.
cat tmp.txt | while read line ; do echo “$line” ; done
That was quick simple and efficient. Thanks, Anton!
Hi,
Here all examples for “while read” are mentioned with a variable like “line”. My question is what variable should we check for the same while loop which has no variable. Example:
while read
do
here a progress bar is generated
done
I need to log the output in text file too.
your help will be appreciated greatly.
Amit
Amit,
read, without a variable, uses the default of $REPLY.
I would recommend setting the variable though just to prevent confusion when you attempt to nest a while read inside another.
Thank you Anton and everybody for messages,
i want make bash file.
everything is allright but i want get variable text other file. I can’t make this.
how can make this?
Erdem,
I’m not entirely sure I understand your question. I’m assuming that you want to place the contents of a file into a variable. You could do this by:
MYVAR=$(cat file.txt)
but i have 12 variable and i want read only one file.
Erdem,
Sorry, I don’t fully understand your question, can you try to rephrase it?
Thanks,
Anton
Anton,
— THIS IS VARIABLE FILE NAME download.txt —
prg_1_name=”Anton’s Blog”
prg_1_file=”http://……./anton.tar.gz”
prg_1_fname=”anton.tar.gz”
prg_1_dname=”anton”
etc.
.
.
i want get this file contents when bash file is execute..
how can make it?
Hey
I needed a refresher on how to read line by line from a file using a loop. I just wanted you to know that your post was extremely helpful, thank you!
Erdem,
If you have a script that’s got commands in it (like setting variables) then you can run it by using ‘.’.
Something like:
#!/bin/bash
. ./config.sh
echo $prg_1_name
damn. Because of such tricks people will never learn perl.
Thanks for a great and simple guide.
Thanks, just what I needed.
Hi Anton:
I’m usually always on windows, but today, I got a shell account from a friend, and although I knew bits and pieces about *nix and shell scripting because of being a web dev, you really helped point me in the right direction here.
Thanks,
Curtis
Hi Anton,
The piece of code on this page helped me in a lot of ways.
Thanks a lot.
Praveen
hey anton,
hope all is well, i have a question that i am sure you can answer in a heartbeat… i have a command that is returning two lines, essentially, i want to take each of those lines and make a variable out of them, how do i do that with bash?
Great help. Thanks!
I’m accessing ‘http://www.random.org/cgi-bin/checkbuf’
It returns the single-line file ‘checkbuf’ with these contents”
4%
In this case I wouldn’t want to request a number of random bytes.
How do I check the number (note it’s 4%, not 004%) and do the Right Thing?
Ted
Ted,
If you get the 4% in a var, then you could always just split in on %, take the first part, and use a regular if to handle it.
X=$(curl -s -o – http://www.random.org/cgi-bin/checkbuf)
if [ "${X%\%*}" -get 10 ] ; then
echo “X is more than 10%!”
fi
Anton,
Love the tutorial, great stuff and very helpful. I do have a question however. Suppose I am wanting to read specific contents within a text file like name, address, phone, etc… and put it into another file to archive. How do I read the file using Bash and make it look for keywords to recognize what I need it to read? (hope that makes sense) =)
hi,
what if file names with more than one whitespace are listed?
m
/tmp/file with space.txt is listed above and it has more than one whitespace.