Bash Xargs Command the Hard Way by Example

When you just start to learn bash programming the hard way, no one tells you about the xargs command. Since then, you've managed to pick up enough xargs to get by without breaking too much. Now, you find yourself asking ― What about all those other options? Can I do this? What if I want to… ?

By now you probably have a good enough understanding of how to execute command in bash. But what if you want to execute streams of commands in sequence or sometimes in parallel? That is where we find ourself using xargs.

Here we hope to answer all these questions and more about bash and the xargs command by example.

What is xargs in bash?

xargs is an external command used to convert standard input to command line arguments that is said to stand for “extended arguments.” It was mainly created for use with commands not built to handle piped input or standard input such as rm, cp, echo ¹ and other external commands only accepting arguments as parameters.

¹ Although most systems come with an echo command, echo is a bash built-in; That is unless calling command echo, the built-in echo is used. Similarly, bash builtins are not aware of piped input.

Xargs options with bash examples

Let's run through xargs and its options with examples in bash. Along with the conventional treatment for command line options expected with xargs, the options are grouped by objects such as getting info or modifying behavior.

Xargs info

Here are options providing information about xargs.

Xargs help

xargs --help
Usage: xargs [OPTION]… COMMAND [INITIAL-ARGS]…
Run COMMAND with arguments INITIAL-ARGS and more arguments read from input.
Mandatory and optional arguments to long options are also
mandatory or optional for the corresponding short option.
-0, --null                   items are separated by a null, not whitespace;
disables quote and backslash processing and
logical EOF processing
-a, --arg-file=FILE          read arguments from FILE, not standard input
-d, --delimiter=CHARACTER    items in input stream are separated by CHARACTER,
not by whitespace; disables quote and backslash
processing and logical EOF processing
-E END                       set logical EOF string; if END occurs as a line
of input, the rest of the input is ignored
(ignored if -0 or -d was specified)
-e, --eof[=END]              equivalent to -E END if END is specified;
otherwise, there is no end-of-file string
-I R                         same as --replace=R
-i, --replace[=R]            replace R in INITIAL-ARGS with names read
from standard input; if R is unspecified,
assume
-L, --max-lines=MAX-LINES    use at most MAX-LINES non-blank input lines per
command line
-l[MAX-LINES]                similar to -L but defaults to at most one non-
blank input line if MAX-LINES is not specified
-n, --max-args=MAX-ARGS      use at most MAX-ARGS arguments per command line
-P, --max-procs=MAX-PROCS    run at most MAX-PROCS processes at a time
-p, --interactive            prompt before running commands
--process-slot-var=VAR   set environment variable VAR in child processes
-r, --no-run-if-empty        if there are no arguments, then do not run COMMAND;
if this option is not given, COMMAND will be
run at least once
-s, --max-chars=MAX-CHARS    limit length of command line to MAX-CHARS
--show-limits            show limits on command-line length
-t, --verbose                print commands before executing them
-x, --exit                   exit if the size (see -s) is exceeded
--help                   display this help and exit
--version                output version information and exit

Refer to xargs help as a quick reference to xargs usage and options.

Xargs version

xargs --version
xargs (GNU findutils) 4.6.0

Xargs limits

Even xargs has its limits. The -show-limits option for xargs shows limits used by xargs prior to running the commands. The actually limits depend on your environment. However, for most users, this is enough. The limits may be adjusted at the command line, see examples.

Example) Your environment xargs limits

xargs --show-limits
Your environment variables take up 6234 bytes
POSIX upper limit on argument length (this system): 23718
POSIX smallest allowable upper limit on argument length (all systems): 4096
Maximum length of command we could actually use: 17484
Size of command buffer we are actually using: 23718
Maximum parallelism (--max-procs must be no greater): 2147483647

Execution of xargs will continue now, and it will try to read its input and run commands; if this is not what you wanted to happen, please type the end-of-file keystroke.

Warning: echo will be run at least once. If you do not want that to happen, then press the interrupt keystroke.

Note that the command that would run as a consequence of xargs is echo, the default command of xargs.

Example) Xargs limits with an adjusted command buffer limit

xargs --show-limits -s 1
Your environment variables take up 9479 bytes
POSIX upper limit on argument length (this system): 20473
POSIX smallest allowable upper limit on argument length (all systems): 4096
Maximum length of command we could actually use: 10994
Size of command buffer we are actually using: 1
Maximum parallelism (--max-procs must be no greater): 2147483647
…
Warning: echo will be run at least once. If you do not want that to happen,
then press the interrupt keystroke.
xargs: cannot fit single argument within argument list size limit

Note that errors will appear on the bottom following Warnings if-any. We have a ” xargs: cannot fit single argument within argument list size limit” error, which just means that we are attempting to work outside of the allowed command buffer size which is set to a character.

The command buffer contains the command followed by all arguments including spaces.
In the case of the command in this xargs option example, the command buffer is

"echo"

which contains 4 characters.
So, we need to set the command buffer size to a value greater than or equal to 5 as follows. Note that command buffer consumption will equate to length_of_command + length_args_including_spaces_plus_one + 1.

xargs --show-limits -s 5

# No more "xargs:cannot fit single argument within argument list size limit" error

But what if our command has args?

yes | xargs -t --show-limits -t -s 6 # will run with the following output
echo y
…

Xargs verbose

lhs | xargs -t other_xargs_options_if_any | rhs

The -t option can be used to show commands run by xargs as output to fd2, standard error. That is xargs -t can be nullified by redirecting standard error to /dev/null as follows.

xargs -t 2>/dev/null

Example) Yes once

yes | head -n 5 | xargs -t true
true y y y y y

Example) Yes 5 times

yes | head -n 5 | xargs -t -I true
true y
true y
true y
true y
true y

Xargs behavior

No command is complete without options to modify runtime behavior. Xargs is no different. Here are the options that allow you to change its behavior.

Xargs null

lhs | xargs -0 other_xargs_options_if_any | rhs

The -0 option can be used to tell xargs to use null instead of whitespace. It also disables quotes and escape sequences.

Xargs null use case

The intended used for xargs null is for handling cases such as when items contain spaces such as files containing spaces or newline characters.

Suppose that you have a directory “a b c” including spaces in a directory name.

ls "a b c"
de/ fg/ h/ 'i j k l'/

You want to run a command on each directory in “a b c” using the find command.

You may try the following:

find "a b c" -type d | xargs du -d 0 -h
du: cannot access 'a': No such file or directory
du: cannot access 'b': No such file or directory
du: cannot access 'c': No such file or directory
du: cannot access 'a': No such file or directory
du: cannot access 'b': No such file or directory
du: cannot access 'c/de': No such file or directory
du: cannot access 'a': No such file or directory
du: cannot access 'b': No such file or directory
du: cannot access 'c/fg': No such file or directory
du: cannot access 'a': No such file or directory
du: cannot access 'b': No such file or directory
du: cannot access 'c/h': No such file or directory
du: cannot access 'a': No such file or directory
du: cannot access 'b': No such file or directory
du: cannot access 'c/i': No such file or directory
du: cannot access 'j': No such file or directory
du: cannot access 'k': No such file or directory
du: cannot access 'l': No such file or directory

It works incorrectly because our directory names are riddled with spaces. This was not your intention.

You can fix this by adding xargs replacement, i.e -I as follows.

find "a b c" -type d | xargs -i du -d 0 -h
0       a b c
0       a b c/de
0       a b c/fg
0       a b c/h
0       a b c/i j k l

It works correctly when using xargs replacement. Note that we used -i, which is shorthand for -I .

Another way we can achieve the same result is using xargs null, -null, in combination with the find -print0 option as follows.

find "a b c" -type d -print0 | xargs --null -i du -d 0 -h
0       a b c
0       a b c/de
0       a b c/fg
0       a b c/h
0      a b c/i j k l

Great! Now we have more than one way to pummel our way deep into the filesystem universe without having to worry about colliding with space junk our journey. Right on.

Xargs interactive

Maybe you don't trust xargs to run all the commands without confirmation. In that case, xargs interactive or -p is the option that you need to control which commands are run by xargs as follow.

find "a b c" -type d -print0 | xargs --null -i -p du -d 0 -h
du -d 0 -h a b c ?…
du -d 0 -h a b c/de ?…
du -d 0 -h a b c/fg ?… y
0 a b c/fg
du -d 0 -h a b c/h ?… Yes
0 a b c/h
du -d 0 -h a b c/i j k l ?… no

Here any command starting with 'y' or 'Y' is run. Otherwise, commands are ignored.

Xargs file

You already have a file, arg-file, ready to be read into xargs. Your program may be waiting around a directory somewhere for someone else or another instance of yourself to drop in an arg-file. In this case, you can specify the file as an option to xargs using -a arg-file instead of having to use cat file | xargs… Xargs file examples follow.

Just for fun, let's turn your desktop cleanup routines into an arg-file that we can use.

ls cleanup-desktop | tee arg-file
190605_cleanup_desktop_files_sh.txt
190605_cleanup_desktop_lnk_files_sh.txt
190605_cleanup_desktop_un_files_sh.txt

Each file contains a routine that can be run using bash. That means the command we will be using is bash.

Let's run the cleanup routines using xargs.

xargs -a arg-file -i -P 99 bash -c ' echo ; . cleanup-desktop/ ; '
190605_cleanup_desktop_files_sh.txt
190605_cleanup_desktop_lnk_files_sh.txt
190605_cleanup_desktop_un_files_sh.txt

And it works!
Just in case we need to specify an arg-file instead of using piped input, the xargs file option comes in handy.

Xargs replace

lhs_if_any | xargs -i other_args_etc | rhs_if_any

Last but not least, xargs replace -i allows you to take full control over the command format before it is run. Any character may be used. However, following convention most bash programmers use this or this %. Note that the default is . -i tells xargs that you will be used the default. And it is considered shorthand. -I followed by a replacement character of your choice tells xargs what character you will be using. Avoid using common characters like the letter a. That will break your code more than any space or newline ever did.

Xargs parallel

lhs_if_any | xargs -P n_ge_1 other_args_etc | rhs_if_any

Xargs parallel -P allows commands to be run concurrently instead of in sequence. Refer to xargs limits -show-limits for valid arguments for n_ge_1 or concurrency. For example, if

Maximum parallelism (--max-procs must be no greater): 2147483647

You can set -P 2147483647 without any errors. In practice, you may find a nicer setting like -P 99 that improves overall performance without increasing overhead to manage concurrent processes.

Examples showing how using xargs parallel may improve performance follow.

Example) Counting in sequence versus parallel using xargs

Let's see what happens when we time counting in sequence using xargs.

time echo 1… 1000 | xargs '-d ' -i bash -c 'echo '
…
998
999
1000
real    1m13.927s
user    0m6.994s
sys     0m15.184s

Now see what happens if we time counting in parallel using xargs.

time echo 1… 1000 | xargs -P 200 '-d ' -i bash -c 'echo '
…
998
999
1000
real    0m13.554s
user    0m6.446s
sys     0m14.293s

Significant improvement in performance is observed using xargs parallel to run simple commands with no shared resources.

Example) Comparing the order and timing of xargs parallel

Let's see what happens when a command consumes CPU time.

time echo 1… 10 | xargs '-d ' -i bash -c 'sleep $(( $RANDOM % 2 )) ; echo '
1
2
3
4
5
6
7
8
9
10
real    0m5.601s
user    0m0.180s
sys     0m0.334s

Note that all commands are complete in order.

Now see what happens when the same command is run in parallel.

time echo 1… 10 | xargs -P 10 '-d ' -i bash -c 'sleep $(( $RANDOM % 2 )) ; echo '
3
4
6
7
8
1
2
5
9
10
real    0m1.257s
user    0m0.060s
sys     0m0.225s
Commands 1, 2, 5, and 9 went to sleep. However, we were able to reduce the
time to complete by as much as 78 percent.

Xargs Parallel Conclusion

Simple adding xargs parallel as a command line option can improve performance tenfold. However, you should proceed with caution when using on order dependent procedures or when commands are sharing resources.

Xargs delimiter

lhs_if_any | xargs '-dc' other_args_etc | rhs_if_any

Xargs delimiter -d allows you set the item separator to any character c the same way delimiter characters are set in the cut command.

By default -dc is set to whitespace created by the newline character -d\x0a.
When using xargs null -0, -dc is set to the null character -d\x00.

For example, you may set the delimiter to the space character, i.e. -dc is '-d ' in a command line or your bash script.

You may set the delimiter to the comma character, i.e. -dc is '-d,'.

The delimiter option in xargs -d allows you to set the item separator to any character you wish.

Bash xargs examples

Here we cover example uses of the xargs command in bash including example use in the command line as well as scripts.

Bash xargs command examples

Here we cover command line example uses of the xargs command in bash including example use with and without piped input.

Example) Make your own input: fun with xargs without input

What does xargs do home alone?

xargs
Hello, Is anyone home?
…
(Ctrl-D)
Hello, Is anyone home?…

It appears we got our question back as an answer but it appears to be just an echo.

Why?

As you may have read in what xargs is all about, it converts standard input to command lines arguments. If no options and arguments are provided it behaves like a pipe aware echo command. That is:

xargs
Hello, Is anyone home?
…
(Control-D) 2

Produces the equivalent echo command line expression implicitly

echo Hello, Is anyone home?…
2 In a script, heredoc may be used as follows.
xargs << EOF
Hello, Is anyone home?
…
EOF
echo Hello, Is anyone home?…

Example) Use xargs as a placeholder for interactive pipes

Using xargs on the left-hand-side of a pipe is paradoxical⁴so let's run bash in super restricted mode⁵.

⁴ Pipe aware commands don't need xargs. Pipe unaware commands don't know about xargs

⁵ Restricted mode that resets every line. Other restrictions may be added later.

xargs -I bash -cr ""
i=1
echo $i
echo Hello!
Hello!
!!
bash: !!: command not found
i=1; echo $i
1
cd…
bash: line 0: cd: restricted

Example) Use xargs as a placeholder for interactive arguments

According to the 2019 HackerRank Developer Skills Report³, “Calculators are the new games.” More developers under 38 are pushing calculators as their first coding project. ³ Insights based on 71,281 developers

So, let's build a calculator using xargs!

_() echo $(( "$@" )) ; # calculator
while :
do
_ $( xargs )
done
1 + 2 + 3 + 4
(Ctrl-D)
10
1 - 2 + 3 - 4 + 5
(Ctrl-D)
3
1**2+2**2
(Ctrl-D)
3
1+
2+
3+
4+
5
(Ctrl-D)
15

Example) Static site generator

Suppose that you have a few thousand plain text files that you wish to use to generate a static site for and none of the files is named index. File names include lowercase ascii characters and a hyphen if any.

Here is what a line or two in the terminal of a theoretical machine running bash would look like. The machine will have other external commands including findutils and pandoc. You may use any equivalent command of your choice.

# we are in the directory
# see lots of files

test -d "html" || mkdir -v $_
find . -mindepth 1 -maxdepth 1 -type f \
| xargs -P 6000 -i bash -c "echo ; cat | sed -e 's/$/ /' |
pandoc -thtml -o .html"

# see twice as many files now including html files
# done

Bash xargs script examples

Example) Use xargs to generate square matrices

Here's a script I cooked up to generate square matrices using xargs. Specifically, it takes advantage of the behavior using the -n option and uses the seq command to sequences of numbers to be used in matrices.

#!/bin/bash
## square-matrix
## - generates square matrices
## version 0.0.1 - initial
##################################################
square-matrix-help()

cat << EOF
square-matrix
1 - order
EXAMPLES
> square-matrix 1
1
> square-matrix 2
1 2
3 4
> square-matrix 3
1 2 3
4 5 6
7 8 9
EOF

square-matrix() local -i order ; order=$1 ;
test "$order" || $FUNCNAME-help ; return ;
test $order -gt 0 || $FUNCNAME-help ; return ;
_()
seq $(( $1 ** 2 )) | xargs -n $1

_ $order

##################################################
if [ ! ]
then
true
else
exit 1 # wrong args
fi
##################################################
square-matrix $@
##################################################
## generated by create-stub2.sh v0.1.2
## on Wed, 29 May 2019 13:44:06 +0900
## see
##################################################

Source: square-matrix.sh

I have also included a test program to show the script in action, generating all square matrices up to 10 x 10.

#!/bin/bash
## test-square-matrix
## - generates square matrices up to 10 x 10
## version 0.0.1 - initial
##################################################
test-square-matrix()
test -f "square-matrix.sh"
. $_ 1>/dev/null
local i
for i in 1… 10
do
echo "square-matrix($i)"
square-matrix $i
done

##################################################
if [ $# -eq 0 ]
then
true
else
exit 1 # wrong args
fi
##################################################
test-square-matrix
##################################################
## generated by create-stub2.sh v0.1.2
## on Wed, 29 May 2019 13:40:08 +0900
## see
##################################################

Source: test-square-matrix.sh

Here is what to expect:

bash test-square-matrix.sh | head
square-matrix(1)
1
square-matrix(2)
1 2
3 4
square-matrix(3)
1 2 3
4 5 6
7 8 9
…

Exercise: Improve the display in the terminal by applying padding to numbers

When we try to generate the square matrix of order 10 by 10, we get the following output:

bash square-matrix.sh 10
1 2 3 4 5 6 7 8 9 10
11 12 13 14 15 16 17 18 19 20
21 22 23 24 25 26 27 28 29 30
31 32 33 34 35 36 37 38 39 40
41 42 43 44 45 46 47 48 49 50
51 52 53 54 55 56 57 58 59 60
61 62 63 64 65 66 67 68 69 70
71 72 73 74 75 76 77 78 79 80
81 82 83 84 85 86 87 88 89 90
91 92 93 94 95 96 97 98 99 100

As an exercise extend square-matrix.sh to allow for output as follows.

bash square-matrix.sh 10
001 002 003 004 005 006 007 008 009 010
011 012 013 014 015 016 017 018 019 020
021 022 023 024 025 026 027 028 029 030
031 032 033 034 035 036 037 038 039 040
041 042 043 044 045 046 047 048 049 050
051 052 053 054 055 056 057 058 059 060
061 062 063 064 065 066 067 068 069 070
071 072 073 074 075 076 077 078 079 080
081 082 083 084 085 086 087 088 089 090
091 092 093 094 095 096 097 098 099 100

Bash xargs practical use examples

Example) Search files for a pattern using xargs grep

list-files¹ | xargs grep -e pattern

¹ list-files is a command returning candidate paths to file to be taken as input to grep through the xargs command

As a practical use example of xargs in bash, I dug up xargs from a secret code base.

find -type f -name \*.sh | wc -l
994

There are 994 bash scripts. Let's see how many the xargs command.

find -type f -name \*.sh | xargs grep -e xargs lists all occurrences
of xargs in the codebase.
…
find -type f -name \*.sh | xargs grep -e xargs | wc -l
18

There are 18 matches for xargs in the codebase. Now we might want to figure out how many scripts use xargs.

Great! There are 10 scripts with xargs in the codebase. Let's see what they are.

find -type f -name \*.sh | xargs grep -e xargs | cut '-d:' '-f1' | sort | uniq
vim $( !! )

Summary of findings

- Search and destroy files containing a pattern in the file name

find $path -type f -name \*$pattern\* | xargs rm -vf

- List information about files on lhs of pipe

find-files | xargs ls -al

- Make files executable

find-files | xargs chmod +x

- List file dirnames

find-files | xargs -I dirname ""

- Purge all

paths-to-purge | xargs rm -rvf

- Zip files

find-files-to-zip | xargs -I archive-$( date +%s )

- List file basenames

find-files | xargs -I basename

Example) Install software from a list using apt-get in Ubuntu

When upgrading Ubuntu, after backing up your system you may have to install new software. Suppose you have a list of software to install using apt-get in Ubuntu.

sudo dpkg --get-selections | grep '[[:space:]]install$' |
\awk 'print $1' > install_software
#…
cat install_software | xargs sudo apt-get install

Example) Curl seige using xargs in bash

Suppose you have a bunch of urls leading to a single node somewhere on the internet and you would like to lay seige using curl in bash. Preferably, this is one of your nodes and the seige is not in your production environment. Here is how we lay seige using xargs in bash.

#declare -f filter
filter ()

grep -o -e 'loc[^<]*' | cut '-d>' '-f2'

get-arg-file()

curl https://linuxhint.com/sitemap.xml --silent \
| filter \
| xargs -i curl --silent \
| filter \
| xargs -i echo > arg-file

#declare -f curl
curl ()

echo (fake) $FUNCNAME "$@"

declare -xf curl
payload()

test -f "arg-file" || return
xargs -P 1000 -a arg-file -i echo curl | bash 1>/dev/null

seige()

test -f "arg-file" || get-$_
payload

seige

Bash xargs debug

When you sit in front of the terminal you are the boss. However, when something goes wrong it helps if you know how to actually debug bash scripts like a boss.

Playing it safe, having a method to validate success when using xargs in bash rather than blindly expecting everything will be okay is highly recommended. That is, you have to make sure that all commands complete with success and failure are not left unchecked.

Methods

Size of standard error output
If standard error contains 1 or more characters, something went wrong
Sum of command exit codes
If the sum up exit codes is greater than 0, something went wrong
Payload validation
If a piece of the payload is missing, something went wrong
End of script validation
Running xargs commands as a script, if the end of script is not reached something went wrong. Note that errexit is set and commands are run from inside a function.
Other method
If the outcome differs from what is expected, then something may have gone wrong

Example) Debugging xargs using size of standard error output

Here is an anonymous function that we test debugging xargs using stardart error.

# declare -f _, i.e. I didn't write the code below
# if you find that you are a little rusty on declare I wrote another tutorial on
#
how the declare command works in bash
_ ()

rm -vf errout;
touch $_;
echo 1… 10 | xargs -x -P 10 '-d ' -i bash -c "test $(( $RANDOM % $1 )) -eq 0 ||
echo 1>&2 ; exit ; ; echo " 2> errout;
test ! $( wc < errout -c ) -gt 0 || echo something went wrong…

## test
_ 1 # probabilty of failure (=1-1/1=0%)
_ 2 # probabilty of failure (=1-1/2=1/2=50%)
_ 3 # probabily of failure (=1-1/3=2/3=60%)
…

If you don't use standard error for anything else using the size of standard error to debug xargs is one method that may work for you.

Bash xargs functions

Sometimes what you want to do is use functions that you have defined in xargs. To do that we need to make the function available to xargs. Here is how.

#declare -f _
_ ()

echo $@^^

echo a… z1… 9 | xargs '-d ' -i bash -c "_ "
bash: _: command not found
…
declare -xf _
echo a… z1… 9 | xargs '-d ' -i bash -c "_ "
A1
A2
A3
…
# or
echo a… z1… 9 | xargs '-d ' -i echo "_ " | bash
…
Z7
Z8
Z9

Note that the above interactive session example can be sped up using bash xargs parallel.

Conclusion

Xargs is one of the many external commands that you are better off knowing in bash. In writing this guide on the xargs command, I learned a few extra options myself. Review every so often is recommended. Only then may you be able to use xargs to its true potential. Until then, code on.