[lang]

Present Perfect
Personal
Projects
Packages
Patches
Presents
Linux

Picture Gallery
Present Perfect

using xargs on a list of paths with spaces in a file

Filed under: Hacking,sysadmin — Thomas @ 19:18

2011-12-30
19:18

Every few weeks I have to spend an hour figuring out exactly the same non-googleable thing I've already needed to figure out. So this time it's going on my blog.

The problem is simple: given an input file listing paths, one per line, which probably contain spaces - how do I run a shell command that converts each line to a single shell argument ?

Today, my particular case was a file /tmp/dirs on my NAS which lists all directories in one of my dirvish vaults that contains files bigger than a GB. For some reason not everything is properly hardlinked, but running hardlink on the vault blows up because there are so many files in there.

Let's see if wordpress manages to not mangle the following shell line.

perl -p -e 's@\n@\000@g' /tmp/dirs | xargs -0 /root/hardlink.py -f -p -t -c --dry-run

14 Comments »

  1. “given an input file listing paths, one per line, which probably contain spaces – how do I run a shell command that converts each line to a single shell argument ?”

    $ IFS=’
    ‘
    $ # now the shell arguments are separated by \n

    Comment by ZeD — 2011-12-30 @ 19:52

  2. I used to have the same problem… I ended up creating a “yargs” bash script that looks like this:

    #!/bin/sh
    tr ‘\n’ ” | xargs -0 “$@”

    Of course, when I visit another machine I tend to keep typing yargs instead of xargs…

    Comment by Evan — 2011-12-30 @ 20:09

  3. Hey nice solution!
    Generally when I need to handle filenames like this I use the xarg’s -I option. As an example I have the following directories structure:
    $ find | sort
    .
    ./test 1
    ./test 1/file1
    ./test 1/file2
    ./test 1/file3
    ./test 2
    ./test 2/file4
    ./test 2/file5
    ./test 2/file6
    ./test 3
    ./test 3/file7
    ./test 3/file8
    ./test 3/file9

    Using xargs like ‘ls | xargs ls -l’ doesn’t work but with -I it does works:
    $ ls | xargs -I % ls -l %
    total 0
    -rw-r–r– 1 magnun users 0 Dez 30 17:04 file1
    -rw-r–r– 1 magnun users 0 Dez 30 17:04 file2
    -rw-r–r– 1 magnun users 0 Dez 30 17:04 file3
    total 0
    -rw-r–r– 1 magnun users 0 Dez 30 17:05 file4
    -rw-r–r– 1 magnun users 0 Dez 30 17:05 file5
    -rw-r–r– 1 magnun users 0 Dez 30 17:05 file6
    total 0
    -rw-r–r– 1 magnun users 0 Dez 30 17:05 file7
    -rw-r–r– 1 magnun users 0 Dez 30 17:05 file8
    -rw-r–r– 1 magnun users 0 Dez 30 17:05 file9

    The -I % tells xargs to replace the % by the folder name, if you don’t like % you can use any strange character (@, ¬, & and etc).

    Comment by Magnun — 2011-12-30 @ 20:18

  4. There is xargs -d ‘\n’ as well.

    For some complicated commands, piping the list to `while read -r arg` can be more convenient than `xargs`, although it doesn’t have the advantage of multiple arguments per command.

    Comment by Mantas — 2011-12-30 @ 20:34

  5. Would not using “xargs -d ‘\n'” suffice?

    Comment by Joachim Breitner — 2011-12-30 @ 21:19

  6. Just use find -print0 | xargs -0

    Comment by Alexander Larsson — 2011-12-30 @ 21:44

  7. (Well, that only works if you’re generating /tmp/dirs using find, but it should be quite possible to use find to find large files)

    Comment by Alexander Larsson — 2011-12-30 @ 21:53

  8. With GNU xargs: »xargs -d ‘\n’ … </tmp/dirs« :-)

    Comment by Maxi — 2011-12-30 @ 21:56

  9. An alternative (old-fashioned) trick is to do shell escaping for each character, rather than use the NUL termination.

    sed -e ‘s/\(.\)/\\\1/g’ < /tmp/dirs | xargs /root/hardlink.py -f -p -t -c –dry-run

    Comment by CB — 2011-12-30 @ 22:14

  10. while read name ; do /root/hardlink.py … “$name” … ; done < /tmp/dirs

    Alternatively:

    xargs -d '\n'

    Comment by Anonymous — 2011-12-30 @ 23:39

  11. I have been using the following which supports spaces:

    ls -1 | while read each; do echo “$each”; done

    This handles perfectly files names with spaces. Replace echo “$each” by the command(s) that you would like to run, and $each will be expanded with the line including spaces.

    Comment by Jeremy — 2011-12-31 @ 05:06

  12. Use GNU Parallel instead:

    parallel /root/hardlink.py -f -p -t -c –dry-run :::: /tmp/dirs

    Watch the intro videos to learn more: http://nd.gd/039

    Comment by Ole Tange — 2012-01-01 @ 17:49

  13. I just noticed that some of my previous comment (#2) seems to have been eaten. The second argument to tr should be backslash zero enclosed in single quotes. Trying again (in case I ate it, not this blog):

    #!/bin/sh
    tr ‘\n’ ” | xargs -0 “$@”

    Comment by Evan — 2012-01-01 @ 21:29

  14. The command you are looking for, to convert newlines into nulls, is tr ‘/n’ ‘/0’ (replace slash with backslash.

    Comment by Rudd-O — 2012-01-02 @ 01:23

RSS feed for comments on this post. TrackBack URL

Leave a comment
picture