Parameter expansion to loop over pairs: ${a%b} and ${a#b}

[karmag]

Hey, a command line community? That's a pretty neat thing to have.

So to introduce myself, here's something I figured out pretty recently. It's in one of those eyes-glazing-over parts of the shell man page (but I had a specific problem to solve so I had to figure it out). Behold! It is a piece of "parameter expansion" magic:

$ TEST='foo:bar' ; echo ${TEST%:*} ${TEST#*:}
foo bar

Ooh, exciting! No wait, it's not. But here's the problem I had: How can I loop over pairs of values in a script or one-liner?

The solution I came up with looked something like this:

$ for f in url1:filename1 url2:filename2 url3:filename3;
do URL=${f%:*}; FILENAME=${f#*:} ;
...
; done

...and so for each pair, URL and FILENAME gets set to their respective portion. (And you'll be glad to learn that I got my Youtube-mediated crime drama fix in the end.)

Comments

[foxfirefey]

Ooooh I did not know this and it is useful!

[jld]

Other thing: #blah and %blah will take off the shortest match, while ##blah and %%blah will remove the longest. Also, all of these just give the unchanged value if there's no match.

This can be useful for things like ${foo##*/}, or for generalizing those pairs to more things.

[karmag]

I just wanted to share the "pairs" trick, but yeah, there's certainly more you can do, and with the ${} construct more generally too. (did someone say "man sh"?)

On a side note, parameter expansion is apparently one of the areas where bash differs from plain sh, and not everything you can do will be portable. This kind of matching (both shortest and longest match) is, though.

[karmag]

That is, for those of you here who has bash installed as the default shell. I shouldn't make the sloppy assumption that everyone reading this has...

[pne]

In any case, when using bash-isms, use a shebang of #!/bin/bash - some people use #!/bin/sh under the assumption that everyone uses (certain distributions of) Linux /bin/sh is bash.

If you want bash, say so explicitly, then you won't be bitten on systems where /bin/sh is the original sh, or something else like the Almquist shell (ash/dash).

[karmag]

I (mostly) agree. But on the other hand you could argue that doing so will make perfectly cromulent scripts fail for users that don't have bash at all. (Or don't have bash in /bin, like me.) That's still better than failing silently and doing the wrong thing, though.

A related question is how you can know that you're using a bashism in the first place. It seems that the man page doesn't point out what are extensions and what is POSIX – so if your /bin/sh is bash, how will you know? And to make things more exciting, when started with the --posix flag bash still accepts (at least some of) its own extensions, so you can't use that for testing. (But I don't have a very recent version, so perhaps this is no longer a problem.)

I guess the only half decent way out of this is to install a bunch of different shells (ash, ksh, zsh, ...) and test anything you don't already know is portable. Yay, more work for everyone.

For reference and/or comparison, here is the /bin/sh man page for my system. I don't think there are many that can beat NetBSD when it comes to being conservative and boring, so perhaps it can be useful in setting a lowest common denominator. :-)

[afuna]

Oh hmm!

*tucks it into her toolbox for future reference*