In rocket science they have this notion called Delta-V (literally, change in velocity). It's basically the most useful number in orbital mechanics, or at least it is to those of us who learned orbital mechanics by way of playing kerbal space program.
ANYHOW, the reason delta v is really useful is that it's this one number that basically lets you know what you need to do to get there. Are you on earth and you want to go to space? Then you need to generate a delta V of 4600. Are you in orbit, but you want to go to higher orbit? Then you need a Delta V of 350, and so on. Delta V gives you a singular goal. There are myriad ways to achieve it, balloons, solid-fuel booster rockets, space planes etc.. but that number is the goal.
I've often wished we had a DevOps Delta-V, like, given where I am right now, what Delta-V do I need to generate to get to continuous integration? If we ever came up with such a thing, I think we operations folks could more or less define our entire careers in terms of it, because we're are all more or less perpetually locked in a struggle to generate the Delta-V necessary to achieve HA infrastructure of one sort or another, even as the current infrastructure collapses around us.
Over time, some of our techniques become less effective, or maybe new techniques come along that are so much more effective that the old ones only seem diminished by comparison. In case you hadn't noticed, we're currently living through the SaaS Web-API Renaissance. The Delta-V I could generate by gluing together command-line tools 10 years ago, seems pitiful compared to the Delta-V I can generate by gluing together Web-API's today. For the most part, this is fine. Most of my tools (Python, Ruby et al..) work fine in either context -- there's no need for me to re-tool. We can use the same fuel, only burning it in a more efficient engine. All that is, except for Shell.
Shell is a fantastic lowest-common-denominator language. Yes it's ugly and utilitarian, and it doesn't have concurrent hashi generators, but it's always there, this handy little ever-present force multiplier. Waiting right there at my fingertips to automate away this or that. I've probably written orders of magnitude more shell than any of these other languages I profess to adore. There's a mind boggling array of Fortune-500 business processes I don't even remember writing in shell that are still chugging along all these years later. It comes back to haunt me now and again, over beers with increasingly grey-headed compatriots from here or there. Anyway it'd be a shame to see Shell become irrelevant in my day-to-day problems.
But the JSON! Generating Delta-V by working with API's requires working with JSON. Parsing it, Extracting it, transforming it; the JSON is everywhere -- ubiquitous. unavoidable. And Shell just hasn't had a very good answer for the question of JSON.
Until, that is, jq, a fast, lightweight, flexible, CLI JSON processor. JQ stream processes json like awk stream processes text. JQ, coupled with cURL has me writing Shell to generate Delta-V with Web-API's, which is pretty great. It helped me write shellbrato, a shell library for the Librato API as well as myriad other little tools that I use day to day for things like looking up PR's assigned to me via Github API's and resolving AWS Instance tags to IP's via the AWS API.
Anyway I think JQ is pretty boss, so I thought I'd share what I've learned so far about writing IT-Pro-grade shell scripts that interact with WebAPI's with cURL and JQ.
I'm sure you're familiar, but cURL is one of the two popular command-line "web browswers" out there. The other being wget. I prefer wget for ad-hoc manually downloading stuff, and cURL for programming; we can argue about that over beers whenever you want.
For writing scripts there are only a few things to know:
- -k turns off ssl
- --max-time Use this to control the entire length of time a transfer can take (usually something like 300-500 seconds)
- --connect-timeout Use this to control connect time-outs (usually something like 3-5 seconds)
- --silent squishes extraneous output leaking into stderr
- -A sets a user-agent which is very polite and also metricky
JQ, as I've already mentioned stream-processes json. You can pipe standard-in to it, or provide a filename full of json as the last argument. In this introduction, we'll use both of these methods.
In jq syntax, . is the input. Here, try this:
echo '[][]' | jq '.'
See? Dot is the current input, except as you can see, jq reformats JSON for us.
So if you have a huge blob of json like the blob inside in.json, and
you run it through jq, and ask for it back using ., you'll get a nicely
formatted version.
jq '.' in.json
.[] is whatever's inside the input. Here try this:
echo '["foo"]' | jq .[]
See? Those brackets basically unwrap the top layer of the input. In jq you
can always ask for either the thing by just naming it (thing), or for whats
inside the thing by adding some unwrappy brackets (thing[]). You can add
more unwrappy brackets to unwrap more layers:
echo '[["foo"]]' | jq .[][]
All of this works, even if there's more than one top layer thingie in the input:
echo '["foo"]["bar"]' | jq '.'
echo '["foo"]["bar"]' | jq '.[]'
echo '[["foo"],["bar"]]' | jq '.[]'
echo '[["foo"],["bar"]]' | jq '.[][]'
In jq . and .[] (and many other things) are referred to as filters. In jq
every filter has an input and an output, and you can pipe the output of one
filter into the input of another filter just like in shell:
echo '[["foo"],["bar"]]' | jq '.|.[]|.[]'
Jq has lots of filters, and at first glance, many of them will seem useless and silly but later they will seem atom-bomb powerful. For example, any string literal works as a filter that completely ignores it's input and outputs itself:
echo '[["foo"],["bar"]]' | jq '"best filter ever"'
Why on earth would you ever want a filter that ignores the input and outputs
itself? Well let me introduce you to the , operator; it tee's the input to
whatever it separates. We can use the comma and string literals together to
make record separators:
echo '["foo"]["bar"]' | jq '.,"************"'