While creating a EC2 user data script to copy tags from an EC2 instance to EBS volumes, I got to learn a bit more about using the jq tool.

I needed to be able to transform JSON output from the aws ec2 describe-tags AWS CLI command to be usable as input for the aws ec2 create-tags command, so I can copy tags from auto scaled EC2 instances to EBS volumes.¹

The Output I Have

The JSON output from the aws ec2 describe-tags --instance-id i-xxxxxxx command, looks like:

{
  "Tags": [
    {
      "ResourceType": "instance",
      "ResourceId": "i-026ee8ba50631fe87",
      "Value": "test",
      "Key": "billing_category"
    },
    {
      "ResourceType": "instance",
      "ResourceId": "i-026ee8ba50631fe87",
      "Value": "asgtest",
      "Key": "Name"
    },
    {
      "ResourceType": "instance",
      "ResourceId": "i-026ee8ba50631fe87",
      "Value": "asgtest",
      "Key": "aws:autoscaling:groupName"
    }
  ]
}

The JSON I Need

You can get the JSON input which the aws ec2 create-tags --cli-input-json command uses , by running aws ec2 create-tags --generate-cli-skeleton. That looks like:

{
  "DryRun": true,
  "Resources": [
    ""
  ],
  "Tags": [
    {
      "Key": "",
      "Value": ""
    }
  ]
}

The Job Of The JQ Filter

I needed a jq filter which:

• Removes entries which are AWS tags starting with the string aws:, such as aws:autoscaling:groupName. Tags beginning with aws: are only able to be created by AWS services, not by customers.
• Only include the JSON keys KEy and Value, removing other JSON keys such as ResourceType and ResourceId which are invalid input to the aws ec2 create-tags AWS CLI command.

After consulting the jq manual and trial and error, I ended up with this jq filter:

{ Tags: [ .Tags[] | select(.Key | test("^aws:"; "i") | not ) | {Key: .Key  , Value: .Value} ] }

This constructs JSON in the format I want, while «selecting» only the records I want (without aws:... tags).

The Transformed JSON

The aws ec2 describe-tags JSON output now looks like this after being piped through the above jq filter:

{
  "Tags": [
    {
      "Key": "billing_category",
      "Value": "test"
    },
    {
      "Key": "Name",
      "Value": "asgtest"
    }
  ]
}

That JSON can now be fed into aws ec2 create-tags!

Tonight we had an enjoyable dinner with Elaine, my step grandmother (on my mom’s side of the family). As we dropped her off at her place, Elaine shared some of her memoirs with us. As part of her foreword, she eloquently describes that these are her memories (whether accurate, they are what she remembers), and her writings are not meant as a history nor to include absolutely everyone nor everything; «a full telling of my life would likely be quite boring.» The two chapters we got to hear, narrated live by the author, are far from boring - her writing is engaging, and her portrayal of her stories evoke emotion and a desire to keep reading. You will read these stories attentively, you will marvel, contemplate, cry, marvel, then cry again. 😂

Elaine wants to share her stories with the world, and I agree - she should. Whether this happens in print, audio, blog, or in other forms, I’ll be glad to read more, and to help her share her well crafted heart-felt collection of memories. Personally I think her narration adds a special flavor and personalization to her memories - audio book time, Elaine!

Elaine also writes about the importance of sharing our memories - things which our family and those close to us may never know, if they are never recorded. Rather than be concerned with the completeness or chronology or perfection of your writing, «just write…» She even dedicates a few pages at the back of her book, as a space to get started writing - these pages contain funny and supportive sayings along the way like «just get started,» «see, you made it this far,» and «see you’ve done it, now get a notebook and keep writing your memories…»

This experience is a great reminder of the value of writing and sharing it. It’s easy to think my writing ay be boring, instead of awesome. Writing thoughts and memories, with no particular end in mind, helps me improve - somehow I had forgotten that. Thanks for the great reminder, Elaine.

This also reminds me that I have recordings from two interview sessions with my grandmother (on my dad’s side of the family) - we are blessed to be able to continue adding to those, if we can convince or trick her into participating.

Have you, written lately…?

This year I participated in my first «AudioMo» - an online challenge to record something every day during the month of June. Record what, exactly? What ever strikes your fancy - much like what folks post to social media; your mileage may vary. :) I enjoyed hearing other AudioMoers (1), and getting to know new folks. The #AudioMo hash tag brings people together around these recordings in a pretty cool way. I’m glad I completed the challenge - making a recording every day wasn’t always something I was in the mood for, but it was worth it.

Check out my AudioMo recordings on my audioBoom page.

I treated my AudioMos like mini podcasts - rather than just talking about how many coffees I drank (2), or recorded the sound paint makes when dries, I discussed things which were both authentic and (hopefully) interesting to listeners. While I don’t have a clear idea for a podcast, nor a podcast co-host, my AudioMo experience does confirm that I would enjoy podcasting some day.

1. Are AudioMoers sort of like lawn mowers...? LOL
2. I don't drink coffee, although I do love the smell of it. That was a humorous example of the sort of benign detail I sometimes see on social media.

Yesterday I made some quick notes in Text Edit on Mac OS X, then left work without saving. Later I wanted to remotely access those notes, with only SSH access to my Mac. It turns out you can get to open but not-yet-saved Text Edit documents - they are auto-saved in ~/Library/Containers/com.apple.TextEdit/data/Library/Autosave Information as .rtf files. I think it auto-saves in RTF format to retain formatting until you actually save the document.

I can recall a lot of appreciation and praise during both my child and adult hood, for avoiding harm and minimizing risk. Everyone wants their investments and responsibilities to stay safe, whether we’re talking about one’s children, students, data, reputation, secrets, or money. I have built a lot of my career and decisions around these pillars.

Last night I listened to episode 3 of Charlie Gilkey’s «The Creative Giant Show.» Charlie starts off this podcast describing his hesitancy about releasing this show - I’m sure glad he did. Charlie and Seth Godin discuss a lot of great topics - here are how some of them positively impacted me:

• Making your own significant impact by picking yourself instead of waiting for someone else to pick you. This has prompted me to reflect on how I have allowed my progress, or gage of success, to be granted by others. In some ventures, I have been handicapping myself by giving some of my power away to other unwitting people, a lot of whom didn’t want this role anyhow.
• Goals (reaching people with a message) vs. tactics (publishing a book). I have gotten so excited about working with a particular company, or on a certain project, or using a particular tool, that I lost sight of my larger desire because of tunnel vision. Sometimes this has turned into a lot of unnecessary mental and physical work - like proverbially going on an elaborate hunting trip just because you wanted to site in a new rifle.
• Letting out the genius which exists within all of us, and expressing our art. (this isn’t the ego form of “genius,” nor the super-literal form of “art).” This one is still a bit fuzzy to me, but it does “speak to me.” My wife, one of my co-workers, and likely others, will hope that this means I will be playing piano more often. :)
• Follow, and focus on, the feeling we get before doing something great, vs. the feeling which accompanies shipping something not-so-great. I do often know the difference between these two feelings, but I frequently allow my certainty to be clouded by doubt and the input of others.
• Give yourself 24 hours and then ship something - get in the habit of shipping something every day, that’s a habit you’ll be glad you have. In fact, this blog post is something I’ve pushed to ship - ideas which I am not 100% comfortable with, and which are certainly far from complete, but still worth expressing and implementing.

This podcast has given me a burst of clarity. Circling back to the opening paragraph of my post, I will develop an even more frequent habit of shipping and taking action, and reminding myself that part of my definition of success includes a different definition of “safe” from that which I have been used to - the new “safe” is adaptable and agile and has 50 irons in the fire instead of only 4 piles in the ground. I love how Seth describes success as it relates to safety, in The Truth About Shipping: «Understand that the only thing between you and the success you seek in a chaotic world is a lizard that figures out that safe is risky and risky is safe. The paradox of our time is that the instincts that kept us safe in the day of the saber tooth tiger and General Motors are precisely the instincts that will turn us into road kill in a faster than fast internet-fueled era.”

Similarly to how  I have used results fromThe Passion Test to help me course-correct small and large life decisions to be aligned with my passions, my next step RE: what I have talked about here, is to create some helpful habit-forming reminders for myself.

I use Keyboard Maestro for automation, but also to speak information like the time, and Mac battery status.

When I upgraded my Mac to Yosemite I first dealt with the Keyboard Maestro Yosemite Accessibility workaround (accessibility in this case, referring to granting Keyboard Maestro permission to control one’s Mac).

Next, I noticed that my macros which spoke things, no longer worked. The Keyboard Maestro Frequently Asked Questions mentions this issue, and that one possible solution is to call a shell script instead of Keyboard Maestro’s internal Speak Text action:

There is a bug in the NSSpeechSynthesizer that returns isPlaying as false immediately. This means the Speak Text Action in Keyboard Maestro finishes immediately, so the text is not spoken. As a workaround, you can set a variable named “Text to Speak” to the text you want, then you can use the Execute Shell Script action with the command: say “$KMVAR Text to_Speak”

I’d rather keep using the Keyboard Maestro provided Speak Text action, presuming that this bug will be fixed. I also didn’t like the extra layer of a shell script to maintain along side my speaking macros.

It turns out that a workaround with less impact, is to follow the macro Speak Text action, with a Pause action which waits for at least the number of seconds you expect it will take to speak your text. I believe this causes the macro to continue running, while the text has a chance to be verbalized, keeping isPlaying from incorrectly returning false to Keyboard Maestro and causing the macro to exit prematurely.

I got this idea when I realized that macros which speak something and then perform another action which takes a while, were able to speak the text without being cut off. rsync script, my «starting to copy…» text is spoken.

For more information on the Speak Text and Pause Keyboard Maestro actions, see the Keyboard Maestro Actions page.

I have virtual machines in VMware Fusion which need to find one-another by hostname, such as Puppet clients talking to a Puppet master. The VMware Fusion DNS forwarder will resolve names in my Mac /etc/hosts file for my VMs, but the lookup returns an extra CName record, and then multiple NXDOMAIN responses because additional unnecessary lookups were performed.

[root@vm etc]# host -v web1.fetch.
Trying "web1.fetch"
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 41003
;; flags: qr aa; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;web1.fetch.			IN	A

;; ANSWER SECTION:
web1.fetch.		5	IN	CNAME	web1.fetch.
web1.fetch.		5	IN	A	172.16.74.31

Received 68 bytes from 192.168.78.2#53 in 21 ms
Trying "web1.fetch"
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 6379
;; flags: qr aa; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;web1.fetch.			IN	AAAA

;; ANSWER SECTION:
web1.fetch.		5	IN	CNAME	web1.fetch.
web1.fetch.		5	IN	A	172.16.74.31

Received 68 bytes from 192.168.78.2#53 in 18 ms
Trying "web1.fetch"
Received 103 bytes from 192.168.78.2#53 in 31 ms
Trying "web1.fetch.localdomain"
Trying "web1.fetch.fetch"
Host web1.fetch not found: 3(NXDOMAIN)
Received 109 bytes from 192.168.78.2#53 in 28 ms

I can sitll reach the host - it isn’t as though the DNS lookup doesn’t produce the right result in the end:

[root@vm etc]# ping -c1 web1.fetch.
PING web1.fetch (172.16.74.31) 56(84) bytes of data.
64 bytes from 172.16.74.31: icmp_seq=1 ttl=64 time=0.025 ms

--- web1.fetch ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 2ms
rtt min/avg/max/mdev = 0.025/0.025/0.025/0.000 ms

While the above technically works; I can ping web1 by name, why the extra CName in the result? Even though I looked up a fully qualified hostname with a trailing period, why the extra lookups of .fetch.fetch and .local?

A proper DNS server which both my Mac and VMs can use seems like a good idea:

• I can resolve my local DNS from both my Mac and my VMs, without having to rely on a particular VM being started. This means the solution should run on my Mac.
• The DNS server is only consulted by my Mac, when looking up hostnames in my private domain; zone, otherwise I want the normal name resolution to be used. This allows me to leverage DHCP-provided DNS servers to resolve hostnames on networks I’m visiting.
• What ever I run should be light weight - I don’t want to have to remember to stop and start something when I am working with my VMs - I only need this local DNS 30-40% of the time.

I chose Dnsmasq because it is lightweight, designed for small networks, and gives me the option to provide more customizable DHCP (meaning network boot) options to VMs. I will use the OSX DNS resolver’s /etc/resolver/* syntax, to point a private DNS domain at Dnsmasq, while leaving other DNS lookups untouched. I can still add private DNS entries to the /etc/hosts file on my Mac (which Dnsmasq reads and serves by default), or put Dnsmasq DNS entries in a separate file.

The only down-side is that I have to configure a static DNS server on my VMs, but they use static IP addresses anyway. If I wanted DHCP to configure VMs to use Dnsmasq instead of using Fusion’s DNS forwarder, I could either hack the Fusion DHCP server, or disable the Fusion DHCP service and use Dnsmasq to provide DHCP on Fusion’s private subnets.

###Install Dnsmasq I used Homebrew to install Dnsmasq. Homebrew installs software into it’s own directory, helping to avoid an installation spewing files throughout your Mac.

Install Dhsmasq using the brew install dnsmasq command.

After Dnsmasq is installed, Homebrew presents you with some commands to run in order to configure Dnsmasq to start at boot. IF you miss these commands, you can re-display this information by running brew info dnsmasq. You should use the commands which Homebrew tells you to run at your time of installation, but here is what was displayed to me:

To configure dnsmasq, copy the example configuration to /usr/local/etc/dnsmasq.conf
and edit to taste.

  cp /usr/local/opt/dnsmasq/dnsmasq.conf.example /usr/local/etc/dnsmasq.conf

To have launchd start dnsmasq at startup:
    sudo cp -fv /usr/local/opt/dnsmasq/*.plist /Library/LaunchDaemons
Then to load dnsmasq now:
    sudo launchctl load /Library/LaunchDaemons/homebrew.mxcl.dnsmasq.plist

After running the commands Homebrew tells you to, you will have a running Dnsmasq server - you can verify it is running and query it:

```ifetch@mac:~$ host google.com localhost Using domain server: Name: localhost Address: ::1#53 Aliases:

google.com has address 74.125.225.206 google.com has address 74.125.225.195 ….. and so on …..

###Choosing a Private Domain
I use a top-level domain of _fetch_ which means my hosts will be puppet.fetch, web1.fetch, Etc. Obviously .fetch is not a real top-level domain (TLD), but is alright for local use, only on my Mac. Note that if I were using this domain on a private network, beyond only my Mac, I would more closely follow the guidelines below.

The best practice when picking a private domain, is to use a sub-domain of one which you already have registered (like lab.MyCompany.com), or to use a domain under one of the reserved top-level domains in RFC 2606.

* Choose an internal domain name which will not be used on the Internet. In theory any top-level domain, like .fetch, could be used on the Internet in the future. The safest bet is to use a sub-domain under a domain which you have registered; control of.</li>
* [RFC 2606](http://tools.ietf.org/html/rfc2606) lists the top-level domains .test, .example, .invalid, and .localhost as reserved for testing. The domains example.com, example.net, and example.org are also reserved, if you really lack originality and want to use one of those.

### Configuring Dnsmasq
Edit /usr/local/etc/dnsmasq.conf as root - below are some minimal options to add.

* The `local=` option tells Dnsmasq to not send queries for this domain to any upstream DNS servers. This option is not strictly necessary because Dnsmasq will serve any entries in my Mac's /etc/hosts file, but to play it safe, we'll avoid the potential of querying the Internet for my .fetch domain by listing it here.
* I don't want Dnsmasq to listen on all of my Mac's network interfaces. This can be controlled by either specifying an explicit list of interfaces to listen on, or specifying a list of interfaces to explicitly <strong>not</strong> be listen on. IF you often change your VMware Fusion network configuration, or add/remove networks, it may be more desirable to tell Dnsmasq which network interfaces you do not want it to listen on. I told Dnsmasq to not listen on my ethernet, wireless, and VPN interfaces using multiple occurrences of the `except-interface=` option.

Here is the configuration I've added to my dnsmasq.conf file:

Queries for .fetch should never be sent to upstream nameservers.

local=/fetch/

Listen on all interfaces except wired, wireless, and our VPN.

except-interface=en0 except-interface=en1 except-interface=jnc0 except-interface=utun0

#### Other Interesting Options
Here are some other interesting Dnsmasq options you may want to look more into. Each of the bullets below lists multiple options, as they tend to be used in combination with one-another.
* `expand-hosts` and `domain` - to expand short; non-fully-qualified hostnames in /etc/hosts, into fully-qualified hostnames
* `no-hosts` and `addn-hosts` - Use a file other than /etc/hosts to supply local entries to Dnsmasq.
* `address` - map an entire domain to a single IP address, similar to defining a wild-card DNS

### Create Local DNS Entries
By default Dnsmasq will serve entries from the Mac's /etc/hosts file. To store local DNS entries in a separate file, add these two options to the dnsmasq.conf configuration file:

Do not read entries from /etc/hosts, and use a separate file instead.

no-hosts addn-hosts=/etc/hosts-fetch

Then put your private DNS domain entries in /etc/hosts, or which ever file you chose to use.

172.16.74.31 web1.fetch 172.16.74.32 puppet.fetch puppetmaster.fetch

###Restart Dnsmasq
Restart Dnsmasq, now that you have edited it's configuration file. IF killed, the launched OSX service will restart Dnsmasq. Therefore, if you need to restart Dnsmasq after editing it's configuration file, you can just kill Dnsmasq using `sudo pkill dnsmasq` and let launched restart it.

Alternatively, run `sudo launchctl stop homebrew.mxcl.dnsmasq` - this will stop Dnsmasq, and launched will promptly restart it. IF you forget the homebrew.mxcl.dnsmasq identifier, you can find it by running something like `sudo launchctl list |grep -i dns`

####Stopping Dnsmasq All Together
IF you do not want Dnsmasq to be running for some reason, you can stop it with `sudo launchctl unload /Library/LaunchDaemons/homebrew.mxcl.dnsmasq.plist`. You can then start it again with `sudo launchctl load /Library/LaunchDaemons/homebrew.mxcl.dnsmasq.plist`. IF you unload the Dnsmasq service, then reboot your Mac, Dnsmasq will start at boot.

### Point The OS X Resolver at Dnsmasq
The OSX DNS resolver will only consult Dnsmasq when a hostname in the .fetch domain needs to be looked up.

Create the /etc/resolver directory, and then create a resolver file named after your domain - in my case, fetch - pointing to Dnsmasq on 127.0.0.1.

sudo mkdir /etc/resolver sudo vi /etc/resolver/fetch

The contents of /etc/resolver/fetch should be:

‘ nameserver 127.0.0.1

The Mac will use Dnsmasq when ever I attempt to access a hostname within my private DNS domain, such as when using ping or a web browser. Using the nslookup or host command to lookup such a hostname will not work, because those commands are querying upstream DNS servers directly.

### Point VMs at Dnsmasq
Configure the DNS resolver in virtual machines, to point to the IP address of the Mac in which ever network the VM is using. This is the .1 IP address in the subnet - for example, if a VM has an IP address of 172.16.74.30, configure the VM to use 172.16.74.1 for DNS. For a VM running Linux, edit /etc/resolv.conf to look like:

domain fetch nameserver 172.16.74.1 search fetch ```

My Linux VMs happen to have a second NIC, in the host-only Fusion network, which I use to SSH to the VM from my Mac. This has an added benefit of giving me a fixed static IP address to point to for DNS, even if I have changed the VM’s first NIC between NAT and bridged Fusion networks. The two NIC approach may not be desirable for others though, and you may find that switching your VM’s NIC to another Fusion network, breaks the VM’s ability to talk to your internal DNS server.