Fred Clifts Blog

Two Little-Known KVM Options Save the Day

2019-07-05T22:00:00-06:00

Auto Restart KVM VMs while forcing KVM snapshot reversion

(this is mostly notes for my later self - feel free to drop me a line to ask questions)

TLDR My problem:

I have some stateless throwaway vm instances which discard all changes every power-off. Also, the closed-source software in the vm has driver issues that occasionally require a reboot, detected from inside the vm and initiated as an OS reboot. When rebooting, I want to simulate power-off/power-on to trigger the KVM snapshot discard. And, I want to make sure that any vm that shuts down gets restarted. How? Obscure qemu-kvm options and libvirt hooks to the rescue - special use for -snapshot and -no-reboot.

More detail:

I use libvirt (virsh) to manually control some transient, sort of stateless KVM virtual machines. I need to periodically stop/start those vms automatically (from inside so I can do a clean service shutdown). Also I want to auto-restart particular vms any time they shut down. The VMs run kvm-qemu snapshots (-snapshot option) that throw away all system changes at every power-off.

I have a close-source application that occasionally gets into an error state that can not be resolved by restarting the software (driver issues). It requires a full (virtual) system reboot.

A small monitor was written prior to my involvement in this project that can detect the unfixable error state, and initiate a nice clean service shutdown and reboot to resolve the problem. In KVM this results in a warm boot which doesn't throw away the -snapshot saved changes.

So, qemu-kvm has an option -no-reboot that forces process exit when the vm tries to do a warm boot. This shuts off the VM but does not restart it. I need to auto-restart vms that shut themselves down by trying to reboot.

So, I really have three requirements:

qemu-kvm needs to be invoked with -snapshot (fresh image every power off)
Reboots should really be a libvirt stop/start to get a 'cold boot' effect and throw away the snapshot,
Servers can reboot themselves. These warm boots are turned into a VM stop, and they need to be auto restarted asap

While qemu-kvm has the support I need, the version of libvirt on Centos 7 that I use doesn't have direct support for either option. I make a wrapper for qemu-kvm and then specify a custom emulator for these vms. I use both -snapshot and -no-reboot like this:

1 2	#!/bin/sh exec /usr/libexec/qemu-kvm "$@" -snapshot -no-reboot

And then we replace the block in the vm definition. This meets my first two requirements, but any system reboot or periodic shutdown will stay off.

What can I do to make sure they are always running? First mark them to start on host boot:

virsh autostart foo

which makes them start when the host boots but I still need to make sure they get restarted if they stop.

There are a few options I considered to solve this problem.

First, I could add some kind of 'forever' loop to the emulator script above that will just run the emulator again once it exits. e.g.

 #!/bin/sh
    while true; do
        exec /usr/libexec/qemu-kvm "$@" -snapshot -no-reboot
    done

Or Second, I could write some kind of supervisor that tries to start any stopped vms, like a standalone daemon, or an every-minute-run cronjob.

Third, Libvirt supports event hooks. I can run my own code when certain events happen. The hook would watch for shutdown events and then 'virsh start' a vm that just shut down. This is the path I started with (because I use hooks for some autoscaling functionality too). The problem is that there is a deadlock. "virsh start foo" in a hook is running before the VM is actually stopped and the process will hang forever waiting for the shutdown to finish.

I confess to taking the lazy way out. I have a hook that gets shutdown events, forks child in the background and returns. The child sleeps for 3 seconds - presumably enough for the vm to stop, and then does a 'virsh start' like this:

#!/bin/sh
# install into /etc/libvirt/hooks/qemu
# any vm that shuts down will be restarted

if [ "$#" -lt 2 ]; then
    echo "usage: $0 <domname> <event> end -"
fi

# release event signifies shutdown is finished
if [ "$2" == 'release' ]; then
    sh -c "sleep 3; virsh start $1" < /dev/null 2&>1 >/dev/null &
fi

Dirty, but it works. Note that Centos 7 does not by default have /etc/libvirt/hooks - you have to make that directory and restart libvirtd to pick up any hooks you add.

If I didn't have other hooks code (for autoscaling), I probably would have gone with the first option above as it is a trivial few more lines of bash.

Is there some other way I should have done this? Drop me a line if you see something obvious I missed.

Bashcpio - pure bash (almost) cpio archive extraction

2016-06-05T16:00:00-06:00

Ok - let's get this out of the way. Is this important? No. Is it groundbreaking? No. Can I even explain how cool it is to non-technical friends? No. Could it ever possibly be useful? Maybe! see below.

I just spent a few evenings writing a pure-bash cpio extraction implementation. My target was centos/fedora/redhat rpms, which means gzip/xz compression, written with the so-called 'New-Ascii' header format. The other formats wouldn't be that hard to support but I would guess that 98% of any use of cpio in the year 2016 is related to some kind of rpm package.

And so is born bashcpio - check it out on github:

https://github.com/minektur/bashcpio

pure bash?

Also, it's not quite 'pure' bash - but no bash script ever is. I worked hard to ensure that the external dependencies were kept to a bare minimum.

In addition to an actual bash binary, you need a few external dependencies:

dd, mkdir, dirname, chown, chmod, date, touch, ln

There are a couple that might be considered optional. I could probably write some tortured complicated code to replace dirname that has no dependencies on external binaries, though I have not (yet?) done so. If you don't care about time-stamps on files and directories that you extract, date and touch could both go away. The rest though, are pretty difficult to avoid using if you want it to work at all.

Bash (and some other shell utilities that you might consider, like awk and sed) all have this nasty habit of "eating" null bytes in anything they read. So for instance, you can't place the the content of a binary file in a bash variable. You'll get everything up to the first null value. This makes dealing with arbitrary binary data with bash problematic at best. If however, you can be reasonably sure that some portions of your file will be ascii, then you can carefully operate on those parts with bash, using the assistance of dd.

In this case, the header format I'm trying to read is guaranteed to be 100 bytes, followed by a C-style string, and a null terminator, followed by some padding, a bunch of arbitrary data and some padding. The saving grace is that you know where the ascii header starts, how long it is, and inside there is data about how long the file name is, and how long the file is. With that, you can read a header and then safely read the filename, the sizes, and all the other header info.

How? I an pick apart the file with dd, carefully using the count= and skip= flags.

All the rest of the external dependencies are for use in creating and modifying the extracted files, setting ownership, group, permissions, mtime, etc.

Motivation?

I'm toying with the idea of adding a centos target to crouton

created/maintained by David Schneider, with many many other contributors.

Crouton is a tool for making (debian-derived) linux chroots that will run under chromeos in dev mode. (I'll talk more about this in some later blog post). Crouton is an awesome tool and it's quite useful, but I use a lot of yum rpm and not a lot of apt and deb.

On the chromeos devices there is a very minimal linux install that does little besides running chrome browser full time. Many common things you'd expect to be at your fingertips at a bashprompt are missing. Crouton fixes this. And it bootstraps itself using pretty much only bash and an 'ar' extractor written in bash, since that is close to all that is available to start with.

Seeing the clever ar.sh implementation, I got thinking about how one would do the same with rpm based distros... and bashcpio was born. I started with a careful inventory of what shell binaries I had access to, and bash + dependencies does the trick. This of course does not give you a centos chroot - it is only one of several necessary tools. But I had fun making it work, except when I was trying to get the padding right. Ugh, how irritating. I'm no expert bash programmer though I know a lot more now than I used to.

Performance?

My initial implementation only took about 80x as long as the x86_64 debian trusty binary that I got out of a crouton chroot. The current implementation is much faster now and it now only takes approximately 19 times as long at the C-binary. To get a feel for it's performance, I grabbed the 350ish packages that make up a current centos 7 minimal install. It takes less than 4 minutes to unpack them.

Please feel free to send me feedback on how crazy you think I am, or on what glaring deficiency my code has. I'm eager to talk about the ship I built in a bottle.

Apple mail app TLS deficiencies revisited

2015-10-02T09:00:00-06:00

A quick update:

Mac OS X El Capitan has hit the streets so I retested. The mail app still can not make any TLS connections using TLSv1.1 or 1.2.

Also a /u/Tulsagrammer on Reddit pointed out to me that a the mystery of why this is so is even greater: For (outbound) SMTP connections, the mail app can and does speak TLSv1.2. It is ONLY imap connections that are limited to TLSv1.0.

Frustration with Apple mail app on IOS and Yosemite

2015-09-10T09:00:00-06:00

At work I recently have been irritated by a problem that was exposed with Apple's mail app, both on IOS and Yosemite. Among other things, I maintain an imap server (using dovecot ) for our office email.

First some background

Dovecot makes it easy to enable TLS, and disallow unencrypted logins. It actually uses openssl under the hood, so whatever cyphers your version of ssl supports are the set dovecot will support. By default, dovecot has TLSv1.0, TLSv1.1, and TLSv1.2 enabled by default. In the web-browser world, for about the last year, there have been known crypto-attacks against TLSv1.0 and some concern about TLSv1.1. Recommendations for server operators have been to completely disable TLSv1.0. For instance, Qualys has an awesome free ssltest that gives poor grades to 1.0 deployments.

More importantly, for my installation, the "Payment Card Industry" security standard - aka PCI - now strongly discourage the use of what they call "SSL/early TLS". See page 47 of this pdf. When you dig in to what this means, you find that SSLv2 and SSLv3 (which predate TLS) and TLSv1.0 are specifically what they are talkign about. These crypto standards are all pre-Y2K, and have been replaced by newer, better, more secure protocols. Technically my email system is completely isolated from my payment-related systems, and thus might get a pass during any PCI scan/audit. But, to save the hassle, and argument about "compensating controls" from 3rd party vendors who help validate our compliance, I recently disabled TLSv1.0 on dovecot.

In dovecot this is easy. In /etc/dovecot.conf.d/10-ssl.conf you add !TLSv1 to your ssl_protocols. Currently mine looks like this:

ssl_protocols = !SSLv2 !SSLv3 !TLSv1

Now the Real Problem

As soon as I did this every iphone in the company, and every mac user using the mail app on Yosemite all were unable to get their email. The IOS app gives a vauge error about not being able to "connect to the server" and the desktop app gives a 'timeout' error, and the mail connection doctor shows the connection appear to hang during setup.

It turns out that mail.app can only speak TLSv1.0. There is no support for 1.1 or 1.2. TLSv1.0 was first defined as a spec in 1999, 1.1 in 2006 and 1.2 in 2008. 8 years ago. Isn't it time for an update? I've found a few people complaining about this in Apple's support forums, but only recently has 1.0 been deprecated. I expect more people will feel the pain as time progresses.

To be fair to Apple, my 2 year old HTC One M7's mail app has the same issue - I'm told that versions 4.5 and later have TLSv1.2 support.

Testing Notes

While for https there are a few great free tools out there, for other tls-enabled protocols, it is much less fun. Openssl has a nice command line tool you can use for such things. Note that the version on Yosemite is old. Thus you'll need to make sure and test from a more recent version. I used the a linux server on the same network segment as my mail server.

Here is how I tested:

To make a simple connection test:

openssl s_client -connect servername:143 -startls imap

This will connect to "servername" on port 143 and initiate the imap version of starttls. You should get a response back that gives details of the cert and any certificate chain, a copy of the server certificate, some debugging info and a block that might look something like this:

    New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-GCM-SHA384
    Server public key is 2048 bit
    Secure Renegotiation IS supported
    Compression: NONE
    Expansion: NONE
    SSL-Session:
        Protocol  : TLSv1.2
        Cipher    : DHE-RSA-AES256-GCM-SHA384
        Session-ID: 03C689EDC9EC7F09BA904D44DAB8AB9E9CA835B4AED350E8147466FF62B211AF
        Session-ID-ctx:
        Master-Key: 5620257CDF5223C53F3A830B4078F78AAB40C2BDF2D3F41750879642E3C31A29A6BD8802CA8815B494141FBA76830C79
        Key-Arg   : None
        Krb5 Principal: None
        PSK identity: None
        PSK identity hint: None
        TLS session ticket lifetime hint: 300 (seconds)
        TLS session ticket:
        0000 - a4 dc 2e 63 45 fa 69 3c-06 13 23 43 67 09 a5 3c   ...cE.i<..#Cg..<
        0010 - fb e1 93 47 b8 17 95 f8-ee 84 cb 1d 89 c5 a3 a6   ...G............
        0020 - 8d 7e 20 94 f3 35 10 8d-de 55 eb 4b b2 64 9c 2b   .~ ..5...U.K.d.+
        0030 - 33 5a 5a 57 30 20 17 65-8f 20 43 7c 1d 32 ec fe   3ZZW0 .e. C|.2..
        0040 - 80 1c 1a d5 ae 17 97 13-78 b4 b1 01 43 43 27 1e   ........x...CC'.
        0050 - 9a 46 24 84 88 c7 37 9e-58 0e a7 b0 c0 a5 1f f8   .F$...7.X.......
        0060 - f4 90 3b 2c b8 00 65 55-51 8a 6d 36 0e 27 c1 5b   ..;,..eUQ.m6.'.[
        0070 - 03 c1 d9 25 a9 ae 2a 36-37 66 ad cf ea e4 72 b3   ...%..*67f....r.
        0080 - fb 9e 1b a7 6a 00 c3 96-b5 04 ea 0f f7 f9 5a 5f   ....j.........Z_
        0090 - ce d2 03 75 ec 6b 36 2c-01 94 ae 47 28 66 65 7d   ...u.k6,...G(fe}

        Start Time: 1441902734
        Timeout   : 300 (sec)
        Verify return code: 0 (ok)
    ---
    . OK Capability completed.

At this point, if you know the imap protocol you can start typing commands.... For my purposes, seing . OK was enough to know that I had connected.

If you want to see if a specific version of tls is supported you can specify on the command line which version of tls to use:

    openssl s_client -connect servername:143 -starttls imap -tls1

    ....
    SSL-Session:
        Protocol  : TLSv1
        Cipher    : DHE-RSA-AES256-SHA
        Session-ID: B73943E3C764F2A08120794941F3E314A758F8FB6D6D037B31B7ECF7F197C7ED
    ....

That is of course if you haven't disabled TLSv1.0.

If you have, the response looks something like:

    openssl s_client -connect servername:143 -starttls imap -tls1

    CONNECTED(00000003)
    write:errno=104
    ---
    no peer certificate available
    ---
    No client certificate CA names sent
    ---
    SSL handshake has read 271 bytes and written 0 bytes
    ---
    New, (NONE), Cipher is (NONE)
    Secure Renegotiation IS NOT supported
    Compression: NONE
    Expansion: NONE
    SSL-Session:
        Protocol  : TLSv1
        Cipher    : 0000
        Session-ID:
        Session-ID-ctx:
        Master-Key:
        Key-Arg   : None
        Krb5 Principal: None
        PSK identity: None
        PSK identity hint: None
        Start Time: 1441903081
        Timeout   : 7200 (sec)
        Verify return code: 0 (ok)
    ---

Openssl s_client also supports other tls-enbled protocols - see the documentation for details.

Hopefully this will help you test your own servers.

Lousy Workaround

How to work around this? Well, for us, a separate vlan, with a secondary IP address for the mail server, and special vpn access to that vlan. Dovecot lets you have different ssl_protocol settings per IP that the daemon is listening on, so I have a block in my 10-ssl.conf that looks like this:

    ssl_protocols = !SSLv2 !SSLv3 !TLSv1

    local 192.168.128.139 {
    ssl_protocols = !SSLv2 !SSLv3
    }

which disabled TLSv1.0 on the public interface, but leaves it enabled on the more tightly controlled vlan. Theoreticlly this is enough of a compensating control to keep the PCI auditors happy. Yes, to check mail, my iphone users have to VPN in. Ugh. Hopefully Apple will fix this soon.

What do you get from a $200 multimeter?

2014-10-02T09:00:00-06:00

I was recently discussing with friends how I like Apple computers (e.g. the MacBook Pro I use for work) but have a hard time justifying the extra cost when in many cases I can get something 'good enough' for half the price.

And sometime around that conversation, the fact that I own and use a Fluke 87-V multimeter came up.

Considering the fact that I can drive 5 minutes from my house and buy a cheap multimeter, I had to stop and think. I do own several cheapies that I use regularly (e.g. one lives in my toolbox where it gets beat up a lot, one in the trailer, etc)

I do have training in electronics, but it's not what I do for a day job. My uses of a multimeter - do they justify 30x the cost for the fluke? So I decided to put together a list of what I have recently used the fluke for that I can't use the cheapys for.

I've had a couple of projects in the last few weeks that have involved using a multimeter: Repairing a pinball machine, and helping my son diagnose a malfunctioning robot-kit he was assembling.

Here are some of the things I got for my money:

Measures frequency (e.g. clock on PIC controller on robot)
Measures max/min/average from when I press a button to when I press it again - used to see if an AC relay was tripping while i was behind the pinball machine messing with connections in the backbox. I've also used this to help diagnose battery, cable, starter issues on my car. I can see the lowest voltage the battery produced under starter load.
INSTANT continuity checking/beeping - hook one end with alligator clip to solenoid terminal on flipper, quickly drag other lead across many contacts on remote connector/wire-bundle to find the right wire in seconds, without even having to look at the meter...
Quality leads that I can use on high-voltage stuff and not worry that I'm going to injure myself
High enough voltage on 'diode-check' mode to actually light up an LED, or in this case an IR-LED on the robot, while viewing it with my cell-phone-camera to see it light up (seriously fun - point your crappy-cell-phone camera at your IR remote control and see the leds light - https://www.youtube.com/watch?v=_m7454Xt-OI)
Capacitance testing
Auto-ranging is SOOOO nice - there is ONE DC setting - not 8 different DC ranges. The fluke is not the fastest on selecting the best range - I've seen faster, but I dont want to afford meters that are faster.
I didn't think this would be such a big deal... but AUTO-POWER-OFF has saved my batteries so many times.

One thing - I actually like the battery testing on my cheapy centech from harbor freight better. It uses a small load to actually measure current produced by the AA battery I'm testing. It is much more accurate than looking at voltage on it... Plus I worry much less about my kids testing batteries with a $6 multimeter - so that is what lives in the junk drawer in the kitchen.

All that aside, I could probably get most of that list from a $100 multimeter... For me, if you use a tool a lot, you should buy a great one and enjoy it. Did I just talk myself into buying a new macbook?

Not Sharing python command history between python2 and python3

2014-09-16T09:00:00-06:00

I'm finally getting around to switching most-of-the-time to python3.

One minor annoyance was that I had to fix my simple .pythonrc.py file that I use to turn on command history for python2.

In python 3, it appears that readline, tab-completion, and command history between sessions are automatically enabled.

I used to have to do this for python2

in my shell (bash in this case...)

export PYTHONSTARTUP=~/.pypythonrc.py

and then make a file .pythonrc.py in my home directory. The contents of that file, were:

# Add auto-completion and a stored history file of commands to your Python
# interactive interpreter. Requires Python 2.0+, readline. Autocomplete is
# bound to the Esc key by default (you can change it - see readline docs).
#
# Store the file in ~/.pythonrc.py, and set an environment variable to point
# to it:  "export PYTHONSTARTUP=~/.pypythonrc.py" in bash.


import atexit
import os
import readline
import rlcompleter
import sys

sys.path.append('/usr/local/lib/python2.7/site-packages')
historyPath = os.path.join(os.environ['HOME'], '.pyhistory')

readline.parse_and_bind('tab: complete')

def save_history(historyPath=historyPath):
    import readline
    readline.write_history_file(historyPath)

if os.path.exists(historyPath):
    readline.read_history_file(historyPath)

atexit.register(save_history)

del os, atexit, readline, rlcompleter, save_history, historyPath, sys

This would turn on readline, tab completion, and load any existing command history so that I could repeat commands with either the arrow keys or good-ol' emacs-bindings of cntrl-p/cntrl-n. It also arranges to save my current history buffer to a file at exit so that it can be reloaded the next time I invoke a shell. It also twiddles my sys.path to include stuff I want, useful mostly for back when I didn't use virtualenv so much.

The mac Python3 build from python.org aparently already does all of this history and tab-completion stuff. but the format used to save history has changed sometime between the version of readline included with 2.7 and 3.4. So the history files are not compatible. Python3 by default uses the name .python_history. I'm renaming my old history file to be .pyhistory2 so there is no confusion later.

I now do this:

# Add auto-completion and a stored history file of commands to your Python
# interactive interpreter. Requires Python 2.0+, readline. Autocomplete is
# bound to the Esc key by default (you can change it - see readline docs).
#
# Store the file in ~/.pythonrc.py, and set an environment variable to point
# to it:  "export PYTHONSTARTUP=~/.pypythonrc.py" in bash.

import sys

if sys.version_info.major == 2 and sys.version_info.minor == 7:
    import atexit
    import os
    import readline
    import sys
    sys.path.append('/usr/local/lib/python2.7/site-packages')
    historyPath = os.path.join(os.environ['HOME'], '.pyhistory2')
    readline.parse_and_bind('tab: complete')

    def save_history(historyPath=historyPath):
        import readline
        readline.write_history_file(historyPath)

    if os.path.exists(historyPath):
        readline.read_history_file(historyPath)

    atexit.register(save_history)
    del save_history, historyPath, os, atexit, readline

del sys

#other stuff?

While python 3 and 2 don't share history, at least it works in both.

Chromecast - This is why we can't have nice things

2014-09-11T10:00:00-06:00

There was a recent chromecast firmware update from google. This wasn't the long awaited new functionality promised at IO. It ostensibly has better tab-casting performance and some bug fixes (e.g. change in some apis related to subtitles, among others).

There has been some speculation that the reason for the update, without the promised "Summer 2014" new functionality was to address a recent announcement by some smart guys on XDA about a new method of rooting the device - the so-called HubCap Root Release.

One other unmentioned change, which affects me, was the change to how Chrome Remote Debugging works on the device.

It used to be that the remote developer port was open 24x7 on devices in development mode. Now, it appears that the ONLY time a device has the developer port available and open is when running a dev app that you have registered in the developer console. When the app ends, the connection is closed. I presume that there is some firewall-rule magic going on - that the port is only opened after your dev app has successfully launched.

This makes my Display Whatever You Want trick useless. Yes, it still works, but ONLY when you have already launched some app you have control of. I feel like going off on a long rant about the folks at Google breaking my toys and restating the title of this post. But... I want to try to give them the benefit of the doubt.

I'm pretty sure that they didn't rush out a new firmware update just to spite me. They must have had this change in the works for quite a while now.

My current operating theory is that having the remote-debugging stuff available all the time was a security issue for them. Specifically, they have content providers (think "big media streaming companies"), that use DRM - I would guess that having debug access to the receiver app would make it much easier to reverse-engineer and circumvent content protections. DRM is probably a necessary evil, placating content providers, while only keeping honest people honest.

I had a simple automatically starting slideshow program that no longer works. But, I can work around it.

Using my native-api POC from last post, I have been able to launch my app, which only needed slight tweaks to be considered a "real" receiver app.

Here is simplified html similar to what I did before the remote-debugging port was restricted.

<html> <head> <title>castphotosaver</title>
        <script>
            function new_img() {
                var i = document.getElementById('slide');
                i.src='/img?'+Math.random();
            }
        </script> </head>
<style type="text/css">
img { display: block; margin-left: auto; margin-right: auto; max-width: 100%; max-height: 100%; }
</style>

<body>
<img id='slide' src="/img">
</body></html>

I feed it images from a little webservice - each hit on /img gives the next photo. I add the query parameter as a (lame) way of keeping the browser from caching and not reloading the image.

The bare minimum to turn this into a receiver that the cast device will like is this:

<html> <head> <title>castphotosaver</title>
        <script type="text/javascript"
                src="//www.gstatic.com/cast/sdk/libs/receiver/2.0.0/cast_receiver.js">
        </script>
        <script>
            function new_img() {
                var i = document.getElementById('slide');
                i.src='/img?'+Math.random();
            }
        </script> </head>
<style type="text/css">
img { display: block; margin-left: auto; margin-right: auto; max-width: 100%; max-height: 100%; }
</style>
<body>
<img id='slide' src="/img">
<script type="text/javascript">
  window.onload = function() {
    window.castReceiverManager = cast.receiver.CastReceiverManager.getInstance();
    //castReceiverManager.start({maxInactivity: 60});
    castReceiverManager.start();
    setInterval('new_img();',3000);
  };
</script>
</body></html>

All you have to add is code to load the receiver library and instantiate and start a 'CastReceiverManager'. As a side note you can control the PING/PONG frequency for connected senders with that commented out maxInactivity parameter...

The sender code to launch this is left as an exercise to the reader. In reality I never wrote a sender. I used my python-native-api proof-of-concept to load the app.

I fully expect a new update to break this in the future.

In retrospect, I think that perhaps it would have been better if Google had NOT restricted when the debugger port was available on debug devices, but rather restricted the debug device to only run your own registered apps. At least it would make my life easier.

This is why we can't have nice things.

Chromecast - steps closer to a python native api

2014-09-05T15:00:00-06:00

So, after seeing this: https://gist.github.com/TheCrazyT/11263599 I got more interested in being able to speak the native chromecast api from python.

That lead me to this presentation by Sebastian Mauer: http://www.slideshare.net/mauimauer/chrome-cast-and-android-tv-add14 especially slide 20, which lead to a bit more google searching.

This lead to a couple of related projects by Vincent Bernat and Thibaut Séguy, nodecastor and node-castv2/node-castv2-client - both node.js apps that speak approximations of the same protocol that the chrome-browser plugin uses to talk to the chromecast device.

They are both based on the protobuf data format used internally and some places externally at Google. TL;DR: make a .proto file that defines a bunch of data fields and Googles protobuf code will automatically format data into and parse data out of very compact, efficient binary blobs that you can use for data serialization between various platforms and languages.

The .proto file used by chromium builds to enable the browser plugin to communicate with devices is used by nodecastor and node-castv2 to allow them to speak with chromecast devices.

To give you a flavor, here are partial contents of the cast_channel.proto file...

// Copyright 2013 The Chromium Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
syntax = "proto2";
option optimize_for = LITE_RUNTIME;
package extensions.api.cast_channel;
message CastMessage {
  enum ProtocolVersion {
    CASTV2_1_0 = 0;
  }
  required ProtocolVersion protocol_version = 1;
  required string source_id = 2;
  required string destination_id = 3;
  required string namespace = 4;
  enum PayloadType {
    STRING = 0;
    BINARY = 1;
  }
  required PayloadType payload_type = 5;
  optional string payload_utf8 = 6;
  optional bytes payload_binary = 7;
}

Once you have run the google 'proto' compiler, you get a file: cast_channel_pb2.py that you can import into your programs and now have an easy way to encode your data across platforms/languages for rpc/data interchange.

In python it might look at simple as this:

#!/usr/bin/env python
#properly formatted PONG response for a receiver PING request
import cast_channel_pb2
msg = cast_channel_pb2.CastMessage()
msg.protocol_version = msg.CASTV2_1_0
msg.source_id = "sender-0"
msg.destination_id = "receiver-0"
msg.payload_type = cast_channel_pb2.CastMessage.STRING;
msg.namespace=namespace['heartbeat']
msg.payload_utf8 = """{ "type": "PONG" }"""

A bunch of reverse-engineering by those guys yields a pretty good over view of the wire-level chromecast protocol.
See here.

So, I implemented a proof of concept in python using Googles protobuf python bindings, using TheCrazyTs gist as an outline. Eventually, I got the data encoded right, the lengths calculated right and presto! I can start the default media receiver and ask it to stream some media, all from native python code with no browser in the middle.

If you have an existing ssld connection to your chromecast, you could do this:

be_size = pack(">I",msg.ByteSize())
payload = be_size + msg.SerializeToString()

If you had already connected to a chromecast with a 'CONNECT' request, you might get a PING request from the receiver, which you could send this payload back to. If you want to see some actual code that can connect, get status, start the default media receiver and ask it to stream a video, see the actual code

See https://github.com/minektur/chromecast-python-poc for code and instructions to try it out yourself.

Next up: Write some 'real' networking code and integrate into pychromecast

Chromecast - Displaying arbitrary URLs using pychromecast

2014-08-25T15:00:00-06:00

Continuing on with my Chromecast experiments... I have been playing with a python library on github by Paulus Schoutsen called pychromecast. At various points in it's life it has been able to interact with Chromecast devices to do a variety of things. With the official SDK release and firmware update from google earlier this year, much of that broke. It can however find devices on my local network, and get device status, and a few other things.

Referring back to my last post, my ideal app for the chromecast would be a daemon running on my Mac or FreeBSD box that will watch the device for inactivity and, if it wasn't doing anything else, would cause photos or home videos to be streamed to the device.

You may recall that I recently was learning about remote chrome browser debugging. If you register for a Developer Account with google, you can whitelist your Chromecast and get it put in development mode. Once in development mode, the device turns on remote chrome debugging on the standard port 9222.

I dug around to find out about the protocol the debugger speaks over that debug port. You can read all about the Chrome Remote Debugging protocol but here is a brief over view. There is a very simple web server running on that port that gives you a link for each tab showing in Chrome, or in this case each tab showing on your Chromecast. If you click that link, it will actually open the Chrome debugger, connected to a websocket on the device. You can enumerate what the websocket endpoints are - http://cast-IP:9222/json - you get a small bit of json back listing all the details of your open tabs.

>> curl http://192.168.0.138/json
[ {
  "devtoolsFrontendUrl": "/devtools/devtools.html?host=192.168.0.138:9222&page=1",
  "faviconUrl": "",
  "thumbnailUrl": "/thumb/chrome://newtab/",
  "title": "New Tab",
  "url": "chrome://newtab/",
  "webSocketDebuggerUrl": "ws://192.168.0.138:9222/devtools/page/1"
} ]

If you make a websocket connection to webSocketDebuggerUrl you can speak the remote debugging api to your browser, or remote Chromecast-in-developer-mode device.

A "quick" persual of the api docs shows a function that I need. Page.navigate

If you send

{"id":0, "method":"Page.navigate", "params":{"url":"http://www.clift.org/fred"}}

over the wire, then presto, what you see on your TV is my website.

You dont have to do all the work yourself. Paulus has been kind enough to merge my code into pychromecast.

See: https://github.com/balloob/pychromecast/blob/master/pychromecast/remotedebug.py

If you join the Google developer program for $5, and install pychromecast, you can do this very simply:

>> devices = pychromecast.get_chromecasts_as_dict()
>> devices['castdevicename'].crd_open_url("https://github.com/balloob/pychromecast")

This wont work for any old device, and you can't ship apps that work this way, but for my "ideal" app, It should be pretty simple, with pychromecast. I've got a rudimentary proof of concept working now.

Note that shipping apps that do this would be against Googles TOS, and they wouldn't work anyway...

Have fun with your personal projects.

P.S. Someone else did this in ruby too!

Chromecast - both cool and frustrating

2014-08-21T08:00:00-06:00

I recently purchased a Chromecast device. For the price it's a great media streamer. You can control it from your Android or IOS smartphone from many different apps. For chrome browser, you need to install a browser plugin. There are many websites that, when viewed in Chrome browser give you the option of casting the content.

By casting I mean, you tell the Chromecast device to load it's own special app and have that app directly stream the content. In Chromecast dev terms the app you tell the device to load is a 'receiver' app.

Youtube is a prime example. On your computer when watching a Youtube video, you find an extra button that allows you to send the video directly to your Chromecast device. This allows your Chromecast to directly stream the content, rather than needing your browser or smartphone in the loop.

You can also mirror the contents of any chrome tab and some android-devices to the screen. It takes a lot of bandwidth, but if you have a good network, the quality is good. (built on WebRTC)

There is a lot of clever magic going on behind the scenes to make all of this work on your home network. Initial setup for instance, is done by having the device create it's own wireless network, and having a setup program on your PC connect to the network and then to the device to talk to it directly.

Once it's set up, your devices find Chromecasts in the local network via Multicast DNS (... used to use DIAL/SSDP/UPnP ...) Some low-level code in the Chromecast API impelemntation, supported, at least in the Chrome browser takes care of sending IP multicast dns queries asking for any devices that will respond to the name _googlecast._tcp.local. All the responses are collected and cached.

Your Chromecast-api using app can then, at the user's request, contact any of the devices, ask the device to load a receiver app, and then establish a control channel to the app to tell it what media to load, to control playback, etc.

All the api communication with the Chromecast device is over ssl'd websockets.

Much more technical detail about how Chromecast devices work can be seen at Justin Loutsenhizer's excellent Github page, and at Sebastian Mauer's presentation.

Google is pretty uptight about the user-experience they will allow with published apps. See their Terms of Service. This agreement is what people who want to develop and distribute a Chromecast app must agree to to get their app usable by the general public. There is also the Google Cast Design Checklist and some User Interface guidelines.

I have some real problems with the terms of service. Some of it is understanable. Section 3.4.7 for instance is about user-data and privacy - common sense. Or, section 3.4.3 - don't encourage users to engage in high-risk activities... What I don't like are the terms that forbid exactly what I actually want to do with the device. 3.4.8 - must not automatically cast content, 3.4.1 - must not load arbitrary apps - only Google-Approved ones, 3.4.2 - don't allow usage of the device without the official API...

I wouldn't care (much) about being required to use only the official API if they would actually support a platform I cared about developing for. The three supported APIs are for Android, IOS, and chrome-browser apps. There is no native SDK/API for anything else.

What I really want? An daemon to run on my FreeBSD server in the basement, or on the family mac-mini that has access to the 60 Gig of family photos and home-videos, and to play random ones, if, and only if, the Chromecast device is on the home-screen. It would poll periodically to see if the device is busy, and restart showing photos if the device wasn't in use. I want it to start automatically, as a system service, find the device by name, and turn my tv into a giant digital photo frame.

Next post I'll talk about pychromecast, and the code I contributed that will let me make my dream a reality, while technically not violating Google's TOS.

refs, for me mostly:

<https://github.com/vincentbernat/nodecastor/blob/master/lib/cast_channel.proto>
<http://www.slideshare.net/mauimauer/chrome-cast-and-android-tv-add14>
<https://github.com/wearefractal/nodecast>
<http://dashkiosk.readthedocs.org/en/latest/chromecast.html>
<https://www.npmjs.org/package/nodecastor>
<https://github.com/thibauts/node-castv2>

Notes on chrome remote debugging

2014-08-07T12:00:00-06:00

These are mostly notes for me, but you might find them useful also.

On my laptop, I run chrome and usually have many, many tabs open across a few windows. Google searching for me usually ends up with me open-in-new-tabbing the first 5 or 10 links concurrently... It bugs me when I have to close my browser and loose my set of open tabs, though, in the case of crashes, modern browsers do a decent job of remembering what I had up.

Chrome at least does not remember my open tabs if I quit chrome cleanly. Now and then, I need to use Chrome's Developer Tools. Sometimes I need to run the developer tools remotely. Chrome Developer Tools are pretty sweet - lots of debugging and performance tools for web development. Generally you run the developer tools in the same instance of chrome as whatever you're debugging, but you can actually run the UI on another instance of Chrome, either on the same machine or remotely.

Running them remotely is almost indistinguishable from running them within the same browser instance - cool technology the Chrome guys have written.

All you need to do is to use the command line option

--remote-debugging-port=9222

or whatever port you want and chrome will start up listening on that port for connections from a remote debugger. To debug, just point your OTHER chrome browser, on the same or other box at

http://<YOUR IP>:9222/

and you'll get a list of the open tabs/frames. Click on one and off you go.

So, my problem. I have chrome open with 60 or 70 tabs, across 3 or 4 windows. I want to run chrome with that command line option so I can play around with some command-line tools to to remote debugging/control. I don't want to shut down chrome...

So, I use another command line option

--user-data-dir=/tmp/foo

and I can invoke a second instance of chrome, that is completely separate from what I already have running.

On my macbook:

/Applications/Google Chrome.app/Contents/MacOS/Google\ Chrome --user-data-dir=/tmp/foo --remote-debugging-port=9222

Presto - a new chrome instance, using a separate profile, with remote debugging enabled, without having to shut down my precious tabs.

Why the Nook won my dollars over Kindle

2014-07-18T11:10:00-06:00

Summer Vacation Time.

So I vacation on the beach every summer. I read a lot. This year I decided I would try out an ebook reader. Because the plan is to be outdoors on the beach, E-Ink readers seem to be the way to go. There are two big names for new readers Barnes & Nobles Nook (Simple Touch Glowlight) and Amazons Kindle (Paperwhite - with light...).

This is not going to be one of those long drawn out technical comparisons. There are plenty of those out there. For me, it came down to two things - supported formats, and manufacturer business model.

The kindle supports amazon's proprietary formats AZWx, txt pdf, mobi, prc. The Nook supports epub and pdf. Note that both claim to support more formats either through some form of file conversion, which I'm fully capable of doing myself, or only "support" them with air quotes. For instance the Nook supports several popular image formats, but only for it's screensaver. I own a large number of ebooks in epub format, and the reader software I use on my android phone (FBReader) uses epub as it's primary format also. I occasionally use the great open-source software Calibre to convert ebooks from one format to another, even though the the process of doing so is clunky and complicated.

As far as business models go, with the kindle, unless you pay a non-trivial amount more, you get (not very obtrusive) advertisements. If I pay for something, I'd rather not become a revenue source and ad target because of it. Also, the Nook has a much simpler register-without-a-credit-card process, though you can successfully register a kindle without a credit card, irritating but possible.

Technically, I think I might like the kindle more... but... Nook is what I spent my money on.

Managing Lots Of Pregenerated HTML And Other Files With Pelican

2014-06-14T23:08:00-06:00

For one of the Pelican-managed websites I maintain, I have a lot of files that I don't really want to manage with Pelican, and in some cases, I can't easily, without lots of gymnastics.

In this case, I have about 30k html files that are a dump of an old phpNuke web forum. And, one directory that has a php app installed (Simple Machines Forum). I also have a a lot of other miscellaneous files that I'd like to go into the DocumentRoot of my web server

For the HTML, I could write a script to modify each file to include the meta tags that pelican needs, then put all the images into my images directory, and then put all the CSS into my template. For my archival content, that would be acceptable, but for the SimpleMachines forum, which is maintained by 3rd parties and updated regularly, it isn't practical. I would have to redo my changes every time they released a new version...

I could use STATIC_PATHS, with ARTICLE_EXCLUDES and PAGE_EXCLUDES to do what I want, but for some reason pelican still wants to look at each file. Doing this increased my build times 10x or so with my data. In short, I could use the static_paths to add all my files, and then the excludes settings to tell pelican to not process them.

Instead I tried the following, and use this now. In my top-level directory, I made a directory 'output-skel' and then I edited my Makefile. In the html target, I added this line as one of the commands that is run:

rsync -SHqav output-skel/ $(OUTPUTDIR)

so my html make block looks like this:

html:
    $(PELICAN) $(INPUTDIR) -o $(OUTPUTDIR) -s $(CONFFILE) $(PELICANOPTS)
    rsync -SHqav output-skel/ $(OUTPUTDIR)

This way, I can put ANY file I want to include in my output in it's relative path in output-skel and have it show up where I want it.

This sure beats having 30k entries in my STATIC_PATHS. I put .htaccess files, static-html, some 'downloads' files there.

As an aside, I also add the following lines to my html make target:

    chown -R root:www $(OUTPUTDIR)
    find $(OUTPUTDIR) -type d | xargs chmod 750
    find $(OUTPUTDIR) -type f | xargs chmod 640

Just to keep things neat and tidy - both secure the files and allow the webserver to see them.

Why I wont fix your computer, part 3

2014-06-12T12:00:00-06:00

Not quite a computer, but...

I have an Embryon Pinball machine made by Bally in the early 80s - one of their early solid-state machines.

See either the Internet Pinball Database or Pinside for pictures and more info.

When I bought this machine, I got a great deal on it because the sound was not working and it was missing almost all of the playfield plastics (still looking for an original set if you have a lead...) AND the playfield is trashed.

I've repaired the power supply, replaced a bunch of caps on the soundboard, and done a bunch of little repairs and cleaning on switches, pop-bumpers etc. I dont want to put to much into the machine because I think the quality of the playfiled is so low that it's not worth the effort.

Now, on to why I wont fix your computer - I mean, pinball machine. One of the things that WAS in good condition when I bought the machine was the apron. It has some little flecks of paint or something on it and in my stupidity, I thought that I should try a little water and magic-eraser.

The excellent results speak for themselves. Oh wait, I mean, what an idiot I am. At least the kids still have fun playing it.

This is why I wont fix your pinball machine.

PS - I have a bunch of pictues, a few of my machine and many others I've collected. Including 3 sets of plastics scans I've bummed off people (thanks all) and even one relatively low res playfield scan. (maybe a photo?). See my embryon archive. My current playfield is shown in painful detail in the playfield-crappy directory, and most of the good scans are in source-materials/donors. Hopefully this helps someone out there.

Why I wont fix your computer, part 2

2014-06-11T12:00:00-06:00

On being stupid with my own hardware

I built a mame machine into a dead arcade cabinet for the breakroom of a former employer. Well, it was for me, and for a few others that had regularly been playing an original upright Vs Super Mario Brothers, and then later an original Marble Madness (gimme that double WAND).

I was in the middle of a system upgrade (old motherboard died...) and I had a pretty good chance that my very-small-linux-install on the drive would 'just work'

I had the whole thing apart and was about to connect power to the IDE drive the system ran on... and while distractedly talking to a co-worker, I attached the new drive, while the other system was still on. It fried the drive...

What would probably have been a 5 minute hardware swap turned into a several-hour reinstall/re-setup of my mame front end (AdvanceMENU), on a new drive, which I fortunately had laying around.

What did I learn? Turn off the power. Sigh. this is why I dont want to fix your computer.

Embedding PHP in Pelican-generated Static pages

2014-06-10T09:37:00-06:00

So, I wanted to make a simple website with pelican. But I had a little legacy php code that I still wanted to function.

It sure would be nice, I thought, if I could take my php app and with only small tweaks make it work as PART of pelican staticly generated page. But pelican doesn't directly parse php. I could just staticaly copy my php pages into my docroot and have some links to them but then I loose the menu, the CSS etc.

Long story short, I made a reStructuredText page in content/pages that has the following:

Sample Php App
##############

:date: 2014-05-03 10:20
:url: pages/sample.php
:save_as: pages/sample.php


.. raw:: html
    :file: php/sample.php

php/sample.php is relative to the directory that this RST file is in.
The php page can run standalone and produces the output I want, minus the site integration. PHP is not one of the formats that pelican knows how to process so it ignores the php code.

When I generate my html output (e.g. 'make html') pelican generates a page http://mysite/pages/sample-php-app.php which has a big html wrapper around some php code. Works great and it lets me integrate some legacy php tools into an othewise static site.

Trying out Pelican static site generator.

2014-06-10T09:36:00-06:00

TLDR

I'm running a couple of websites with pelican static website generator because it's easy to maintain, and lightweight, and kind of futurer-proof.

History

So I've tried a bunch of tools over the years to make personal web pages. I hand-rolled html (for a class) in the 90s, I used a variety of templates that were table based, and then later CSS based, and then a bunch of different blog and micro-blog software. Lighter weight ones like Blosxom and Serendipity, and several heavy-weight contenders like MovableType and Wordpress.

I have worked in hosting related fields off and on for 15 years and because of the nature of the business, my perspectives are a bit skewed. Density, multi-tennancy, and automatic upgrades are important, Working at NTT/Verio also kept me focused on localization. All that is important when you run software for other people on a large scale. But, I don't do that any more. Setting software up for myself makes me want to find lightweight and very easy to maintain systems. Blosxom is great in that it is mostly run from the cli - mostly flat files. It also has a lot of cool plugins. I used Serendipity with the sqlite backend, which again removed one service dependency and upgrade-pain-point.

My problem

I also recently got the opportunity to migrate an old old phpNuke based website (with phpBB forums) which were installed around 2002 and barely upgraded since. We had been limping along with all the admin functionality disabled because of the many many security vulnerabilities. We had upgraded from php3 to php4 on FreeBSD 8 with modest tweaks. The 5 year old hardware started to fail, and my 'free' colo with a friend was straining relationships.

My Solution

So, I set up a kvm instance at BetterServers which is a sister company to my employer BetterLinux, installed FreeBSD 10, and started to migrate the Website, the game, and shell accounts. PHP4 could be made to work, theoretically, but I didn't really want to have to build by hand (no longer supported by the ports system - the last release was 6ish years ago). Then I started trying to migrate the apps, mostly unchanged to work with PHP5. Ugh. Somwhere in there I decided that it would be better to throw out the old app, make an easier to maintain website and migrate a most of the content. I looked at several different dynamic and static simple cms systems that I could use. I knew I needed forum software and so I tried a few of those hoping that they might have themes or plugin systems that would allow a simple static site along with the forum. I tried some CMS systems like GetSimple that I could embed some forum software into... Eventually... I stumbled across Pelican which generates a website from a template and a variety of different markup-formatted files. (I mostly use markdown, but several others are supported) Pelican is in python, (works in 2.x and 3.x) and is aimed at someone who wants to, and is not afraid of tinkering.

So I set up a new website blackmud and embedded SimpleMachines forum into it and then migrated a lot of content in various forms from the old site.

Verdict? I like pelican. The templates are mostly aimed at people who want to write a blog. I was brushing up on my html, php, and markdown and did not want to have to learn Jinja2 templates also, though they look pretty simple.

I'd love a multi-level static website menu template. I kind of faked it with a heavily modified 'syte' template. I found a way to embed some legacy php utilities into the static page system (inline raw 'html' file includes in a reStructuredText doc) so that I get my old php utilties but get to keep the static menu and css around it. I'll write up the details sometime. The benefit of this is that I end up with mostly static html that should be very portable and not take much babysitting for years. Yes, I'll have to stay on top of SimpleMachines updates... I swear I'll be vigilant, no really I will! I hope.

I liked pelican enough that I thought I would replace my 5 year old one page site, and broken dead blog site with pelican generated stuff. This is the result

Why I wont fix your computer, part 1

2014-06-10T06:00:00-06:00

How not to upgrade a server

I was working at a unix admin at a private university. A research lab wanted an OS upgrade on their lab NFS and web server, which was indirectly my responsibility. User data, webserver, web content, all on an external (scsi) drive. I showed up with install media and popped in the disk, booted, installed to scsi ID 6 - the local default for OS on this class of server. Shortly after first boot I go to mount the filesystems on the external drive, twiddle rc scripts to spin up the web server, mount user directories etc, and there doesn't seem to be anything but the old OS...

Immediate panic - apparently both internal OS drive and external data drive are SCSI Id 6, on controllers 0 and 1. I had just successfully installed hpux 10.10 over the lab's important data. The backup that the grad students had assured me as safely locked away? 1 month old.

So what did I learn? how to apologize, to double check-backups before upgrades, to double-check target drive for installs, to not offer to help!