Home
login

Linux

Reverse a Binary Stream Using Busybox

Today I had the need to reverse a binary stream using only bash and commonly-available command-line utilities. Not tac, sed, or rev, which are all line-oriented utilities that work best on ASCII data. I needed something that I could trust with binary data. This is what I came up with. Feel free to point out my weakness.

The first round was this:

reverse() {
    local i=0
    cat | xxd -c 1 | awk '{print $2}' | tac | \
        while read F; do
            printf "%06x: %s\n" $i $F; i=$((i+1))
        done | xxd -c 1 -r
}

I wasn't a huge fan of the while loop to prefix the lines with addresses for 'xxd -r'. The streams that I am using this for are only several kB max, so efficiency was not my first goal, but why not try to make it faster if you have the option? Some reading reveals that 'tac' is not available on every Unix platform. And 'xxd' is only available if you have vim installed. I swapped in 'hexdump' for 'xxd', but hexdump does not have a reverse, so I had to find a way to do that. This is where awk comes into play, doing and integer to character conversion for each line. This happens to run in about 6 times faster than the original version and uses stuff that even busybox has.

My final version was this:

reverse() {                                                                                                            
    cat | hexdump -v -e '/1 "%d\n"' | \
        sed -e '1!G;h;$!d' | \
        awk '{printf "%c", $0}'
}

You might use it like this:

$ reverse < file > file.reversed
# or
$ command -in -a | pipeline | reverse | process | reverse > some_output

Kenwood TH-D72 and Linux

I recently found a buyer for my Icom IC-92AD, which enabled me to buy one of the new Kenwood TH-D72 radios. This is my first GPS-enabled device and a new radio to boot. I am thrilled. I got it in the mail in just enough time to scan through the instruction manual to figure out how to use it for the Monday night Beaverton CERT Net. I got on the air without any problem. The manual is not nearly as nice as the Icom manual was. First of all, they don't give you the complete manual printed, only a getting started guide. The manual is on a CD in PDF format.

The TH-D72 has a mini-B USB connector and comes with a cable. Curious, I plugged it in to my computer and saw that it loaded the cp210x driver and gave me a /dev/ttyUSB0 device. Hooray!!! It didn't work. :( It turns out that the Natty kernel I am running has a regression in it (a story for another day). I tried out the Maverick kernel and it works just fine. So running the Maverick kernel, I was able to open up minicom, set the baud rate to 9600, and establish communication with the radio. It is NOT self discoverable. Grrr. I type in something and it gives me back '?'. It appears that there are two modes. With the packet12 TNC enabled, it will echo your keystrokes and give you a 'cmd:' prompt. If you type something wrong, it will say '?EH'. Without the TNC enabled, it does not echo keystrokes and will give you a '?' if it did not understand the command you sent it.

Not seeing an obvious way to figure out the command set, I figured that we should try to reverse engineer it. I installed the MCP-4A program in wine. I tried to run it and it complained that it needed .NET 2.0. I tried installing dotnet20 and found that is not quite enough -- it wants dotnet20sp1 or greater. dotnet20sp2 does not install. dotnet30 does not install. When I run MCP-4A with dotnet20, it throws a few errors and does not give me full use of the program (no menubar, for example), but it does run. I was able to use Wireshark to sniff the USB traffic as I performed a read and write. Then I turned to python to whip up something that can do this natively. This is what I have so far:

#!/usr/bin/python
# coding=utf-8
# ex: set tabstop=4 expandtab shiftwidth=4 softtabstop=4:
#
# © Copyright Vernon Mauery, 2010.  All Rights Reserved
#
# This is free software: you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as  published
# by the Free Software Foundation, either version 3 of the License, or (at
# your option) any later version.
#
# This sofware is distributed in the hope that it will be useful, but WITHOUT
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
# FITNESS FOR A PARTICULAR PURPOSE.  See the GNU Lesser General Public
# License for more details.
#
# You should have received a copy of the GNU Lesser General Public License
# along with this software.  If not, see <http://www.gnu.org/licenses/>.

def command(s, command, *args):
    cmd = command
    if args:
        cmd += " " + " ".join(args)
    print "PC->D72: %s" % cmd
    s.write(cmd + "\r")

    result = ""
    while not result.endswith("\r"):
        result += s.read(8)

    print "D72->PC: %s" % result.strip()

    return result.strip()


def l2b(*l):
    r = ''
    for v in l:
        if type(v) is str:
            r += v
        else:
            r += chr(v)
    return r

def bin2hex(v):
    r = ''
    for i in range(len(v)):
        r += '%02x '%ord(v[i])
    return r

def bin_cmd(s, rlen, *b):
    if b is not None:
        cmd = l2b(*b)
    else:
        cmd = ''
    print "PC->D72: %s" % cmd
    s.write(cmd)
    result = bin2hex(s.read(rlen)).strip()
    print "D72->PC: %s" % result
    return result

def usage(argv):
    print "Usage: %s <serial device> <read-image>" % argv[0]
    sys.exit(1)

if __name__ == "__main__":
    import serial
    import sys

    if len(sys.argv) < 3:
        usage(sys.argv)

    s = serial.Serial(port=sys.argv[1], baudrate=9600, xonxoff=True, timeout=0.25)

    #print get_id(s)
    #print get_memory(s, int(sys.argv[2]))
    print command(s, 'TC 1')
    print command(s, 'ID')
    print command(s, 'TY')
    print command(s, 'FV 0')
    print command(s, 'FV 1')
    print bin_cmd(s, 4, '0M PROGRAM\r')
    s.setBaudrate(57600)
    s.getCTS()
    s.setRTS()
    of = file(sys.argv[2], 'wb')
    for i in range(256):
        sys.stdout.write('\rfetching block %d...' % i)
        sys.stdout.flush()
        s.write(l2b(0x52, 0, i, 0, 0))
        s.read(5) # command response first
        of.write(s.read(256))
        s.write('\x06')
        s.read()
    print
    of.close()
    print bin2hex(s.read(5))
    print bin2hex(s.read(1))
    print bin_cmd(s, 2, 'E')
    s.getCTS()
    s.setRTS()
    s.getCTS()
    s.getCTS()
    s.close()

You run it like this:

$ python thd72.py /dev/ttyUSB0 d72-dump.dat

Unfortunately from what I have seen, two consecutive reads without any changes on the radio seem to have very big differences. It is as though some of the chunks of the file are rotated or shifted by a few bytes (and the shift is not constant throughout). Not seeing an immediate reason for this, I suspect that it is some form of obfuscation. Call me a pessimist.

I will continue to work on this, but I would love to see what others in community are doing as well.

Update:

I forgot to mention that the whole point of this exercise was to find a way to work it into CHIRP. I am currently working on a driver for this radio to enable it in CHIRP. And as I was looking over the tmv71 code in CHIRP, I noticed that I should be reading a response to the read block command _before_ I actually read the block data. This seems to help things out a bit (and I modified the above code to match).

Musings on the transition to IPv6

As a part of the he.net IPv6 certification program, you have an opportunity to earn extra points by daily activities: IPv6 traceroute, AAAA dig, IPv6 PTR dig, IPv6 ping, and IPv6 whois. Among those, I would say the most commonly used by me are ping and traceroute, then the digs and finally whois. But I use ping and ping6 everyday. Yes, the tool to ping something is actually 2 tools. It seems odd that while so many other programs seamlessly handle both network protocols just fine, this one requires a whole new binary. The 'route' command, is a single binary, using a command line argument '-4' or '-6' to determine which protocol to use. It defaults to IPv4, since that was around first. On the other hand, nc, dig, ssh/scp, telnet, mtr, wget, curl, ip (though it uses 'inet' and 'inet6'), and many more default to IPv6, only choosing IPv4 if specified or if it is the only protocol available. Then we have the dark side: ping/ping6, traceroute/traceroute6, iptables/ip6tables (along with the -save and -store variants), and maybe some others.

What prompted the authors of those programs to fork the code to add IPv6 support? I would bet that most of the logic is the same. In fact the man page for ping lists ping6 too; they have the same options. Why can't I just say `ping -6 he.net` or `traceroute -4 google.com`? Last night, I finally got fed up with ping and wrote a little python wrapper for ping, which parses the arguments, ignoring all except '-4' and '-6' and then passing everything else on to the appropriate ping. If neither '-4' nor '-6' are passed, it does a quick dig for an AAAA address to test if the remote host is even capable of IPv6 and then automatically chooses the right ping. I shouldn't have to do this though. Please give me a reason besides breaking legacy stuff to have an entirely new program. Legacy is not a good enough reason, you can only pack around so much baggage before it pulls you down. Just ask Intel. :)

Finally, you have the ubiquitous browsers, which if they are well behaved do both IPv4 and IPv6, favoring the latter when it is available (and not even really letting you pick or telling you what you are using; the only way to know for sure is to have the webserver report you IP address). This is helpful for the chicken/egg scenario that we are seeing with this transition. We don't need to support IPv6, nobody is using it. But if we can at least get all the dual-stack folks to use it by default, that will give a little push. This leads us to other services. Most of the services that I have come across now support IPv6 as well; http (apache2, lighttpd, cherokee, and more), smtp (postfix, exim, qmail, and more), imap (courier, uw-imap, and more), dns (bind, dnsmasq), and many more. The services vary on whether they listen by default on IPv6 if it is available. Most you have to configure to suppress IPv6 support; that is a good default.

The truth is, most people don't care about the transition to IPv6; most don't even know what IPv6 and don't want to know. But the end of IPv4 is looming on the horizon and it may hurt when it gets here unless we, the geeks who run the networking backbone of the planet, make sure we are ready. The current policy of many applications to default to IPv4 for legacy's sake needs to stop. If the computer has IPv6 connectivity, it should be using that by default and falling back to IPv4 only when explicitly requested or when the service is not available on IPv6. Then we just have to get all those lumbering service providers to move. But there's no business justification for IPv6.... Make one and save the planet.

Server Tinkering

I was born to tinker. I think this must be the opposite of the optimizer. I see a project in anything that I could tweak to make it a little better. This not only applies to computers, which are the easiest thing to tinker with, but food, DIY projects, and more. This particular post is centered a little more around computer tinkering, just as a warning to the technophobes.

My host for the past 2+ years for this server has been Site5. They have been adequate. I had never used a Web Hosting Service before so this was a whole new experience. Moving there from a private server took a lot of tweaking. Server wise, they were pretty good. I think my site got its fair share of the server pie, but it is not a really demanding site. Service wise (meaning the people), I think they only get 4 out of 5 stars. Whenever I had a problem, they did finally resolve it, but it took some work and push-back from me to make it happen. Usually the first contact would try to blow me off. I would patiently explain that they were contractually obligated to fix the problem and then 'level 2' support would fix it. I could deal with this if they had all the features I wanted, but I wanted more. Sure, they have 'unlimited' disk space (as long as you don't use it), and unlimited bandwidth, which with my vast sea of devoted readers, I don't really need. But what I do need is IPv6. And they have no plans for that (at least I am privy to none).

So I jumped ship. The market for dual stack hosting is not yet very big so there really aren't that many service providers yet. I finally found BurstNET®, which seemed to offer IPv6 as well as very low-priced VPS (Virtual Private Server). So low, in fact that I could get a whole VPS for less than I was paying at Site5. That's very cool. Being a tinkerer, I really need w00t. Still, since BurstNET uses OpenVZ technology instead of Xen or KVM, I don't quite have complete control over everything. I don't get to configure my network, for instance. But I do have two static IPv4 IP addresses; doing my part to reduce the remaining pool of IPv4 addresses. And after a quick service request, they granted me two IPv6 addresses. Yes, only two, not an entire subnet. I thought that was odd, but hey, at least it is something. Their service department has been nothing but good. I have made several requests for help:

  • Request for IPv6 connectivity
  • Request for reverse-DNS mapping IPv4 and IPv6 addresses
  • Request to get ip6tables working

All there responses were quick and positive. This was the best service I had ever gotten and for what? Yup, $5/mo. This month I got more than my money's worth in support man-hours. I am hoping that the tinkering I have done over the last week is sufficient to have my VPS in decent shape.

Also as part of my tinkering, I managed to set up my VPS as a master name server for the three DNS zones that I control (mauery.org, mauery.com, and my he.net IPv6 arpa reverse zone). Then, using HE.net's DNS service, I can push to their DNS slave servers. This means that I have five geographically diverse, topologically diverse, redundant nameservers. So even though almost nobody reads my blog, you will never not be able to track it down.

Now on to the next tinkering project....

IPv6 Certified

IPv6 Certification Badge for vmauery

Some will care and some will not, but I can now boast that I have finished all the IPv6 certification tests at at ipv6.he.net. The last one was a real stinker. A while back, I registered mauery.org because I wanted to tinker with DNS stuff. But it turns out that I registered with a registrar that doesn't support IPv6 glue records, which were the entirety of the last certification step. I gave up after a while, since I didn't know what I was doing and didn't have the time to sit down and figure it out. Recently I got the itch to finish up my certification with he.net. I looked into glue records, which are basically the link that breaks the recursion in DNS. For IPv4, the glue records are apparently pretty easy to come by, but few registrars will do IPv6 glue records yet. Especially few actually have a way to do it without raising a support ticket. I found one that does (gkg.net), moved mauery.org over to them, and got my glue records all up in the TLD's nameserver. Hooray.

Now, mauery.org is a fully functional IPv4/IPv6 domain. You can access it via IPv4 only, dual stack, or IPv6 only. Too bad it doesn't really do anything that you could get access to... I don't run any public webservers in the domain or anything. It is basically my home network. Like I said, I purchased the domain because I wanted to play around with DNS and learn some things. For a long while, I was using mauery.home as my domain. I know, .home is not a real TLD, but I figured that it would make sure that there were no namespace collisions with any legitimate domain names. I have since moved over to the mauery.org domain.

Does my certification make me superior to you? Maybe not. But my domain kicks your domain's arrobase. And the free t-shirt doesn't hurt.

MythTV 0.23 running on Lucid

Bella, my mythbox has been overheating for the past few months when the furnace has been running. Whenever we would watch HD content, decoding (and likely resizing) the stream to our screen would take too much CPU horsepower. Watching an HD stream would use nearly 100% of one of the two 2.1GHz Athlon cores. Then her over-sized, quiet fan would kick into high gear. Sometimes I would feel sorry for her and pull the top cover off to let out all that hot air. VDPAU has been around for a while now, and Bella's video card is supported, so I have been itching to upgrade her to Lucid. I started a couple of weeks ago by installing to an external USB drive. This way, I didn't interrupt the regular programming. I could boot to the drive, tinker, fix, test, etc., and then boot back to the main drive when I was done.

I tried to copy over the original database so I didn't lose any settings, but that didn't seem to work very well. The database upgrade scripts kept dying on me. I finally rolled up my sleeves and dug in a little deeper. The error messages that it was giving me were about duplicate columns in tables. I am not sure how they got there, but with my mysql hacker-foo skills, I manually altered the tables to drop the columns, which allowed the script to successfully update the mythconverg database. Then it seemed to be working. Kind of.

Bella has a little USB card reader that does a variety of memory card types. For whatever reason, when probed, it reports that it has devices there already, even though there are no cards plugged in. It reports /dev/sd{d,e,f,g}. For whatever reason, mythfrontend likes to probe devices and try to mount them??? When it runs into these non-devices, it segfaults. I finally just decided to unplug the card reader from the motherboard and mythfrontend starts up just fine.

With Myth finally up and running, it doesn't take long to make sure all the settings are good. I kick off a show and notice that the processor is still running at full speed. A little more digging and I find the VDPAU setting screen. I turn it on and viola! CPU usage drops to 40%. I was a little underwhelmed by this number, hoping to see something more on the order of 5%. I don't know if it is because of the stream type or maybe my hardware? The video card is several years old and one of the earlier ones that does support VDPAU. And the stream is whatever the broadcasters in my area are using. I assume it is MPEG2, while H.264 is what VDPAU would prefer? Anyway, something is better than nothing. I just hope that the reduction in CPU thermal requirements is sufficient.

Defeating a Ninja

I was recently asked if I had ever used Ninja on any of my machines. I had not even heard of such a thing. I found that it was available in the Lucid repository, so I installed it and configured it for my system. The two changes I made were to have it constantly poll, instead of waiting 1 second between polls; and to kill the offending tasks, instead of merely warning about them. I whipped up a little back door application that gets root. I ran it and sure enough, it was instantly killed by the Ninja. But how instantly? There is a race there to be sure. I had my w00t process run 'killall -9 ninja' instead of giving me a root shell. Sure enough, my hack was faster than the Ninja. It must only be a wannabe or maybe a Ninja Apprentice. Whatever it is, I defeated the Ninja in less than ten minutes. Without even breaking a sweat.

Ninja may be good in concept, but poor in execution (I'm so punny!!!) The race window that exists is wide enough to easily defeat it. What it needs to do is something along these lines:

1) Run the critical section in the kernel as a kprobe parked on the fork, exec, setuid, setgid syscalls.
2) On each watched syscall, check for escalation and pause execution while we pop out to userspace (probably via the kprobe print functions) and do a bit of testing.
3) If we pass muster, let the task go, otherwise kill it.

This way, we can pause the questionable tasks in the kernel before they get a chance to do anything malicious.

Right now, all Ninja does is make you think you are safe, while really, it does very little for you.

Stupid Ubuntu Theme Change

Ubuntu usually does things right, but this time they screwed up. They moved my buttons. It is worse than Windoze, which changes the UI every release (every 2 years), because my UI is changing every 6 months.

STOP CHANGING MY USER INTERFACE YOU RAT FINKS!!!

I installed Lucid Lynx a while back because I wasn't happy with Karmic. And as they are getting closer to the release, they changed all their art. And themes. AND MY BUTTONS. I logged back in after an upgrade and found all my buttons on the left of my window instead of the right. After some digging, apparently this is a configurable thing, but unless you have set it, they changed the defaults.

http://ubuntuforums.org/showthread.php?p=8963718

That tells how to change your order. I prefer gconf-editor because messing with the registry via a command line never works for me. Note that for right-sided icons you want the ":" before all the icon names, not after.

Traffic Shaping and Policing

Recently I had the opportunity to work with a customer that needed some help with traffic shaping and policing on their network. I had poked around in the past with this, trying to get guaranteed bandwidth for my VoIP phone, but the last time I checked, that setup no longer worked, so it was shelved until further notice. I just had to take care that when I was on the phone, I could not do any large downloads that would rob the bandwidth from my voice packets.

The customer gave me impetus to re-learn Linux Traffic Control. The main tool offered to us is called tc, meaning traffic control. You can learn all about tc at the Linux Advanced Routing and Traffic Control website. I spent several hours there trying to remember all I had forgotten. I also looked around at several other websites with howtos on the matter, but it seemed that they were all pointing back to lartc.org anyway. I poked around long enough to understand the recipes in their cookbook and then wrote up a script of my own.

I wanted to have about 90kb/s of guaranteed bandwidth for VoIP and then some other high priority bandwidth for things like ICMP packets, TCP ACK packets, and other low-latency stuff (things that mark the TOS field in the IP header.) In other words, I wanted to be able to:

  1. Make sure my VoIP traffic gets through so I don't have choppy phone calls
  2. Perform uploads without killing my downloads (let the ACK packets through)
  3. Be able to type in an SSH session while doing a large download
  4. Not starve my VPN to work when the network is busy (no more 3-12 second latencies, please.)
  5. Have fast ping times so I can brag to all my friends

Sounds like I am hoping for a miracle, right? Well, not really. Simply dividing the traffic into several classes and then giving each one a slice of the pie will do a lot on my quest for the Well Tempered Network. I know the VoIP bandwidth, so that is easy. Then the rest, I decided to split into quarters -- high priority gets at least 1/4 of the remaining bandwidth, medium priority gets the same, while bulk transfers and the rest of the stuff get anything that is left over (a little less than 1/2 the pipe).

Without this QoS script, I am unable to do a large download (or upload) without killing my VoIP call, uploads kill downloads, ssh is very non-interactive, and pings range in the 400-1100ms range. With this QoS script, I can do simultaneous large downloads and large uploads without hurting my VoIP call quality AND at the same time, ssh interactivity goes up (to the same as with no other traffic) and ping times range in the 80-200ms range. VPN traffic seems to be better too, though sometimes it suffers from latencies beyond my control. I think this means I reached all my goals. I was very happy with it and thought it might be nice to share.

I finally git it!!

I was so happy to learn that the 2.6.26 kernel had a free as in speech alternative to the madwifi Atheros driver, ath5k. I have not been so happy that it has been crashing my machine periodically. Since diagnosis of a hung machine without a serial console that is running X is nigh on impossible, I had no choice but to fold and go back to madwifi. However, since then, I have moved on to 2.6.27, which has some changes to two wireless APIs, causing the madwifi driver to fail to build. I wanted to leave my desk with its hard-wired connection but didn't want my machine to hang again.

12next pagelast page