Automating tcpdump in Test Scripts

 It's not unusual for me to create a shell script to test networking software, and to have it automatically run tcpdump in the background while the test runs. Generally I do this "just in case something gets weird," so I usually don't pay much attention to the capture.

The other day I was testing my new "raw_send" tool, and my test script consisted of:

INTFC=em1
tcpdump -i $INTFC -w /tmp/raw_send.pcap &
TCPDUMP_PID=$!
sleep 0.2
./raw_send $INTFC \
 01005e000016000f53797050080046c00028000040000102f5770a1d0465e0000016940400002200e78d0000000104000000ef65030b
sleep 0.2
kill $TCPDUMP_PID

Lo and behold, the tcpdump did NOT contain my packet! I won't embarrass myself by outlining the DAY AND A HALF I spent figuring out what was going on, so I'll just give the answer.

The tcpdump tool wants to be as efficient as possible, so it buffers the packets being written to the output file. This is important because a few large file writes are MUCH more efficient than many small file writes. When you kill the tcpdump (either with the "kill" command or with control-C), it does NOT flush out the current partial buffer. There was a small clue provided in the form of the following output:

tcpdump: listening on em1, link-type EN10MB (Ethernet), capture size 262144 bytes
0 packets captured
1 packets received by filter
0 packets dropped by kernel

I thought it was filtering out my packet for some reason. But no, the "0 packets captured" means that zero packets were written to the capture file ... because of buffering.

The solution? Add the option "--immediate-mode" to tcpdump:

tcpdump -i $INTFC -w /tmp/raw_send.pcap --immediate-mode &

Works fine.

Python and Perl and Bash (oh my!)

I've been thinking a lot recently about code maintainability, including scripts. I write a lot of shell scripts, usually restricting myself to Bourne shell features, not GNU Bash extensions. I also write a lot of Perl scripts, mostly because I'm a thousand years old, and back then, Perl was the state of the scripting art.

Anyway, it's not unusual for me to write a shell script that invokes Perl to do some regular expression magic. For example, I recently wanted to take a dotted IP address (e.g. "10.29.3.4") and convert it into a string of 8 hexadecimal digits representing the network order binary representation of the same (i.e. "0a1d0304"). My kneejerk reaction is this:

HEX=`echo "10.29.3.4" | perl -nle 'if (/^(\d+)\.(\d+)\.(\d+)\.(\d+)$/) { printf("%02x%02x%02x%02x\n", $1, $2, $3, $4) } else {die "invalid IP $_\n"'}`

But since maintainability has been on my mind lately, most programmers (especially younger ones) would have a steep Perl learning curve to maintain that. So my second thought was to do it in Bash directly. I haven't done much regular expression processing in Bash, given my habit of staying within Bourne, but really, that habit has outlived its usefulness. Open-source Unix (Linux and Berkley) have relegated other Unixes to rare niche use cases, and even those other Unixes have a way to download Bash. I should just accept that Bash extensions are fair game. Here's my second try:

pattern='^([0-9]+)\.([0-9]+)\.([0-9]+)\.([0-9]+)$'
if [[ "$MCAST_ADDR" =~ $pattern ]]; then
  GRP_HEX=$(printf '%02x%02x%02x%02x' ${BASH_REMATCH[1]} ${BASH_REMATCH[2]} ${BASH_REMATCH[3]} ${BASH_REMATCH[4]})
else echo "invalid IP addr"; exit 1
fi

But even as I feel fairly confident that more programmers would be able to maintain that than the Perl version, I still realize that the vast majority of programmers I've known have very basic shell scripting skills. I'm not sure the Bash version expands the pool of qualified maintainers by much. I think the best that can be said is that a programmer with basic shell scripting skills can learn regular expression matching in Bash a lot faster than learning enough Perl to do it.

So, how about Python?

Well, with some help from Claude, here's a Python one-liner:

HEX=`python3 -c "import sys, ipaddress; print('{:08x}'.format(int(ipaddress.ip_address(sys.argv[1]))))" 10.29.2.3`

So, not only does this use a language that many programmers know, it also completely avoids regular expressions, which is another uncommon skill among the programmers I've known.

*sigh*

What's a dinosaur to do? I haven't been paid to be a programmer for a lot of years, so the programming I do is mostly for my own entertainment. And dammit, I *like* Perl! I've done enough Python programming to know that ... I just don't like it that much. And let's face it: the code I write is unlikely to be widely used, so who cares about maintainability?

(pause for effect)

I do.

I have standards on how programming should be done. If programming is now largely my hobby, I get the most personal reward by doing it according to my standards. I think it's time for me to say a fond farewell to Perl and bow down to my Python overlords.

SIGINT(2) vs SIGTERM(15)

This is another of those things that I've always known I should just sit down and learn but never did: what's the difference between SIGINT and SIGTERM? I knew that one of them corresponded to control-c, and the other corresponded to the kill command's default signal, but I always treated them the same, so I never learned which was which.

• SIGINT (2) - User interrupt signal, typically sent by typing control-C. The receiving program should stop performing its current operation and return as quickly as appropriate. For programs that maintain some kind of persistent state (e.g. data files), those programs should catch SIGINT and do enough cleanup to maintain consistency of state. For interactive programs, control-C might not exit the program, but instead return to the program's internal command prompt.
  
  
• SIGTERM (15) - Graceful termination signal. For example, when the OS gracefully shuts down, it will send SIGTERM to all processes. It's also the default signal sent by the "kill" command. It is not considered an emergency and so does not expect the fastest possible exit; rather a program might allow the current operation to complete before exiting, so long as it doesn't take "too long" (whatever that is). Interactive programs should typically NOT return to their internal command prompt and should instead clean up (if necessary) and exit.

This differentiation was developed when the Unix system had many users and a system operator. If the operator initiated a shutdown, the expectation was that interactive programs would NOT just return to the command prompt, but instead would respect the convention of cleaning up and exiting.

However, I've seen that convention not followed by "personal computer" Unix systems, like MacOS. With a personal computer, you have a single user who is also the operator. If you, the user and operator, initiate a shutdown on a Mac, there can be interactive programs that will pause the shutdown and ask the user whether to save their work. It still represents a difference in behavior between SIGINT and SIGTERM - SIGINT returns to normal operation while SIGTERM usually brings up a separate dialogue box warning the user of data loss - but the old expectation of always exiting is no longer universal.

New Outlook: Security/Privacy Issues

 So, my old MacBook Air finally became unusable as my primary machine (the screen backlight burned out, possibly due to my having dumped a can of Diet Coke onto it). I can still VNC to it, so it will continue life as a "headless" Mac, but I needed a new laptop.

Based on my satisfaction with my work system, which is Windows and WSL2, I decided to go that route. But I didn't want to go with Office 365 and its annual subscription - it just seemed a bit steep to me. So I uninstalled Office 365 and bought Office Home and Business 2021.

At first it seemed fine, until I wanted to sync the contacts with my iCloud contacts. I didn't have tons of time to mess with it, and there isn't an easy way to do it, until I noticed there there is a "New Outlook" that I could turn on. I did, and lo and behold, it was able to download my iCloud contacts without a problem. Mind you, it is still not SYNCING, it just took a snapshot, but I don't update my contacts very often, so I'm calling it a win.

But then I noticed that "New Outlook" immediately downloads pics on emails. They claim it's secure because it uses a Microsoft server as a proxy -- i.e. the sender of the email cannot see my IP address -- but we know that there are tracking images whose file names are just identifiers for the recipient. So in effect, the sender is informed when I look at their email. I don't like that.

Worse yet - when I hover over a clickable link, it doesn't tell me what the URL is! THIS IS UNACCEPTABLE! I think it might be going through Microsoft's "safe URL" thing, which is better than nothing, but not nearly good enough. Now I have to right-click the link, copy it, and paste it into my browser to make sure it looks OK before hitting enter.

What's next? Does the new Outlook automatically execute code sent with the email? It used to, maybe they're doing that again.

Here I was all ready to admit that Microsoft was finally taking security and privacy seriously enough for me to switch; boy was I wrong. It will irk me no end to pay for Office 365 after having paid for Outlook 2021.

I guess I'll try to figure out a way to complain to MS, but I'm also confident my complaint will go nowhere.

Downpour Games

One of my favorite YouTubers, Tom Scott, has an email list that he posts to periodically listing things that interest him. His latest one pointed to downpour.games which makes it easy to create simple games (Tom called them "interactive story games"). Here's an example to get the idea: https://downpour.games/~holly/where-s-madeleine

While the games can be played on a laptop, you need to use the phone app to create the games. It takes a bit of experimenting and exploring to figure out the user interface. Being a text-based thinker, it probably took me longer than average. My biggest stumbling block for me was not realizing that the bottom row of buttons ("Box", "Photo", "Gallery") has more than three buttons. You have to swipe left on the buttons to see "Text". Without "Text", I can't really make much of a game.

I, of course, immediately thought of making an "adventure" game. But then realized that all state is carried in the page you are on. So, for example, if you "take the coin", there isn't any way to carry that state forward. (I mean sure, you could have a completely duplicated maze, one with the coin in its home room and the other with the coin in your pocket, but what if you drop the coin in a different room? You would need (N+1)**2 rooms. And that's for a single takable object. So only exploratory adventure games here.)

On reflection, I think "Choose your own adventure" is a better analog than "interactive story game".

Anyway, a few years ago I experimented with trying to do something like this, just using web pages and links. It quickly became tedious and error-prone.

I don't know if I'll do much of anything with this. But it was interesting enough for me to bring up.