Simple C REPL

I'm a fan of language REPLs (Read, Evaluate, Print, Loop). These are interactive programs that let you experiment with the language. For example, running the Python REPL let's you enter Python code interactively and get immediate output. No edit/compile/run cycle. REPLs are useful for experimenting with language features, exploring APIs, reproducing bugs, etc.

But I'm a C programmer, and C doesn't have a REPL. And sometimes I just want to explore details of the language, like sign extension rules and implicit type conversions. C just isn't well-suited to having a REPL. 

But since when has "not well-suited" ever stopped me?

Introducing crepl.sh : https://github.com/fordsfords/crepl

The doc is fairly comprehensive (Thanks Claude!), so I'll just show an annotated sample session:

$ ./crepl.sh
C REPL - Enter C statements or expressions
Type !help for commands
c> int x = 1;
c> x                         -- Omit semicolon to auto-print the expression.
i 1 (0x00000001)             -- The leading "i" indicates type, not the variable name
c> unsigned short j = 2      -- A declaration is not a legal expression; no autoprint!
Compilation error, line rejected. Enter '!errs' for details.
c> unsigned short j = 2;     -- Semicolon suppresses autoprint
c> j
us 2 (0x0002)                -- Autoprint knows its an unsigned short.
c> x+j
i 3 (0x00000003)
c> j=j+1                     -- An assignment statement is an expression.
us 3 (0x0003)
c> char c = -1;
c> c
c -1 (0xff)
c> x = c
i -1 (0xffffffff)                     -- Nice sign extension!
c> int inc(int x) { x++; return x; }; -- Define a function all on a single line.
c> inc(88)
i 89 (0x00000059)
c> x
i -1 (0xffffffff)                     -- Naturally "inc()" has its own local x.
c> int inc_x() { ++x; };              -- New funct. Oops, I forgot to return something.
c> int inc_x() { ++x; return x; };    -- Fix the funct? No, you can't re-define it.
Compilation error, line rejected. Enter '!errs' for details.
c> !vi                                -- This edits the code so far. I deleted inc_x().
c> int inc_x() { ++x; return x; };    -- Now I can define it properly.
c> inc_x()
i 0 (0x00000000)                      -- It treated x as a global. But is it?
c> inc_x()
i 1 (0x00000001)
c> x
i 1 (0x00000001)
c> !help
Commands:
  !help  - Show this help.
  !errs  - Show compilation/runtime errors from last attempt. Note that line numbers refer to the 'crepl_temp.c' file.
  !new   - Clear all accumulated code.
  !list  - Show current accumulated code.
  !vi    - Edit accumulated code in vi.
  !source filename - read input from filename.
  !sh    - start an interactive subshell. Exit shell to return to crepl.
  !quit  - Exit the REPL
Autoprint types handled:
  char, unsigned char, short, unsigned short,
  int, unsigned int, long, unsigned long,
  long long, unsigned long long, float, double
c> !list
Current code:

int x = 1;
x;
unsigned short j = 2;
j;
x+j;
j=j+1;
char c = -1;
c;
x = c;
int inc(int x) { x++; return x; };
inc(88);
x;
;
int inc_x() { ++x; return x; };
inc_x();
inc_x();
x;
c> !quit
Goodbye!

So, is the variable 'x' a global? The inc_x() function incremented it, so it must be, right? (Spoiler, it's not. See this doc for explanation.)

Python and Perl and Bash (oh my!)

I've been thinking a lot recently about code maintainability, including scripts. I write a lot of shell scripts, usually restricting myself to Bourne shell features, not GNU Bash extensions. I also write a lot of Perl scripts, mostly because I'm a thousand years old, and back then, Perl was the state of the scripting art.

Anyway, it's not unusual for me to write a shell script that invokes Perl to do some regular expression magic. For example, I recently wanted to take a dotted IP address (e.g. "10.29.3.4") and convert it into a string of 8 hexadecimal digits representing the network order binary representation of the same (i.e. "0a1d0304"). My kneejerk reaction is this:

HEX=`echo "10.29.3.4" | perl -nle 'if (/^(\d+)\.(\d+)\.(\d+)\.(\d+)$/) { printf("%02x%02x%02x%02x\n", $1, $2, $3, $4) } else {die "invalid IP $_\n"'}`

But since maintainability has been on my mind lately, most programmers (especially younger ones) would have a steep Perl learning curve to maintain that. So my second thought was to do it in Bash directly. I haven't done much regular expression processing in Bash, given my habit of staying within Bourne, but really, that habit has outlived its usefulness. Open-source Unix (Linux and Berkley) have relegated other Unixes to rare niche use cases, and even those other Unixes have a way to download Bash. I should just accept that Bash extensions are fair game. Here's my second try:

pattern='^([0-9]+)\.([0-9]+)\.([0-9]+)\.([0-9]+)$'
if [[ "$MCAST_ADDR" =~ $pattern ]]; then
  GRP_HEX=$(printf '%02x%02x%02x%02x' ${BASH_REMATCH[1]} ${BASH_REMATCH[2]} ${BASH_REMATCH[3]} ${BASH_REMATCH[4]})
else echo "invalid IP addr"; exit 1
fi

But even as I feel fairly confident that more programmers would be able to maintain that than the Perl version, I still realize that the vast majority of programmers I've known have very basic shell scripting skills. I'm not sure the Bash version expands the pool of qualified maintainers by much. I think the best that can be said is that a programmer with basic shell scripting skills can learn regular expression matching in Bash a lot faster than learning enough Perl to do it.

So, how about Python?

Well, with some help from Claude, here's a Python one-liner:

HEX=`python3 -c "import sys, ipaddress; print('{:08x}'.format(int(ipaddress.ip_address(sys.argv[1]))))" 10.29.2.3`

So, not only does this use a language that many programmers know, it also completely avoids regular expressions, which is another uncommon skill among the programmers I've known.

*sigh*

What's a dinosaur to do? I haven't been paid to be a programmer for a lot of years, so the programming I do is mostly for my own entertainment. And dammit, I *like* Perl! I've done enough Python programming to know that ... I just don't like it that much. And let's face it: the code I write is unlikely to be widely used, so who cares about maintainability?

(pause for effect)

I do.

I have standards on how programming should be done. If programming is now largely my hobby, I get the most personal reward by doing it according to my standards. I think it's time for me to say a fond farewell to Perl and bow down to my Python overlords.

Examining a Running Process Environment

 I don't know why I keep forgetting about this technique. I guess it's because while it is worth its weight in gold when you need it, it just isn't needed very often.

Say you're helping somebody with a networking problem. Their process isn't behaving well.

"Is it running under Onload?"

"I don't know. I think so, but how can we tell for sure?"

$ tr '\0' '\n' </proc/12345/environ | grep LD_PRELOAD
LD_PRELOAD=libonload.so

(You need the "tr" command because Linux separates entries with a null, not a newline.)

"OK cool. What Onload env vars do you set?"

$ tr '\0' '\n' </proc/12345/environ | grep EF_
EF_RXQ_SIZE=4096
EF_UDP_RCVBUF=16000000
EF_POLL_USEC=100

BAM! No need to rely on memory or what the env "should" be. We know for sure.

Bash Process Substitution

I generally don't like surprises. I'm not a surprise kind of guy. If you decide you don't like me and want to make me feel miserable, just throw me a surprise party.

But there is one kind of surprise that I REALLY like. It's learning something new ... the sort of thing that makes you say, "how did I not learn this years ago???"

Let's say you want the standard output of one command to serve as the input to another command. On day one, a Unix shell beginner might use file redirection:

$ ls >ls_output.tmp

$ grep myfile <ls_output.tmp

$ rm ls_output.tmp

On day two, they will learn about the pipe:

$ ls | grep myfile

This is more concise, doesn't leave garbage, and runs faster.

But what about cases where the second program doesn't take its input from STDIN? For example, let's say you have two directories with very similar lists of files, but you want to know if there are any files in one that aren't in the other.

$ ls -1 dir1 >dir1_output.tmp

$ ls -1 dir2 >dir2_output.tmp

$ diff dir1_ouptut.tmp dir2_output.tmp

$ rm dir[12]_output.tmp

So much for conciseness, garbage, and speed.

But, today I learned about Process Substitution:

$ diff <(ls -1 dir1) <(ls -1 dir2)

This basically creates two pipes, gives them names, and passes the pipe names as command-line parameters of the diff command. I HAVE WANTED THIS FOR DECADES!!!

And just for fun, let's see what those named pipes are named:

$ echo <(ls -l dir1) <(ls -1 dir2)

/dev/fd/63 /dev/fd/62

COOL!

(Note that echo doesn't actually read the pipes.)

VARIATION 1 - OUTPUT

The "cmda <(cmdb)" construct is for cmda getting its input from the output of cmdb. What about the other way around? I.e., what if cmda wants to write its output, not to STDOUT, but to a named file, and you want that output to be the standard input of cmdb? I'm having trouble thinking here of a useful example, but here's a not-useful example:

cp file1 >(grep xyz)

I say this isn't useful because why use the "cp" command? Why not:

cat file1 | grep xyz

Or better yet:

grep xyz file1

Most shell commands write their primary output to STDOUT. I can think of some examples that don't, like giving an output file to tcpdump, or the object code out of gcc, but I can't imagine wanting to pipe that into another command.

If you can think of a good use case, let me know.

VARIATION 2 - REDIRECTING STANDARD I/O

Here's something that I have occasionally wanted to do. Pipe a command's STDOUT to one command, and STDERR to a different command. Here's a contrived non-pipe example:

process_foo 2>err.tmp | format_foo >foo.txt

alert_operator <err.tmp

rm err.tmp

You could re-write this as:

process_foo > >(format_foo >foo.txt) 2> >(alert_operator)

Note the space between the two ">" characters - this is needed. Without the space, ">>" is treated as the append redirection.

Sorry for the contrived example. I know I've wanted this a few times in the past, but I can't remember why.

And for completeness, you can also redirect STDIN:

cat < <(echo hi)

But this is the same as:

echo hi | cat

I can't think of a good use for the "< <(cmd)" construct. Let me know if you can.

EDIT:

I'm always amused when I learn something new and pretty quickly come up with a good use for it. I had some files containing a mix of latency values and some log messages. I wanted to "paste" the different files into a single file with multiple columns to produce a .CSV. But the log messages were getting in the way.

paste -d "," <(grep "^[0-9]" file1) <(grep "^[0-9]" file2) ... >file.csv

Done! :-)

More Perl "grep" performance

In an earlier post, I discovered that a simple Perl program can outperform grep by about double. Today I discovered that some patterns can cause the execution time to balloon tremendously.

I have a new big log file, this time with about 70 million lines. I'm running it on my newly-updated Mac, whose "time" command has slightly different output.

Let's start with this:

time grep 'asdf' cetasfit05.txt

... 39.388 total

time grep.pl 'asdf' cetasfit05.txt

... 21.388 total

About twice as fast.

Now let's change the pattern:

time grep 'XBT|XBM' cetasfit05.txt

... 24.787 total

time grep.pl 'XBT|XBM' cetasfit05.txt

... 18.940 total

Still faster, but nowhere near twice as fast. I don't know why 

Now let's add an anchor:

time grep '^XBT|^XBM' cetasfit05.txt

... 25.580 total

time grep.pl '^XBT|^XBM' cetasfit05.txt

... 3:08.25 total

WHOA! Perl, what happened????? 3 MINUTES???

My only explanation is that Perl tries to implement  a very general regular expression algorithm, and grep implements a subset, and that might cause Perl to be slow in some circumstances. For example, maybe the use of alternation with anchors introduces the need for "backtracking" under some circumstances, and maybe grep doesn't support backtracking. In this simple example, backtracking is probably not necessary, but to be general, Perl might do it "just in case". (Note: I'm not a regular expression expert, and don't really know when "backtracking" is needed; I'm speculating without bothering to learn about it.)

Anyway, let's make a small adjustment:

time grep.pl '^(XBT|XBM)' cetasfit05.txt

... 17.910 total

There, that got back to "normal".

I guess multiple anchors in a pattern is a bad idea.

P.S. - even though this post is about Perl, I tried one more test with grep:

time grep 'ASDF' cetasfit05.txt

... 26.132 total

Whaaa...? I tried multiple times, and lower-case 'asdf' always takes about 40 seconds, and upper-case 'ASDF' always takes about 27 seconds. I DON'T UNDERSTAND COMPUTERS!!! (sob)

Using sed "in place" (gnu vs bsd)

 I'm not crazy after all!

Well, ok, I guess figuring out a difference in "sed" between gnu and bsd is not a sign of sanity.

(TL;DR this works on both: sed -i.bak -e "s/x/y/" x.txt)

I use sed a fair amount in my shell scripts. Recently, I've been using "-i" a lot to edit files "in-place". The "-i" option takes a value that is interpreted as a file name suffix to save the pre-edited form of the file. You know, in case you mess up your sed commands, you can get back your original file.

But for some scripts, the file being edited is itself generated, so there is no need to save a backup. So, just pass a null string in as the suffix. Right?

[ update: useful page: https://riptutorial.com/sed/topic/9436/bsd-macos-sed-vs--gnu-sed-vs--the-posix-sed-specification ]

BSD SED (FreeBSD and Mac)

$ echo "x" >x.txt
$ sed -i '' -e "s/x/y/" x.txt
$ cat x
y
$ ls
x.txt

Looks good. Let's try Linux.

GNU SED (Linux and Cygwin)

$ echo "x" >x.txt
$ sed -i '' -e "s/x/y/" x.txt
sed: can't read : No such file or directory
$ cat x
y
$ ls
x.txt

Hmm ... that's odd. It "worked", which is to say the file was properly edited. But what's with that "no such file" error? Man page to the rescue:

$ man sed
...
       -i[SUFFIX], --in-place[=SUFFIX]
              edit files in place (makes backup if SUFFIX supplied)

Interesting, you can omit the suffix. And you mustn't supply a space between the "-i" and the suffix; it thinks you've omitted it and treats the empty string as an input file. Here's an example with a non-empty suffix:

$ echo "x" >x.txt
$ sed -i .bak -e "s/x/y/" x.txt
sed: can't read .bak: No such file or directory
$ cat x
y
$ ls
x.txt

See? With the space, it thinks ".bak" is an input file. But we don't want a backup file, so let's try just omitting the suffix, like the man page says.

$ echo "x" >x.txt
$ sed -i -e "s/x/y/" x.txt
$ cat x
y
$ ls
x.txt

Works. Let's try it on BSD.

BSD SED (FreeBSD and Mac)

$ echo "x" >x.txt
$ sed -i -e "s/x/y/" x.txt
$ cat x
y
$ ls
x.txt       x.txt-e

Wait, what? Again, the file was edited properly, but what's with that file "x.txt-e"? Oh, BSD sed doesn't support a missing suffix. You can supply an empty one, but you can't just omit it. So sed looked at the above command line and thought "-e" was my desired suffix. And the "-e" option is optional in front of an in-line sed program.

ARGH!

I use both Mac and Linux, and want scripts that work on both!

THE SOLUTION

There is no portable way to tell both seds that you want in-place editing but don't want a backup suffix. So just go ahead and always generate a backup file. And remember, GNU doesn't like a space after the "-i". This works on both:

$ echo "x" >x.txt
$ sed -i.bak -e "s/x/y/" x.txt
$ cat x.txt
y
$ ls
x.txt           x.txt.bak

Works on Mac and Linux. Just delete the .bak file.

It took a long time to figure all this out, largely because the incorrect usages basically worked. I.e. the intended file did get edited in place, but with undesired side effects. So I didn't notice there was a problem until I really looked at things and saw the error or the "x.txt-e" file.

Corner cases: the bane of programmers everywhere.

Sometimes you need eval

 The Unix shell usually does a good job of doing what you expect it to do. Writing shell scripts is usually pretty straight-forward. Yes, sometimes you can go crazy quoting special characters, but for most simple file maintenance, it's not too bad.

I *think* I've used the "eval" function before today, but I can't remember why. I am confident that I haven't used it more than twice, if that many. But today I was doing something that seemed like it shouldn't be too hard, but I don't think you can do it without "eval".

RSYNC

I want to use "rsync" to synchronize some source files between hosts. But I don't want to transfer object files. So my rsync command looks somewhat like this:

rsync -a --exclude "*.o" my_src_dir/ orion:my_src_dir

The double quotes around "*.o" are necessary because you don't want the shell to expand it, you want the actual string *.o to be passed to rsync, and rsync will do the file globbing. The double quotes prevents file glob expansion. And the shell strips the double quotes from the parameter. So what rsync sees is:

rsync -a --exclude *.o my_src_dir/ orion:my_src_dir

This is what rsync expects, so all is good.

PUT EXCLUDE OPTIONS IN A SYMBOL: FAIL

For various reasons, I wanted to be able to override that exclusion option. So I tried this:

EXCL='--exclude *.o'  # default
... # code that might change EXCL
rsync -a $EXCL my_src_dir/ orion:my_src_dir

But this doesn't work right. The symbol "EXCL" will contain the string "--exclude *.o", but when the shell substitutes it into the rsync line, it then performs file globbing, and the "*.o" gets expanded to a list of files. For example, rsync might see:

rsync -a --exclude a.o b.o c.o my_src_dir/ orion:my_src_dir

The "--exclude" option only expects a single file specification.

SECOND TRY: FAIL

So maybe I can enclose $EXCL in double quotes:

rsync -a "$EXCL" my_src_dir/ orion:my_src_dir

This passes "--exclude *.o" as a *single* parameter. But rsync expects "--exclude" and the file spec to be two parameters, so it doesn't work either.

THIRD TRY: FAIL

Finally, maybe I can force quotes inside the EXCL symbol:

EXCL='--exclude "*.o"'  # default
... # code that might change EXCL
rsync -a $EXCL my_src_dir/ orion:my_src_dir

This almost works, but what rsync sees is:

rsync -a --exclude "*.o" my_src_dir/ orion:my_src_dir

It thinks the double quotes are part of the file name, so it won't exclude the intended files.

EVAL TO THE RESCUE

The solution is to use eval:

EXCL='--exclude "*.o"'  # default
... # code that might change EXCL
eval "rsync -a $EXCL my_src_dir/ orion:my_src_dir"

The shell does symbol substitution, so this is what eval sees:

rsync -a --exclude "*.o" my_src_dir/ orion:my_src_dir

And eval will re-process that string, including stripping the double quotes, so this is what rsync sees:

rsync -a --exclude *.o my_src_dir/ orion:my_src_dir

which is exactly correct.

P.S. - if anybody knows of a better way to do this, let me know!

EDIT: The great Sahir (one of my favorite engineers) pointed out a shell feature that I didn't know about:;

Did you consider setting noglob? It will prevent the shell from expanding '*'. Something like:

    EXCL='--exclude *.o' # default
    set -o noglob
    rsync -a $EXCL my_src_dir/ orion:my_src_dir
    set +o noglob

I absolutely did not know about noglob! In some ways, I like it better. The goal is to pass the actual star character as a parameter, and symbol substitution is getting in the way. Explicitly setting noglob says, "hey shell, I want to pass a string without you globbing it up." I like code that says exactly what you mean.

One limitation of using noglob is that you might have a command line where you want parts of it not globbed, but other parts globbed. The noglob basically operates on a full line. So you would need to do some additional string building magic to get the right things done at the right time. But the same thing would be true if you were using eval. Bottom line: the shell was made powerful and flexible, but powerful flexible things tend to have subtle corner cases that must be handled in non-obvious ways. No matter what, a comment might be nice.

FULL DISCLOSURE: I tried it and it didn't work as expected. It's probably related to all the crazy quoting. Since my "eval" solution worked, I didn't invest the time to figure out why the "noglob" method didn't work. So I'm still using eval even though noglob is arguably better for this purpose.

Perl Faster than Grep

So, I've been crawling through a debug log file that is 195 million lines long. I've been using a lot of "grep | wc" to count numbers of various log messages. Here's some timings for my Macbook Pro:

$ time cat dbglog.txt >/dev/null
real 0m35.423s

$ time wc dbglog.txt
195177935 1177117603 28533284864 dbglog.txt
real 1m44.560s

$ time egrep '999999' dbglog.txt
real 7m39.737s

(For this timing, I chose a pattern that would *NOT* be found.)

On the Macbook, the man page for fgrep claims that it is faster than grep. Let's see:

$ time fgrep '999999' dbglog.txt

real 7m11.365s

Well, I guess it's a little faster, but nothing to brag about.

Then I wanted to create a histogram of some findings, so I wrote a perl script to scan the file and create the histogram. Since it performed regular expression matching on every line, I assumed it would be a little slower than grep, since Perl is an interpreted language.

$ time ./count.pl dbglog.txt >count.out

real 3m9.427s

WOW! Less than half the time!

So I created a simple grep replacement: grep.pl. It doesn't do any histogramming, so it should be even faster.

$ time grep.pl '999999' dbglog.txt

real  2m8.341s

Amazing. Perl grep runs in less than a third the time of grep.

For small files, I bet Perl grep is slower starting up. Let's see.

$ time echo "hi" | grep 9999

real        0m0.051s

$ time echo "hi" | grep.pl 9999

real        0m0.113s

Yep. Grep saves you about 60 milliseconds. So if you had thousands of small files to grep, it might be faster to use grep.

See https://github.com/fordsfords/grep.pl

UPDATE:

I got another big log file today (70 million lines) and saw something pretty surprising given my initial findings.

See https://blog.geeky-boy.com/2021/07/more-perl-grep-performance.html

Beginner Shell Script Examples

As I've mentioned, I am the proud father of a C.H.I.P. single-board computer.  I've been playing with it for a while, and have also been participating in the community message board.  I've noticed that there are a lot of beginners there, just learning about Linux.  This collection of techniques assumes you know the basics of shell scripting with BASH.

One of the useful tools I've written is a startup script called "blink.sh".  Basically, this script blinks CHIP's on-board LED, and also monitors a button to initiate a graceful shutdown.  (It does a bit more too.)  I realized that this script demonstrates several techniques that CHIP beginners might like to see.

The "blink.sh" script can be found here: https://github.com/fordsfords/blink/blob/gh-pages/blink.sh.  For instructions on how to install and use blink, see https://github.com/fordsfords/blink/tree/gh-pages.

The code fragments included below are largely extracted from the blink.sh script, with some simplifications.

NOTE: many of the commands shown below require root privilege to work.  It is assumed that the "blink.sh" script is run as root.


1. Systemd service, which automatically starts at boot, and can be manually started and stopped via simple commands.

I'm not an expert in all things Linux, but I've been told that in Debian-derived Linuxes, "systemd" is how all the cool kids implement services and startup scripts.  No more "rc.local", no run levels, etc.

Fortunately, systemd services are easy to implement.  The program itself doesn't need to do anything special, although you might want to implement a kill signal handler to cleanup when the service is stopped.

You do need a definition file which specifies the command line and dependencies.  It is stored in the /etc/systemd/system directory, named "<sevrice_name>.service".  For example, here's blink's definition file:

$ cat /etc/systemd/system/blink.service 

# blink.service -- version 24-Jul-2016

# See https://github.com/fordsfords/blink/tree/gh-pages

[Unit]

Description=start blink after boot

After=default.target

[Service]

Type=simple

ExecStart=/usr/local/bin/blink.sh

[Install]

WantedBy=default.target

When that file is created, you can tell the system to read it with:

sudo systemctl enable /etc/systemd/system/blink.service

Now you can start the service manually with:

sudo service blink start

You can manually stop it with:

sudo service blink stop

Given the way it is defined, it will automatically start at system boot.


2. Shell script which catches kill signals to clean itself up, including the signal that is generated when the service is stopped manually.

The blink script wants to do some cleanup when it is stopped (unexport GPIOs).

trap "blink_stop" 1 2 3 15

where "blink_stop" is a Bash function:

blink_stop()

{

  blink_cleanup

  echo "blink: stopped" `date` >>/var/log/blink.log

  exit

}

where "blink_cleanup" is another Bash function.

This code snippet works if the script is used interactively and stopped with control-C, and also works if the "kill" command is used (but not "kill -9"), and also works when the "service blink stop" command is used.


3. Shell script with simple configuration mechanism.

This technique uses the following code in the main script:

export MON_RESET=

export MON_GPIO=

export MON_GPIO_VALUE=0  # if MON_GPIO supplied, default to active-0.

export MON_BATTERY=

export BLINK_STATUS=

export BLINK_GPIO=

export DEBUG=

if [ -f /usr/local/etc/blink.cfg ]; then :

  source /usr/local/etc/blink.cfg

else :

  MON_RESET=1

  BLINK_STATUS=1

fi

The initial export commands define environment variables with default values.  The use of the "source" command causes the /usr/local/etc/blink.cfg to be read by the shell, allowing that file to define shell variables.  In other words, the config file is just another shell script that gets included by blink.  What does that file contain?  Here are its installed defaults:

MON_RESET=1       # Monitor reset button for short press.

#MON_GPIO=XIO_P7   # Which GPIO to monitor.

#MON_GPIO_VALUE=0  # Indicates which value read from MON_GPIO initiates shutdown.

MON_BATTERY=10    # When battery percentage is below this, shut down.

BLINK_STATUS=1    # Blink CHIP's status LED.

#BLINK_GPIO=XIO_P6 # Blink a GPIO.

4. Shell script that controls CHIP's status LED.

Here's how to turn off CHIP's status LED:

i2cset -f -y 0 0x34 0x93 0

Turn it back on:

i2cset -f -y 0 0x34 0x93 1

This obviously requires that the i2c-tools package is installed:

sudo apt-get install i2c-tools

5. Shell script that controls an external LED connected to a GPIO.

The blink program makes use of the "gpio_sh" package.  Without that package, most programmers refer to gpio port numbers explicitly.  For example, on CHIP the "CSID0" port is assigned the port number 132.  However, this is dangerous because GPIO port numbers can change with new versions of CHIP OS.  In fact, the XIO port numbers DID change between version 4.3 and 4.4, and they may well change again with the next version.

The "gpio_sh" package allows a script to reference GPIO ports symbolically.  So instead of using "132", your script can use "CSID0".  Or, if using an XIO port, use "XIO_P0", which should work for any version of CHIP OS.

Here's how to set up "XIO_P6" as an output and control whatever is connected to it (perhaps an LED):

BLINK_GPIO="XIO_P6"

gpio_export $BLINK_GPIO; ST=$?

if [ $ST -ne 0 ]; then :

  echo "blink: cannot export $BLINK_GPIO"

fi

gpio_direction $BLINK_GPIO out

gpio_output $BLINK_GPIO 1    # turn LED on

gpio_output $BLINK_GPIO 0    # turn LED off

gpio_unexport $MON_GPIO      # done with GPIO, clean it up

6. Shell script that monitors CHIP's reset button for a "short press" and reacts to it.

The small reset button on CHIP is monitored by the AXP209 power controller.  It uses internal hardware timers to determine how long the button is pressed, and can perform different tasks.  When CHIP is turned on, the AXP differentiates between a "short" press (typically a second or less) v.s. a long press (typically more than 8 seconds).  A "long" press triggers a "force off" function, which abruptly cuts power to the rest of CHIP.  A "short" press simply turns on a bit in a status register, which can be monitored from software.

REG4AH=`i2cget -f -y 0 0x34 0x4a`  # Read AXP209 register 4AH

BUTTON=$((REG4AH & 0x02))  # mask off the short press bit

if [ $BUTTON -eq 2 ]; then :

  echo "Button pressed!"

fi

Note that I have not figured out how to turn off that bit.  The "blink.sh" program does not need to turn it off since it responds to it by shutting CHIP down gracefully.  But if you want to use it for some other function, you'll have to figure out how to clear it.

7. Shell script that monitors a GPIO line, presumably a button but could be something else, and reacts to it.

MON_GPIO="XIO_P7"

gpio_export $MON_GPIO; ST=$?

if [ $ST -ne 0 ]; then :

  echo "blink: cannot export $MON_GPIO"

fi

gpio_direction $MON_GPIO in

gpio_input $MON_GPIO; VAL=$?

if [ $VAL -eq 0 ]; then :

  echo "GPIO input is grounded (0)"

fi

gpio_unexport $MON_GPIO      # done with GPIO, clean it up

8. Shell script that monitors the battery charge level, and if it drops below a configured threshold, reacts to it.

This is a bit more subtle that it may seem at first.  Checking the percent charge of the battery is easy:

REGB9H=`i2cget -f -y 0 0x34 0xb9`  # Read AXP209 register B9H

PERC_CHG=$(($REGB9H))  # convert to decimal

But what if no battery is connected?  It reads 0.  How do you differentiate that from having a battery which is discharged?  I don't know of a way to tell the difference.  Another issue is what if a battery is connected and has low charge, but it doesn't matter because CHIP is connected to a power supply and is therefore not at risk of losing power?  Basically, "blink.sh" only wants to shut down on low battery charge if the battery is actively being used to power CHIP.  So in addition to reading the charge percentage (above), it also checks the battery discharge current:

BAT_IDISCHG_MSB=$(i2cget -y -f 0 0x34 0x7C)

BAT_IDISCHG_LSB=$(i2cget -y -f 0 0x34 0x7D)

BAT_DISCHG_MA=$(( ( ($BAT_IDISCHG_MSB << 5) | ($BAT_IDISCHG_LSB & 0x1F) ) / 2 ))

CHIP draws over 100 mA from the battery, so I check it against 50 mA.  If it is lower than that, then either there is no battery or the battery is not running CHIP:

BAT_IDISCHG_MSB=$(i2cget -y -f 0 0x34 0x7C)

BAT_IDISCHG_LSB=$(i2cget -y -f 0 0x34 0x7D)

BAT_DISCHG_MA=$(( ( ($BAT_IDISCHG_MSB << 5) | ($BAT_IDISCHG_LSB & 0x1F) ) / 2 ))

if [ $BAT_DISCHG_MA -gt 50 ]; then :

  REGB9H=`i2cget -f -y 0 0x34 0xb9`  # Read AXP209 register B9H

  PERC_CHG=$(($REGB9H))  # convert to decimal

  if [ $PERC_CHG -lt 10 ]; then :

    echo "Battery charge level is below 10%"

  fi

fi

Cute "sed" Trick: rotate file

I wanted a script that would invoke a process, passing in a port number from a circular pool of ports.  When the process exits and restarts, I want the script to pass in the *next* port from the pool.

    PORT=`head -1 port_file.txt`

    sed -i -e '1h;1d;$G' port_file.txt

In the above "sed" command:

• The "-i" causes the file "port_file.txt" to be edited in-place.
• The "1h" sed command yanks the first line into the "hold" space.
• The "1d" sed command deletes the first line (prevents it from being output).
• The "$G" command appends the hold space after the last line of the file.

Thus, given that "port_file.txt" contains:

    12000

    12001

    12002

    12003

the above two commands will leave the file with:

    12001

    12002

    12003

    12000

and "PORT" set to 12000.

Configure a Cron Job with a Wiki

I have some periodic cron jobs that need extra configuration.  For example, one of them generates a report on bug statistics on a code branch basis.  So I need to tell it which code branches to process.  I could just put the list of branch tags on the command line of the report generator, and just use "crontab -e" while logged in to modify it.  However, I want anybody to be able to maintain the list, without having to know my password or the syntax for crontab.

It turns out that we installed Mediawiki locally for our own internal wiki.  So I created a wiki page with a table listing the code branches that are active.  Then I wrote a script which uses "curl" to fetch that wiki page and parse out the branches.  This gives me a nice web-based GUI interface to the tool that everybody is already familiar with.  Everybody here knows how to use Wikipedia, so anybody can go in and change the list of branches.

After doing some additional development, I wanted to be able to include additional configuration for the cron job which I didn't particularly want displayed on the wiki page.  You can use <!--HTML-style comments--> with Mediawiki and it won't display it on the page.  Unfortunately, it completely withholds the comment from the html of the page.  I.e. you can't see it even if you display source of the page.  You only see the comment when you edit the page.

So here's what I ended up with:

#!/bin/sh

# all.sh

# Generate QA report for all active releases.  Runs via cron nightly.

date

# Read wiki page and select rows in the release table ("|" in col 1).

# Uses "action=edit" so that are included (for option processing).

curl 'http://localwiki/index.php?title=page_title&action=edit' | egrep "^\|" >all.list

# read the contents of all.list, line at a time.  Each line is an entry in the table of active releases.

while read ILINE; do :

    # Extract target milestone (link text of web-based report page).

    TARGET_MILESTONE=`echo "$ILINE" | sed -n 's/^.*_report.html \(.*\)\].*$/\1/p'`

    # Extract the (optional) set of command-line options.

    OPTS=`echo "$ILINE" | sed -n 's/^.*--OPTS:\([^:]*\):.*$/\1/p'`

    eval ./qa_report $OPTS \"$TARGET_MILESTONE\" 2>&1

done <all.list

The "sed" commands use "-n" to suppress printing of lines to stdout.  Adding a "p" suffix to the end of a sed command forces a print, if the command is successful.  So, for example, the line:
    OPTS=`echo "$ILINE" | sed -n 's/^.*--OPTS:\([^:]*\):.*$/\1/p'`
If the contents of $ILINE does not contain match the pattern (i.e. does not have an option string), the "s" command is not successful and therefore doesn't print, leaving OPTS empty.

One final interesting note: the use of "eval" to run the qa_report.sh script.  Why couldn't you just use this?
    ./qa_report $OPTS "$TARGET_MILESTONE" 2>&1

Let's say that $TARGET_MILESTONE is "milestone" and the contents of $OPT is:

    -a "b c" -d "e f"

If you omit the "eval", you would expect the resulting command line to be:

    ./qa_report -a "b c" -d "e f" "milestone" 2>&1

I.e. the qa_report tool will see "b c" as the value for the "-a" option, and "e f" as the value for the "-d" option.  But the shell doesn't work this way.  The line:

    ./qa_report $OPTS \"$TARGET_MILESTONE\" 2>&1

will expand $OPTS, but it won't group "b c" as a single entity for -a.  Without "eval", the "-a" option will only see the two-character value "b (with the quote mark).  I found a good explanation for this; the short version is that the shell does quote processing before it does symbol expansion.  So essentially, the thing you need to do is have the shell parse the command line twice.

The "eval" form of the command works like this:
    eval ./qa_report $OPTS \"$TARGET_MILESTONE\" 2>&1
First the shell looks at this command line and parses it with "eval" as the command and the rest as "eval"s parameters.  It does the symbol substitution.  Thus, the thing that gets passed to "eval" is:
    ./qa_report -a "b c" -d "e f" "milestone"
What does the eval command do with that?  It passes it to the shell for parsing!  In this pass, "./qa_report" is the command, and the rest are the parameters.  Since the shell is parsing it from scratch, it will group "b c" as a single entity, letting the "-a" option pick it up as a single string.