Precision Tools from Innocraftsman

In addition to being a software developer, I'm also a little bit of a DIY handyman. Not a very good one - I'm not one to make things *look* good, just function well. But mostly, I just love tools.

A while ago, I got Innocraftsman's mini drill press. Note, this is a VERY small drill press, intended for precision work. I splurged and got the model with the brushless DC motor and the better belt. I've only used it once, but again, I love the precision and quality of the tool. And my wife, an artist and a maker, has used it quite a few times. Color us a satisfied customer.

Another thing I didn't have is a nice vise. I have a cheap one with a suction cup bottom, and it's OK, but sometimes the heart wants what it wants. So I ordered Innocraftsman's 2-Inch Universal Precision Sine Adjustable Angle Vise. No, I don't need it. I don't do the kind of work it was intended for. But dammit, I WANT it. :-)

So, what did I receive? A box of ballpoint pens. They have a custom inscription on them in Spanish. And there's a lot of them, I would guess 500. Obviously shipped from China, so something got mixed up in the process.

I contacted Innocraftsman and we did a little bit of detective work - I found out where UPS picked it up from. The logistics company presumably got a large container from China and re-shipped each product, but put my label on the wrong box. Somebody else presumably got my vise, and we didn't have a lead on where it might have gone.

Fortunately, Innocraftsman didn't make a fuss, they just shipped me a new one, and it arrived yesterday. Even though I haven't used it yet, I just love looking at it and manipulating its controls.

AND I have a box of pens. Anybody want a pen? ;-)

So credit to Innocraftsman for making awesome precision tools, and credit for making good on a problem that wasn't their fault. Thanks!

Gabriel's horn is not a paradox.

 Gabriel's horn is not a paradox.

The "Gabriel's horn paradox" (sometimes called "the painter's paradox") is called a paradox because the result feels contradictory. I think I can explain it in a way that is intuitive; the feeling of contradiction will disappear.

This post assumes you already have an understanding of the paradox and the proof that Gabriel's horn has a finite volume and an infinite surface area. If not, then see https://tomrocksmaths.com/wp-content/uploads/2022/05/gabriels-horn.pdf for a written description, or https://www.youtube.com/watch?v=yZOi9HH5ueU for a video treatment.

The real problem is that the paradox description contains a mistake.

"Make a horn shape by the rotation around the X axis of the curve y=1/x from x=1 to x=infinity. The volume of this horn is finite, and so can be filled with a finite amount of paint. But the surface area of the horn is infinite, and therefore requires an infinite amount of paint. How can painting it require more paint than filling it?"

The mistake is in the claim that painting it requires infinite paint.

Let's say I have a bucket of paint. I can paint a wall with it. How thick is the layer of paint? Depends on how heavily I painted it on. Maybe 1mm?

So now I'm asked to paint a wall that's twice as long but use the same amount of paint. No problem, I just lay down a thinner layer - 0.5 mm.

Double the length again, halve the thickness, and the same bucket paints that even larger wall.

Now there is no such thing as infinitely big things, but we like to talk about them in math. We generally talk about infinities in terms of a limit. I.e. what does an equation do as you "approach" infinity. Take the sequence:

sum1 = 1/2 + 1/4 + 1/8 + 1/16 ...

For a finite number of terms, the value of sum1 approaches 1. We say that the *infinite* series converges to exactly 1. In contrast, consider the sequence:

sum2 = 1/2 + 1/3 + 1/4 + 1/5 ...

For a finite number of terms, the value of sum2 does not approach any particular value. As you add more terms, the value will grow without an upper limit. We say that the infinite series diverges to infinity.

These concepts of converging and diverging infinite series are why Gabriel's horn can have both finite volume and infinite surface area. (See the above links for formal proofs.)

My point is that you can intuitively see that a finite amount of paint can cover a surface that approaches infinity, simply by reducing the thickness of the paint layer to approach zero. A bucket of paint can both fill the horn and paint it.

The reason the original description feels like a contradiction is that in real life, you can't have an arbitrarily thin layer of paint. There's a minimum practical layer thickness, and therefore a maximum practical area of coverage. But this is a math puzzle, where practical matters don't intrude. We can have values that approach infinity and approach zero, no problem.

Switching from Painting to Filling

Here's an interesting thought experiment. Imagine having the horn in front of you, extending downward towards infinity. We have to imagine that gravity is constantly downward for the entire length of the horn, which does not correspond to anything our universe could do, but work with me here.

Now imagine I have my bucket of paint and I let one drop fall into the horn. Let's imagine it doesn't actually stick to the surface of the horn and simply slides down the side of the horn. At some point, the diameter of the horn will equal the diameter of the drop, so the drop will continue falling, but will reshape itself into a thin thread of paint, falling down this ever-narrowing horn. As it continues to fall, the diameter will continue to decrease, meaning the length will continue to increase as it stretches out. So the front of the paint thread will be moving down faster than the back of the thread. But both will continue moving down.

Forever.

Here's the interesting thing: as time tends toward infinity, the front of the paint thread will keep traveling down the horn toward infinity. But the rear will travel more and more slowly, approaching a "convergence" point that it won't ever cross. This is because the original paint drop has a finite volume, and that volume will exactly fill a section of the horn starting at that convergence point and continuing to infinity. 

If you speed up time, you would see the front of the paint thread rocketing down the horn and the back of the paint thread slowing down and appearing to stop.

I find that thought experiment fascinating.

Nick Cave has Nothing to Fear

Nick Cave doesn't like ChatGPT.

Somebody asked Chat to compose a song in the style of Nick Cave. Nick didn't like it, calling it "replication as travesty" among other things.

I think Nick and other successful singer-songwriters have nothing to fear.

First of all, replication is nothing new. Beginner musicians imitate the styles of their favorite artists all the time. The good ones eventually find their own voices. But what about the wannabes that just get REALLY good at emulating their hero's style? Think "tribute band". Nick doesn't fear them. Nick Cave fans will buy Nick's music, even if a tribute band sounds just like him. Having that tribute band use an AI doesn't change that.

It might be a little dicier if somebody uses an AI to compose a song/painting/whatever in the style of a long-dead artist and claims that it is a newly-found genuine creation of the original artist. This is also nothing new. It's called forgery, and people have been dealing with that for as long as there has been an art market. I can't see reducing the cost of entry into the forgery profession will lead to a lot more fraud being perpetrated. If anything, it will make consumers even more suspicious of unlikely "discoveries", which is probably a good thing.

Nick's primary complaint seems to be that good music that touches a human's heart can only come from another human heart (usually a tortured one). Bad news, Nick. There's plenty of successful music out there that does not come from the creator's heart, and has no intention of touching the listener's heart. In my youth, they called it "bubble gum music". Cheery, maybe danceable, maybe a catchy riff that you find yourself humming. Think Monkeys or TV commercials. I suspect Nick wouldn't care much one way or the other if that music started coming from AIs instead of good-but-not-great-musicians-who-need-to-eat.

Is serious music in danger of being AI generated?

Well ... maybe? There are plenty of successful singers who are not songwriters. They mostly get their songs from non-performing songwriters. I'm sure that some of those songwriters are tortured artists whose blood and sweat come out in their songs. A lot of others are fairly non-creative mediocre songwriters who figured out a formula and got good at imitation. Give an uninspired song to a really successful singer, and you can have a hit. Is this something that bothers serious songwriters? Probably. There are way more songwriters, both serious and formulaic, than there are successful singers. Maybe the uninspired songwriters have something to fear with AI replacing them. But is anybody that worried about them? I suspect not.

But what about serious non-performing songwriters who really do pour their blood, sweat, and tears into their work. Will AIs replace them?

Maybe. But they have a hard enough time already getting their songs on the air. I have a hard time believing it will make much of a difference. If .00001% of the population lose their jobs doing what they love, I guess that's kind of sad, but I wouldn't call it a tragedy. The number of artisans creating elegant and artistic horse saddles is a small fraction of what it was 150 years ago. Times change.

Cheating with AI?

I saw an article about a teacher who got an essay from a student that was well-written. Maybe too-well written. Turns out the student used an AI to write it, and turned it in as their own work. The teacher (and the article) predicted massive changes to how school work is assigned, performed, and evaluated.

I'm not sure I understand why.

Cheat Your Way Through School?

Cheating has always been with us. When I was a student, that consisted of copying (verbatim or paraphrasing) from magazines, encyclopedias, or the smart kid in a different class. And while many kids got caught, many others did not. Teachers used to tell us that cheating didn't actually help us prepare for our futures, but kids are too now-focused to understand or care about that. We just knew that our parents would take away our TV privileges if we got a bad report card, so some kids cheated.

The Internet supposedly changed all that since it became trivially easy to cheat. As though lowering the effort would open the floodgates. But it didn't. Sure, you can buy essays on-line now, which makes it easier to cheat, but most kids still don't.

And now AI is about to change all that since it is even more trivially easy (and cheaper) to cheat.

I don't buy it. Cheaters are going to cheat, and it's not obvious to me that making it easier and cheaper to cheat will make a lot more kids into cheaters. 

Cheat Your Way Through Career?

And besides, why do we care? If cheaters make it all the way through college with much higher grades than are deserved, they will more-or-less reach their true level when they start their careers. I've had to fire some programmers who I wonder whether they ever wrote a line of code in their lives. Did they cheat their way through school? Or did the schools just do a bad job of preparing programmers? I don't know, and I don't care. I managed to hire some excellent programmers in spite of getting a few duds. And I suspect the same basic pattern exists in most careers.

I'll focus my discussion on the career of computer programming, but I suspect many of the concepts will apply to other careers.

Maybe the AIs are getting so good that a poor programmer that is good at cheating will produce just as good results as the excellent programmer down the hall. How is that fair? And does it even matter?

My programmers take poorly-described requirements and figure out what the user needs, and then figure out how to incorporate those needs into our existing product. Cheaters can't do that even if they have a great AI at their disposal.

In fact, even that is not what my senior programmers do. They figure out what our users want before the users do. When 29West was just getting started (2003-ish), I don't think there was such a thing as a brokerless general pub-sub messaging system. The financial services industry wanted low latency, but also wanted the flexibility of pub-sub. The idea 29West came up with was to combine peer-to-peer with reliable multicast and the pub-sub model. Figuring out how to do that required dreaming up new ways of doing things. Even if a really good AI existed back then, it would not have been trained on it.

I guess what I'm saying is that the most advanced AI technology available today is still based on the concept of training the AI with a lot of examples. It will be able to report the state of the art, but I can't see it advancing the state of the art. 

When Does Cheating Stop Being Cheating?

There was a period of time when I was in school when we couldn't use a calculator during a math test. You had to do the arithmetic by hand (and show your work). I suspect that still exists for a month or two when kids first learn what arithmetic is, but I suspect that calculators are now standard issue for even very young students. Is that bad?

I remember hearing people complain. "What if your batteries die? How will the supermarket employee add up your total?" Today, if a store's cash register goes down, commerce stops. And it's not because the employees can't do sums in their heads.

I also remember when poor spelling and grammar were impediments to career advancement. I guess it still is -- if you send me an email with lots of misspellings, I will think a little less of you. With spelling checkers built right into the email client, what's your excuse for not using it? (My mother-in-law used to disapprove of modern schooling where Latin is no longer a required subject. Her point was that learning Latin made you better at spelling. My point is, why bother?)

Remember cursive writing? Does anybody under 30 still use it? Do we still need to be good at shoeing horses? Starting fires with two sticks?

Do we really need everybody to be good at writing essays? Maybe it's time to consign that to the computer as well.

And yes, I know that writing essays is supposed to be a tool for exercising research skills and critical thinking. But is it really? Isn't the essay more of a measurement tool? I.e. if you did a good job of researching and thinking critically, then supposedly that will be reflected in the quality of your essay. But does that really work?

I don't know. And I've definitely strayed out of my area of expertise; I'll stop mansplaining now.

P.S.

I cut and pasted this post into ChatGPT and asked it to rewrite it better. It certainly shortened it, and included most of my main points. But it also missed a few points I consider important. And it made it a lot more boring, IMO. Then again, I always have liked to hear myself speak, so I'm biased.

OpenAI GPT-3: Initial Impressions

So, ChatGPT is at capacity right now, and I'm sure not going to sit here and hit refresh every few seconds to try to get in. So I went to https://beta.openai.com/examples and tried a few.

My impressions: disappointing given all the hype, but if I hadn't seen the hype, I would be reasonably impressed.

[UPDATE: I've since gotten into the real ChatGPT, and I must say it is more impressive than the sandbox I played with in this post. The sandbox is based on GPT-3, while ChatGPT is GPT-3.5.]

In the following examples, I am including output from OpenAI's GPT-3 model. In accordance with OpenAI's wishes, my name is Steve Ford, and the text shown in green is generated by an AI.

Analogy maker

My entry is first, followed by the AI response in green.

Get the logic right, then optimize.

It's like building a house: first make sure the foundation is solid, then make it look nice.

Not bad.

Do unto others as you would have them do unto you.

It's like a golden rule of giving and receiving; treat others as you would like to be treated.

Um, I don't think you can use a definition as an analogy. The point of an analogy is to improve understanding of a concept by re-structuring it in different terms that the reader might be more familiar with.

Look out for number one.

It's like putting on your own oxygen mask before helping others on an airplane.

Impressively good and bad. Yes, it kind of understood what the phrase means, but it also completely misses the essence of the phrase. "Watch out for number one" is all about being selfish (Collins dictionary says, "If you are looking out for number one, you are thinking of yourself rather than considering other people"), whereas the analogy is about doing something that might seem selfish at first glance, but is actually in support of generosity and the greater good.

Pro tip: don't use GPT-3 to learn the meaning of idioms.

Do unto others ... and then split Give a hand to those in need ... and then walk away.

This one might not be fair; I wonder how many under-50-year-olds would get that reference. Anyway, the analogy is the exact opposite of the original.

Pro tip: don't use GPT-3 to learn about 50-year-old memes.

There are other example applications that I would like to play with, but who has time for that? Over time, I'll probably have more to say about AI.

Invert the Meaning

I took the analog maker settings and changed the command (or whatever that first line is called) to "Invert the meaning:".

Time flies like an arrow.

Time stands still like a rock.

Not idiomatic, but impressive.

Smoking is bad for you.

Not smoking is bad for you.

Yep, it will lie if you tell it to.

Write an Essay

Took the analog maker settings and changed the command to "Write an essay:"

Why structured programming is bad.

Structured programming is a programming paradigm that relies on breaking down a program into smaller, manageable parts. It is a popular approach to programming because it is easy to understand and can be used to create complex programs. However, there are a number of drawbacks to structured programming that can make it a less than ideal choice for certain projects.

The first issue with structured programming is that it can be difficult to maintain. Because the code is broken down into smaller parts, it can be difficult to keep track of how the parts fit together. This can lead to bugs and errors that can be difficult to debug. Additionally, if changes need to be made to the program, it can be difficult to make sure that all the parts are updated correctly.

Another issue with structured programming is that it can be inflexible. Structured programming relies on a set of predetermined rules and structures, which can limit the options for solving a problem. If a programmer needs to come up with a creative solution, structured programming may not be the best choice.

Finally, structured programming can be slow. The small parts of a program can take a long time to process, especially if there are a lot of them. This can lead to programs that take a long time to run, which can be a problem for projects that require fast results.

In conclusion, structured programming can be a useful approach for certain projects, but it is not without its drawbacks. It can be difficult to maintain, inflexible, and slow. For projects that require creative solutions or fast results, it may be better to use an alternative approach.

Information in the Noise

Wow, a non-math nerd posting twice about math. What's that about?

Derek Muller of Veritasium posted a video about the 100 prisoners puzzle (I like "puzzle" in this context better than "riddle" or "problem"). Unlike my earlier post, I have no complaints about this video. Derek is one of the top-tier educational YouTubers, and he did a fantastic job of explaining it. (As before, I'm not going to explain it here; watch his video. Seriously, just watch it.)

So why do I feel the need to comment? I guess I feel I have a small but interesting (to me) tidbit to add.

Derek et al. describe the puzzle's "linked list" solution (my name) as giving a counter-intuitive result, and I guess I have to agree. The numbers are distributed to the boxes randomly, so how could any strategy give a prisoner a better chance of success than random selection? IT'S RANDOM!!!!!!

AN INTUITIVE UNDERSTANDING

And here's my tidbit: it's not as random as it seems. For this puzzle, the numbers are assigned randomly to boxes, without replacement. I.e., you won't find a given number in more than one box, and no number between 1 and 100 is skipped. This is obvious for the setup of the puzzle, but randomizing without replacement puts constraints on the system. Those constraints add information to the noise.

If prisoner number 13 randomly opens box 52, he knows he has a one in 100 chance of seeing his number in that box. He opens it and sees the number 1. He now knows FOR SURE that no other box has the number 1 in it. So his second random choice will have a one in 99 chance of being his number. Each choice gives some information that affects the probability of the next choice. (I.e., the samples are not independent.)

It is these constraints that lead directly to the cycles that are at the heart of the puzzle. And clever people have calculated the probability of having a cycle greater than 50 to be about 0.688. So the "linked list" strategy has ~= 0.312 probability of the prisoners being set free. That's the point of Derek's video.

Let's ruin the puzzle for a moment. Let's assign a random number between 1 and 100 to each box with replacement. It's entirely possible, even probable, that you'll have duplicates (the same number in more than one box) and skips (a number that is not in any box). One effect of this change is that the numbers will no longer necessarily be arranged in cycles. You can have many numbers NOT in a cycle. So the "linked list" solution to the puzzle doesn't improve your chances of survival over pure chance. Getting rid of the "without replacement" constraint removes the information from the noise.

This is how I get an intuitive feeling that you can have a much higher probability of success with the "linked list" solution to the original puzzle - you're taking advantage of the information that's in the noise.

WITH REPLACEMENT

What about my ruined version, where the numbers are assigned to boxes with replacement? To start with, let's calculate the probability that you get a distribution of numbers in boxes that is even possible for the prisoners to win (i.e., every number 1-100 is assigned exactly once). My probability-fu is weak, but I'll try. I think it is (100!)/(100**100) ~= 9.33e-48. Wow, that's a really low probability.

On the off chance that you get a solvable distribution, the probability of success with the linked list solution is ~= 0.312. So the total probability of success for my ruined version, WITH the linked list solution, is ~= 6.4e-43. If instead the prisoners choose their boxes randomly, then it's ~= 7.36e-73.

The prisoners had better appeal to Amnesty International.

There is no Vase

 I'm not a math nerd, so I probably shouldn't be posting on this subject. But when has a lack of expertise ever stopped me from having an opinion?

I just watched the Up and Atom video: An Infinity Paradox - How Many Balls Are In The Vase? In it, Jade describes the Ross–Littlewood paradox related to infinite pairings. I liked the video but was not satisfied with the conclusion.

I won't give the background; if you're interested in this post, go watch the video and skim the Wikipedia article. Basically, she presents the "Depends on the conditions" solution (as described in the Wikipedia article) without mentioning the "underspecified" and "ill-formed" solutions. And I guess that's an OK choice since the point of her video was to talk about infinities and pairings. But she kept returning to the question, "how many balls are there *actually*?"

Infinity math has many practical applications, especially if the infinity is related to the infinitely small. An integral is frequently described as the sum of the areas of rectangles under a curve as the width of the rectangles becomes infinitesimal - i.e., approaches zero. This gives a mathematically precise calculation of the area. Integrals are a fundamental tool for any number of scientific and engineering fields.

But remember that math is just a way of modeling reality. It is not *really* reality.

There is no such thing as an infinitesimal anything. There is a minimum distance, a minimum time, and the uncertainty principle guarantees that even as you approach the minimum in one measure, your ability to know a different measure decreases. When the numbers become small enough, the math of the infinitesimal stops being an accurate model of reality, at least not in the initially intuitive ways.

But they are still useful for real-world situations. Consider the paradox of Achilles and the tortoise, one of Zeno's paradoxes. (Again, go read it if you don't already know it.) The apparent paradox is that Achilles can never catch up to the tortoise, even though we know through common experience that he will catch up with and pass the tortoise. The power of infinity math is that we can model it and calculate the exact time he passes the tortoise. The model will match reality ... unless an eagle swoops down, grabs the tortoise, and carries it across the finish line. :-)

But models can break down, even without eagles, and a common way for infinity models to break down is if they don't converge. 1/2 plus 1/4 plus 1/8 plus 1/16 ... converges on a value (1). As you add more and more terms, it approaches a value that it will never exceed with a finite number of terms. So we say that the sum of the *infinite* series is *equal* to the limit value, 1 in this case. But what about 1/2 plus 1/3 plus 1/4 plus 1/5, etc.? This infinite series does NOT converge. It grows without bound. And therefore, we cannot claim that it "equals" anything at infinity. We could claim that the sum equals infinity, but this is not well defined since infinity is not a number.

Here's a similar train of thought. What is 1/0? If you draw a graph of 1/X, you will see the value grow larger and larger as X approaches 0. So 1/0 must be infinity. What is 0 * (1/0)? Again, if you graph 0 * (1/X), you will see a horizontal line stuck at zero as X approaches 0. So I guess that 0 * (1/0) equals 0, right? Not so fast. Let's graph X * (1/X). That is a horizontal line stuck at 1. So as X approaches 0, X * (1/X) equals 1. So 0 * 1/0 equals 1. WHICH ONE IS RIGHT???????? What *really* is 0 * (1/0)?

The answer is that the problem is ill-formed. The 1/X term does not converge. The value of 1/0 is not "equal to infinity", it is undefined. My train of thought above is similar to the fallacious "proof" that 1 equals 2. And it seems to me that the "proof" that the number of balls in the vase can be any number you want it to be is another mathematical fallacy.

The only way to model the original vase problems is to draw a graph of the number of balls in the vase over time. Even in the case where you remove the balls sequentially starting at 1, you will see the number of balls growing without bound as time proceeds. Since this function does not converge, you can't say that it "equals" anything at the end. But it tends towards infinity, so claiming that it equals some finite value *at* the end is another example of an invalid application of math to reality.

But I shouldn't complain. Jade used the "paradox" to produce an engaging video teaching about pairing elements in infinite sets. And she did a good job of that.

Keychron K2 Keyboard

 I mentioned buying a Keychron K2 keyboard almost two years ago, but that post was primarily about a different vendor's keyboard which was a fail.

I just bought a second Keychron K2 keyboard ("blue" switches), but not because of a problem. It's because the keyboard is wonderful, and I want a second one to keep at an alternate worksite.

The laptop keyboard on the 2017-vintage Macbook Pro is almost unusable. Really, even the 2015-vintage Air's laptop keyboard is not that great. I prefer a full-stroke "clicky" keyboard with good tactile feedback.

Enter the Keychron K2. Lots of nice features that I won't bother listing since I don't use most of them and the site explains them fine. I like the noisy "blue" style switches, but you can get quieter ones.

Also, it has all the Mac-specific special keys, right there (I don't like the touch-bar at the top of the Macbooks.) And I also like the compact size.

My only complaint is that the key caps are not dual-injected, which means that the paint can wear off the tops of the frequently-used keys (E and A). But this is a problem for most keyboards I use; I apparently have a heavy typing hand.

Count me as a satisfied customer.

Global STAC Live

I normally don't do product or vendor shilling on this blog. But I've always liked Peter Lankford at STAC, and I think he did a good job with this video:

www.STACresearch.com/spring2020

Yes, it's a funny advertisement, but instead of being just plain funny, it's actually pretty clever. And I don't just mean the jokes.

The Coronavirus has really disrupted professional conferences and events. Many are postponed, some are just plain cancelled. I assumed that STAC would be a casualty, but Peter is trying to make lemonade by making his 2020 summit virtual.

Now it's not like that has never been done before. Zoom has made quite a name for itself by saving the business meeting. (In fact, many meetings that used to be simple phone conferences have mysteriously and inexplicably drifted into Zoom meetings, I'm not sure why.)

So when I first heard that the 2020 summit was going virtual, I was like ho-hum ... another endless series of Zoom presentations. And it got me to thinking, what was going to be lost? The sales-oriented participants will of course miss glad-handing prospective customers. But what about engineers like me?

We'll miss getting together with our smart friends and colleagues that we haven't seen for a while, and maybe being introduced to their smart friends and colleagues for interesting conversation, war stories, and maybe even the occasional brilliant technical insight. Not going to get any of that out of a zoom presentation.

But Peter's video suggests that he was fully aware of that drawback, and he is experimenting with approaches to address it. I'm skeptical that I will be blown away by it, but I'm intrigued enough to give it a go. And whether it turns out revolutionary, or just another chat room, I respect Peter for effort.

Plus, I  just plain like the ad. :-)

UPDATE: June 3

At the end of day 2, I can safely say that the STAC Live event was well done. The speakers did a good job of dealing with the new format, and there were very few technical glitches. The system seemed to scale well.

In particular, it is nice to be able to replay sessions that you missed (or just couldn't pay attention to) the same afternoon. This is a clear *advantage* to this format as compared to a traditional in-person conference.

The technical content was also very good, but that's not what I want to talk about.

I'm a little sad to report that the social aspect turned out worse than I had hoped. It's not STAC's fault, nor the fault of Ubivent, the technology behind the virtual event. It's the fault of human nature.

When I travel to a venue to attend a function, I am engaged with the event. Yes, I sometimes check email, and will even sometimes hide in a corner to do an important call. But >90% of my time is spent being fully engaged with the people, both in and out of a session. So conversations happen naturally and productively. You get introduced to new people with similar interests. I made a very good friend at a conference a few years ago, just because I was there.

During STAC Live, the chat feature was there, but wasn't used very much. And like all chats, I might type something, and minutes would go by before the response came back. Why? BECAUSE WE'RE ALL BUSY, THAT'S WHY! During an in-person event, we are mostly engaged with our surroundings, but when attending remotely, we are balancing the virtual event with work. And work usually wins. And working from home means there's also family members and pets who need attention.

So even if the event technology were somehow more conducive to socializing, the attendees were not.

So, overall STAC was a good event, better than it had any right to be given the circumstances. And I've already learned some things that I plan to put into practice, so it's been well worth the time. But it being a virtual event does detract something meaningful. And I miss it.

Matias keyboard FAIL

Like many Mac owners, I don't like the chiclet keyboards. I like an old-fashioned mechanical keyboard. So I finally got one: a very clicky Matias keyboard. And while I would have preferred a smaller keyboard (I don't need the keypad), I was very happy with it. For about a month.

Then the spacebar started "bouncing". I.e. maybe 1 time in 20 I get two spaces instead of one. I experimented quite a bit, and it's not the autorepeat, and it's not *me* that's bouncing. I can slowly press the key, and bam - two spaces. Or I can type my normal (pretty fast) speed and get periodic bounces. (Sometimes I can go an hour without a bounce, but then they come back.)

So I contacted Matias support, and after leading me through a series of unsuccessful steps, they sent me a new one. It arrived with NO BOUNCE! Happy-happy.

For about a month. Now the bounces are back.

I know baseball says 3 strikes are an "out", but I'm only giving them 2. The Matias is going in the trash as soon as my new Keychron K2 (with blue switches) gets here.

Wish me luck!


Update: It's been over 3 months now, and I am VERY happy with Keychron K2!

My only complaint is that the blue tooth times out even if I use an external power supply. Timing out blue tooth is a good thing when running on battery, but is unnecessary when it has external power. (NOTE: there is a  simple solution: don't use blue tooth. The same cable I'm connecting to an external power supply can simply be plugged into my laptop, at which point it's a wired keyboard. But I have some reasons for not wanting to do that. Oh well, I'm still very happy with the K2.)

Update 2: It's been over 7 months now, and the Keycron K2 is still going strong! It's a pleasure to use.

Black Hole Revealed!

I am in awe of the results of the Event Horizon Telescope team in their image of M87*, the supermassive black hole at the center of a distant galaxy!  I can't help but be especially excited at the role that computer scientists played in the analysis and reconstruction of the data collected by the radio telescopes to produce the image.

Radio astronomers have been doing long-baseline interferometry for a while now to produce images.  But the challenges of the Event Horizon Telescope were beyond what the earlier processing algorithms could make sense of.  The software team led by Katie Bouman developed the CHIRP algorithm that kind of blows my mind.  It warms my heart that women in science are finally getting some of the recognition they deserve.  (It also depresses me greatly that misogynist trolls are getting some press; geeze, can't we just enjoy the accomplishment?)  Anyway, Katie did a Ted Talk a few years ago that gives some excellent explanation about the algorithm.

If you want some understanding of why the image looks the way it does, I think that Derek Muller's Veritasium video does the best job that I've seen.  He also has a good follow-up video.

I also really appreciated astrophysicist Becky Smethurst's video that explains why the results are important (it's more than just further supporting Einstein's theory of gravitation).

Some Random Password Generators are Bad

Good for you!  You're taking the advice of experts and clicking "Generate Password", resulting in 10 characters of gibberish.  There!  Now your password will take thousands of years to crack.

Um ... not necessarily.  Try a few days.


RAND() LIMITS ENTROPY TO 32 BITS

When using the pseudo-random number generator supplied by most language libraries, the entropy of the resulting password is limited to 32 bits!

Let's take XKCD's algorithm: ~2000 word dictionary, randomly select 4 words, produces 2000**4 different possible passwords, which is 16 trillion.  Log base 2 gives 43.9 bits of entropy.

But using a pseudo-random number generator with a 32-bit initial seed means that it will only generate 2**32 different sequences, or 4 billion.  That's .027% of the total!  In other words, 99.97% of the possible XKCD-style passwords CANNOT BE GENERATED by that program!

Normally, you can add more bits of entropy by either expanding the dictionary size (the number of words to choose from), or increasing the number of words in the password.  But because of the pseudo-random number generator, you are STUCK at 32 bits of entropy.  An attacker could even pre-generate the 4 billion possible XKCD-style passwords that a standard Linux rand() produces.

My point is not that 32 bits bits of entropy aren't enough, it's that you aren't necessarily going to get what you think you're getting if you use the stock pseudo-random number generator.

So if you're running somebody's application and you click "generate random password" and you see a string of gibberish that claims to have a crack time of thousands of years, it is probably wrong.  32 bits of entropy at 1000 guesses per second has a brute force crack time of under 50 days.  (And modern crackers go MUCH faster than 1000/sec.)


TRULY RANDOM NUMBERS ELIMINATE THE LIMIT

For my program, I offer the "-r" option, which reaches out to https://random.org to get random numbers.  It doesn't need very many -- you only need 4 random numbers to generate an XKCD-style password -- but the important feature is that random.org is truly random.  There is no seed.  Each number is uniformly random and independent from the previous number.  (Or at least, so claims the owner of random.org.)  I'm pretty sure this removes any artificial limit on entropy, so you can get as much entropy as you want by increasing the dictionary size and/or the number of words in the password.

Using random.org is not the only way to solve this problem.  Have you ever generated an SSL certificate?  It can take several seconds while the software "generates" enough entropy for long key lengths.  I'm not personally familiar with how that is done, but I've heard the the OS uses external physical events, like keystrokes, network interrupts, etc.  I think I've heard that it also uses disk interrupts, which makes me wonder if SSD drives make it harder for kernels to generate entropy.

If you're going to be demanding a lot of entropy for your application, you should not abuse random.org.  Instead learn how to use locally-generated entropy.

The vast majority of on-line password generators are written in Javascript.  I'm not sure how to get truly random numbers (i.e. entropy) in Javascript, but this might be a good starting point.

(By the way, a bit of reading on my part shows me that I have a lot to learn.  But my reading to date does reinforce my primary point: simply using rand() or a similar/derived function does not produce passwords that take thousands of years to crack.  At best they rely on "security through obscurity".)


PASSWORD MANAGERS' RANDOMIZER???

I'm a little worried about the "random password generators" included in password managers.  The idea is that you should have a different password for every on-line account, and you let the password manager deal with the hundreds of passwords you end up with.  Since you don't need to remember, or even type those passwords, you might as well make them be random character gibberish.  Only the password manager's master password needs to be memorized.

However, if your password manager just uses the normal pseudo-random number generator in the system, that sequence of random characters will not have as much entropy as you think.  I can tell you that LastPass's online password generator just uses Javascript's get_random() function, which only has 32-bits of entropy.  Now maybe their laptop application uses /dev/random, but also maybe the fact that their on-line generator uses built-in random indicates they didn't give the issue much thought.

I haven't done an exhaustive search, but I would wager that 90% of "generate password" functions just use the language's default random number generator, which has a 32-bit seed (or less!).

My suggestion is to use https://www.random.org/passwords/ to generate your gibberish passwords, or my program to generate XKCD-style passwords.


SEED V.S. PERIOD

The rand() man page says that Linux rand() uses the same algorithm as random().  And srandom()'s man page says that the seed is an unsigned int, which is 32 bits.  It also says:

The period of this random number generator is very large,
approximately 16 * ((2^31) - 1).

I.e. the period is approximately 34 billion, which is about 35 bits.  But the seed is 32 bits.  This means that you cannot start the random number generator at any arbitrary point in its period.  Even if you figure out a way to fully-leverage all 35 bits of random()'s period, that still gives you a crack time of 397 days, at 1000 guesses/sec.  And by the way, modern password crackers go much faster than 1000/sec.

XKCD-style Password Generator

I got to thinking about passwords again today.

I wrote my own program to produce XKCD-style passwords from a list of 2126 common words, and calculated some stats.  I reproduced XKCD's calculation of 44 bits of entropy for 4 randomly-selected words.  And I made a few mildly-interesting discoveries, and one more-interesting realization.


PASSWORD LENGTH: MORE = BETTER?

My average XKCD-style password length is 20.8 letters (over a large sample), which is a lot of typing.  So I decided to limit word size.  By filtering my list of 2129 words to nothing longer than 4 letters, I ended up with 709 words.  That's not many, and 4 of them together only gives 37 bits of entropy.  Not so hot.  But if you string 5 of 709 words together, you get 47 bits of entropy, which is better than XKCD!  And the average password drops to 18.3 characters.

I find that interesting: shorter passwords which produce more bits of entropy than longer passwords.  Seems counter-intuitive, until you realize that opening it up to the full 2129 words increases average word length more than it increases entropy.  (See below for the math.)


THE PASSWORDS

So, what do these passwords look like?  Here's 10 of the XKCD-style: 4 words from the 2126 word set:

password: MostlyRelativeSpinAdvanced

length: 26

password: ForBasicallyThinkingExplain

length: 27

password: CookieArmyMysteryConference

length: 27

password: ExpectConvertQuarterbackPresentation

length: 36

password: EverybodyProductHotDemonstrate

length: 30

password: RockIndexWellFloat

length: 18

password: BehaviorNearlyPromotePercentage

length: 31

password: PocketSurviveFourLab

length: 20

password: MuchWeekWillAnd

length: 15

password: DivideMorePeakSeveral

length: 21

So, how easy are those to remember?  Memorizing an XKCD-style password is about creating a mental picture or story around it.  Use some imagination.  It usually helps to make it amusing.  How about the first one: "MostlyRelativeSpinAdvanced"?  Well, I'm a bit of a science geek, so this one makes perfect sense.  You have a particle stream, and most of the particles are moving at relativistic speeds.  So measuring each particle's spin is a pretty advanced thing to do.  Hmm ... what's the amusing part of that?  Oh maybe that an actual physicist would roll his eyes at my explanation and say that I don't know the first thing about particle physics.  But basically, I was able to imagine a mini-story or mental picture for each of those passwords, so while I might not be able to memorize all 10, I could easily memorize one of them.

What about the shorter passwords consisting of 5 words from the 709 words of 4 letters or less?

password: BuyWingSadRideSeed

length: 18

password: SeedPairTankJailDo

length: 18

password: PanBuryDenyDataOld

length: 18

password: GeneRiceTeaYetSin

length: 17

password: WallJailLabNextTent

length: 19

password: HallSnapCashRichRead

length: 20

password: WarmUsKeepRoseLess

length: 18

password: PortMarkSirYouLeaf

length: 18

password: HiAgoHipAnyBe

length: 13

password: EaseSkyRealTossFate

length: 19

Even though those are shorter (and more secure) passwords, I guess I find them more difficult to remember them.  It's about creating a mental picture or story around those words.  Since the words are random, they don't come out in any conceptually correlated way.  So you stretch your imagination to encompass them.  The more words in the password, the more you have to stretch.

Take the last password up there, "EaseSkyRealTossFate", and drop that last word to make it 4 words: "Ease sky real toss".  My first thought is that "toss" is the children's game "ring toss".  Sky and ease kind of fit since the game is usually pretty easy and you toss things towards the sky.  The word "real" is kind of left out, but I imagine throwing something "real", like a laptop or a dinner plate, instead of a game piece.  So I imagine the ease of tossing a real laptop into the sky.  Yeah, that's stretching the imagination a bit, but maybe not too much.  I could pretty easily remember EaseSkyRealToss.

But now throw "fate" in there and my whole mental picture falls apart.  I guess I could say that when the laptop lands, its fate will be sealed, but ... not sure why ... but I would have much more trouble remembering it.

So I'll be sticking to 4 words and more typing.


THE MATH

Passwords are basically taking a set of N things, and taking L of them out with replacement.  For example, a 4-digit PIN consists of a set of 10 digits (N=10), and you take 4 digits out (L=4) with replacement.  The "with replacement" simply means that you might take the same digit out more than once (e.g. 2338).  So the entropy of a 4-digit PIN is 10**4 *(10 to the power of 4), which is 10,000.  To get that in terms of bits, take the log base 2 of it to get 13.  So 13 bits of entropy.

Another example: 8 randomly-selected letters for a password.  Let's assume lower-case only, and no digits or special characters.  The set of N things is the letters of the alphabet, so N=26.  By taking 8 characters, L=8.  26**8 = 208 billion.  Log base 2 of that is 37 bits of entropy.  Cool.  Now let's do random upper/lower case.  N=52, and 52**8 = 53 trillion, giving 45.6 bits of entropy.  Add in 0-9: N=62, 62**8 = 218 trillion, giving 47.6 bits of entropy.

So, back to my XKCD-style passwords.  My original set of 2126 words, taking 4 at a time, gives 2126**4 = 20 trillion, which is 44.2 bits.  My reduced set of 709 short words, taking 5 at a time gives 709**5 = 179 trillion, which is 47.3 bits.

However, see my next blog post for an observation about random password generators and entropy.


MORE WORDS?

My list of 2126 words actually comes from a list of 3000 words from Education First.  I filtered it to limit word length to 7 or fewer characters, resulting in my 2126 words.  Note to the rigorous: you'll find that I'm 2 words short; it made my code easier to ignore the first and last words.

So how about if I remove that filter and pick 4 words from the entire set of 2998?

2998 ** 4 = 80 trillion, which is 46 bits.  I.e. going from 2126 words to 2998 increases the entropy by 2 bits.  My average password length jumps to 25.6, which is 5 more characters.  I tried a few other word length limits and decided that 7 is best.


THE PROGRAM

See https://github.com/fordsfords/pgen Be sure to use "-r" if generating an actual password you want to use.

WannaCrypt / WannaCry ransomeware

I'm not a security researcher, and I don't follow the subject very closely.  But here is an interesting read by the person who slowed the spread of the recent WannaCrypt / WannaCry ransomware outbreak.

https://www.malwaretech.com/2017/05/how-to-accidentally-stop-a-global-cyber-attacks.html

C.H.I.P. - a small, cheap computer

UPDATE: C.H.I.P. is dead

Or rather, the company is dead.  I will keep my CHIP content laying around, but I won't be doing much with it.  Eventually, I'll probably move to Raspberry Pi.


I just received the C.H.I.P. single-board computer whose kickstart I supported.

I suspect that anybody reading this blog knows what a Raspberry Pi is.  CHIP is like that, with its own spin which I found appealing.  It's goals:

• Low cost.  The base price is currently $9 (although see the $5 Pi-zero).
• Ease and speed of getting started.  It's literally ready to surf the net right out of the box.
• Small size.  It's smaller around than a business card (although obviously it is thicker).

Accessories most people will use to try it out:

• Power supply.  It has a micro USB connector, so you might already have a charger that will work.  But be careful: even though the doc says it draws 300 ma peak, get one that can handle over an amp.  My 700 ma supply craps out.  My 1.5 amp does fine.
• Audio/video cable.  If you have a video camera, you might already have the right one (although I've heard some are wired differently).  If not, you can get one from the CHIP store.
• TV with 3 "RCA" composite audio/video jacks.  Most TVs have them.
• USB keyboard.
• USB mouse.
• Powered USB hub to power the keyboard and mouse.  Maybe you already have one.  But note that this hub will *not* power the CHIP.  You'll still need the micro USB supply.
• Case.  Completely optional, but at $2, I suggest getting it.

I did go that route for my first bootup.  Being packrats, my wife and I had all the above items, so we were able to get the GUI operational in about 15 minutes.  Not being very familiar with Linux GUIs, it took another 15 minutes to be surfing the web.  Yep, it really is a general-purpose computer.  But being composite video, it has very low resolution.  And drawing a couple of watts, it isn't very fast (compared to modern laptops).  And being Linux, it doesn't have much PC-like software out of the box.  But really, who would want to use one of these as a replacement for your laptop?  ... Oh.  Um.  Never mind.  :-)

If you aren't interested in the GUI, you don't need any of the above accessories.  All you need is a laptop and a micro USB cable (I had 3 laying around).  Again, be careful: we had a couple cables that were wired for power only and didn't pass signals.

Here's a minimal "getting started" procedure:
    http://www.geeky-boy.com/w/CHIP_startup.html

Wanna know who *really* landed a man on the moon?

A woman named Margaret.

https://medium.com/@3fingeredfox/margaret-hamilton-lead-software-engineer-project-apollo-158754170da8

More info on Wikipedia.  Excerpt:

Hamilton's work prevented an abort of the Apollo 11 moon landing: Three minutes before the Lunar lander reached the Moon's surface, several computer alarms were triggered. The computer was overloaded with incoming data, because the rendezvous radar system (not necessary for landing) updated an involuntary counter in the computer, which stole cycles from the computer. Due to its robust architecture, the computer was able to keep running; the Apollo onboard flight software was developed using an asynchronous executive so that higher priority jobs (important for landing) could interrupt lower priority jobs. 

Smart engineer.

Grace Hopper documentary (crowd funding)

I suffer from a little bit of hero worship for Grace Hopper (1906 - 1992).  She is often credited with writing the first compiler in 1952, and was an important contributor to the development of COBOL.

There is a crowd-funded effort to produce a documentary on her:

http://www.indiegogo.com/projects/born-with-curiosity-the-grace-hopper-documentary

I would like to see this documentary succeed because I really would like to know more about her than appears in Wikipedia.

Update: Grace Hopper on Letterman:
    https://www.youtube.com/watch?v=1-vcErOPofQ
She really was charming.

Password strength

I disagree with a lot of "password strength" measures.  Most measures want you to include upper and lower case, digits, special characters, etc.  I don't feel they are necessary, and don't give you as much "security" as you might think (like substituting zero for the letter "o").

Then along came Randall Munroe with an XKCD cartoon which does a much better job of explaining it than I ever could:

Most of the password strength "meters" that you see on sites are based on the idea that digits, special characters, and mixed-case are the magic elixir for strong passwords.  I was quite dismayed to discover that most of them consider "P@ssw0rd" to be very secure, which is absurd.  Then I found zxcvbn:

https://dl.dropboxusercontent.com/u/209/zxcvbn/test/index.html

Finally, a password strength meter which knows that "P@ssw0rd" is low security (score=0 of 4, crack time 0 seconds)!  Whereas "correcthorsebatterystaple" is very secure (score=4 of 4, crack time 65 years).

Another password method that I've heard hyped which I disagree with is the haystack approach.  According to this author, the password "D0g....................." (21 dots) is very strong.  This is ONLY true for brute-force password cracking, which is NOT how serious crackers work.  They do dictionary and repeated character analysis.  According to zxcvbn, "D0g....................." is weak (score=0, crack time 84 seconds).  YES!

One fly in all this ointment: many systems limit your password length, sometimes to as few as 8 characters.  This makes it very hard to use 4 random words, meaning that you probably need to go the random route.  For the 8 random character password "0ZhyUQ63", zxcvbn rates it 4 of 4, with centuries required to crack it.  Whereas "saytroll" is weak, with 22-second crack time.  (Note that "S@yTr0ll" is still weak, with a 7-minute crack time - so much for magic elixirs.)

BTW, my wiki has a somewhat longer article on password strength.

Syns, Syn Cookies, TCP Listen Backlog: More Complicated than You Think

No, syncookies don't have anything to do with dieting.  But they did come up as I learned that the TCP listen backlog is more complicated than I thought.  This article should help those of you trying to support TCP servers with lots of clients, especially if large numbers of clients can try to connect at the same time.  (For example, a popular web server.)  This is Linux-oriented; I'm not sure how applicable the info is for other OSes.

---

Here is an article which talks about the TCP listen backlog. Here are some quotes:

The backlog has an effect on the maximum rate at which a server can accept new TCP connections on a socket. ... Many systems (particularly BSD-derived or influenced) silently truncate this value (the backlog parameter to the listen() system call) to 5 — version 1.2.13 of the Linux kernel [really old - SF] does this ... Using small values for the listen backlog was one of the major causes of poor web server performance with many operating systems up until recently. ... The backlog parameter is silently truncated to SOMAXCONN ... defined as 128 in /usr/src/linux/socket.h for 2.x kernels.

---

Here is a brilliant writeup that taught me about "syncookies", and how they can lead to hung clients. Basically, if the listen backlog (a.k.a. the SYN queue) fills up and more client connection requests (SYNs) come in, the server will *act* like it is accepting them by responding with syncookies. But the kernel won't actually set up state for those connections or inform the app of the new connection. Instead, the server waits for the client to respond with the ACK (the third step of the 3-way handshake). That ACK contains enough information for the server to reconstruct the initial SYN, and the kernel proceeds to open the connection as normal. HOWEVER, if the client's ACK gets lost in the switch or the NIC or whatever, then the client will be left thinking the connection was accepted and is ready, and the server will have no memory of it.

This leads to a genuine hang if the application protocol depends on the server sending the first message, like SMTP or MySQL. In these cases, the client app will hang forever waiting for the server to send its message.

---

Here is an article which gives advice on how to set up systems that can accept lots of TCP connections. Here's a quote:

Three system configuration parameters must be set to support a large number of open files and TCP connections with large bursts of messages. Changes can be made using the /etc/rc.d/rc.local or /etc/sysctl.conf script to preserve changes after reboot. In either case, you can write values directly into these files (e.g. "echo 32832 > /proc/sys/fs/file-max").
• /proc/sys/fs/file-max: The maximum number of concurrently open files. We recommend a limit of at least 32,832.
• /proc/sys/net/ipv4/tcp_max_syn_backlog: Maximum number of remembered connection requests, which are still did not receive an acknowledgment from connecting client. The default value is 1024 for systems with more than 128Mb of memory, and 128 for low memory machines. If server suffers of overload, try to increase this number.
• /proc/sys/net/core/somaxconn: Limit of socket listen() backlog, known in userspace as SOMAXCONN. Defaults to 128. The value should be raised substantially to support bursts of request. For example, to support a burst of 1024 requests, set somaxconn to 1024.

Here are some commands I entered on our host Saturn:

   sford@Saturn$ cat /proc/sys/net/core/somaxconn
   128
   sford@Saturn$ cat /proc/sys/net/ipv4/tcp_max_syn_backlog
   2048
   sford@Saturn$ cat /proc/sys/fs/file-max
   3263962
   sford@Saturn$

Looks like the main thing we need to do is increase somaxconn, and maybe tcp_max_syn_backlog as well.

---

One small concern. I saw various references to SOMAXCONN as being a constant in a system include file. It apparently lives in different places, depending on the OS flavor/version; I found it here:

   sford@Saturn$ find /usr/include | xargs egrep SOMAXCONN
   /usr/include/bits/socket.h:#define SOMAXCONN 128

So now the question becomes, if we update the tuning parameter, do we also have to modify the include file? My gut says no. I'm thinking that maybe if you built the kernel from source, you would perhaps change the default via that include, but on a running system you simply override that default and you can magically use larger numbers in the listen() call.

Socket buffers: more complicated than you think

If you are receiving UDP datagrams (multicast or unicast, no difference), how much socket buffer does a datagram consume?  I.e. how many datagrams of a particular size can you fit in a socket buffer configured for a given size?

Well ... it's complicated.

I've tried some experiments on two of our Linux systems, and encountered some surprises.  Note that my experiments were performed with modified versions of the msend and mdump tools, i.e. simple UDP with no higher-level protocol on top of it.  (See my Github project for my modified versions.)  The modified mdump command sets up the socket, prints a prompt, and waits for the user to hit return before entering the receive loop.  I had msend sending 500 messages with 10 ms between sends (nice and slow so as not to overrun the NIC).  Since the mdump is not yet in its receive loop, the datagrams are stored in the socket buffer.  When the send finishes, I hit return on mdump, which enters the receive loop and empties the socket buffer, collecting statistics.  Then I hit control-c on mdump, and it reports the number of messages and bytes received.  Finally, I did experiments on both unicast and multicast; the results are the same.

Here are some results for a two-system test, sending from host "orion", receiving on host "saturn".  The message sizes and bytes received shown are for UDP payload.  Receive socket buffer configured for 100,000 bytes.  Note that 1472 is the largest UDP payload which can be sent in a single ethernet frame (i.e. no IP fragmentation).

message size	messages received	bytes received
1472	61	89792
215	61	13115
214	157	33598
1	157	157

Interesting.  The number of messages seems to not depend on message size, except for a discontinuity at 215 bytes.  I checked a lot of other message sizes, and they all follow the pattern: 61 messages for sizes >= 215, 157 messages for sizes <= 214.

Now let's double the receiver socket to 200,000 bytes:

message size	messages received	bytes received
1472	121	178112
215	121	26015
214	313	66982
1	313	313

The messages received are approximately doubled, with the discontinuity at the exact same message size.  Cutting the original socket buffer in half to 50,000 approximately cuts the message counts in half, with the discontinuity at the same place (I won't bother including the table).

Now lets switch the roles: send from saturn, receive on orion.  Socket buffer back to 100,000 bytes.

message size	messages received	bytes received
1472	77	113344
215	77	16555
214	363	77682
1	363	363

The discontinuity is at the same place, but different numbers of messages are received.  The linux kernel versions are very close to the same - Saturn is 2.6.32-358.6.1.el6.x86_64 and orion is 2.6.32-431.1.2.0.1.el6.x86_64.  Both systems have 32 gig of memory and are using Intel 82576 NICs.  Saturn has 2 physical CPUs with 6 cores each, and orion has 2 physical CPUs with 4 cores each and hyperthreading turned on.  I'm don't know why they hold different numbers of messages in the same-sized socket buffer.


These machines also have 10G Solarflare NICs in them, so let's give that a try.  Send from saturn, receive on orion, socket buffer 100,000 bytes.

message size	messages received	bytes received
1472	110	161920
1	110	110

Whoa! That's right - when using the Solarflare card, the socket buffer held more bytes of data than the configured socket buffer size!  But this isn't necessarily unexpected; the man page for socket(7) says this about setting the receive socket buffer: "The kernel doubles this value (to allow space for bookkeeping overhead)". Finally, it's interesting that there is no discontinuity - 110 messages, regardless of size.


Let's stick with the Solarflare cards, and go back to orion sending, saturn receiving (still 100,000 byte socket buffer):

message size	messages received	bytes received
1472	87	128064
1	87	87

Fewer messages, but still exceeds 100,000 bytes worth with large messages.


Now let's put both sender and receiver on saturn (loopback), with 100,000 byte socket buffer:

message size	messages received	bytes received
1472	87	128064
582	87	50634
581	157	91217
70	157	10990
69	261	18009
1	261	261

Lookie there! Two discontinuities.


Someday maybe I'll try this on other OSes (our lab has Windows, Linux, Solaris, HP-UX, AIX, FreeBSD, MacOS).  Don't hold your breath.  :-)


I did try a bit with TCP instead of UDP.  It's a little trickier since instead of generating loss, TCP flow controls.  And you have to take into account the send-side socket buffer.  And I wanted to force small segments (packets), so I set the TCP_NODELAY socket option (to disable Nagle's algorithm). The results were much more what one might expect - the amount buffered depended very little on the segment size. With 1400-byte messages, it buffered 141,400 bytes. With 100-byte messages, it buffered 139,400 messages. I suspect the reduction is due to more overhead bytes.  (I didn't try it with different NICs or different hosts.)


The moral of the story is: the socket buffer won't hold as much UDP data as you think it will, especially when using small messages.

UPDATE: on a colleague's suggestion, I looked at the "recv-Q" values reported by netstat.  On Linux, I sent a single UDP datagram with one payload byte.  The "recv-Q" value reported was 1280 for an Intel NIC, and 2304 for a Solarflare NIC.  When I set the socket buffer to 100,000 bytes and fill it with UDP datagrams, "recv-Q" reports a bit over 200,000 bytes - double the socket buffer size I specified.  (Remember that socket(7) says that the kernel doubles the buffer size to allow space for bookkeeping overhead.)

UPDATE2:I'm not the first one to wonder about this. See https://www.unixguide.net/network/socketfaq/5.9 (that info is for BSD, not Linux).