Is a lot of spam our own damn faults?

I got an unsolicited sales inquiry from a major company the other day.  Each day, 10 to 20 junk emails make it through our spam filter.  Usually, I can delete them after only a second or two, but this one sounded like I might already have a business relationship with them.  I don't want to risk insulting a customer or vendor, so I responded, asking what it was about.  The salesman was honest; he said that he thought somebody with my title would be interested.  I wasn't.  Not even close.

I've been on the Internet since it became easy to get on it.  When did it become acceptable to send blind solicitations?  When did the word "spam" come to mean only Nigerian princes and phishing schemes?  It used to be only desperate, border-line ethical, fly-by-night companies that sent junk email.  Now it's Box, Oracle, Microsoft, hell, I'm pretty sure my own employer does it!  Why have mainstream companies sunk so low as to send solicitations based on title?

Think back (if you're old enough) 20 years.  There were trade magazines that you could get "for free".  All you had to do is fill out a sheet that indicated in fair detail what your interests were, what industry you worked in, and the kinds of products over which you have purchase influence.  Vendors got very precisely-targeted lists, and we all knew that we would be getting solicitations.  We valued the magazine, so we didn't resent the ads.  Heck, although I don't remember specifically, I suspect I responded positively to one or two solicitations; the advertiser got their money's worth and I got a product that I wanted.

Those magazines don't exist any more, or at least not in my field.  We've all stopped reading the paper versions and instead look to the web for the information we're interested in.  We subscribe to blogs,  podcasts, slash-dot, LinkedIn groups, and any number of other curated content providers.  But the Internet evolved from an early non-commercial birth.  Early adopters resented the commercialization of the Internet, and refused to give information about themselves.  We create throw-away email addresses to subscribe.  We want to remain anonymous.  So the information curators never established the model of "you tell me about yourself for marketing purposes, and I'll give you information you want."  Some companies tried to get that going, but the internet "culture" prevented it from catching on.

So guess what?  I and my fellow-junk-email-haters are suffering from the unintended consequences of our own behavior.  Vendors no longer have precisely-targeted lists available to them.  So they substitute quantity for quality; send a million emails, and you're sure to find some prospects.  It's the new normal.

Idealists like me want a total paradigm change.  We want unsolicited advertisements to go away completely.  Back in the day, if I knew I wanted a C compiler, what did I do?  Open the yellow pages?  Sorry, no entries in the Yellow Pages for C compilers.  No, I *depended* on those trade magazines' advertisers to give me access to vendors of C compilers.  But now that search engines exist, we can do away with outgoing advertisements.  Instead of push marketing, go with pull marketing.  If I want a C compiler, I won't open my "junk" folder to find an unsolicited ad, I'll do a web search.  And this model *does* work!  We put some useful information on our web site, and attracted more than one customer who came for that information and stayed for our product.

And yet, the realist in me knows that human nature is what it is.  Research has proven again and again that advertising works.  I suspect modern email campaigns generate a lot of "unsubscribe me" responses, some of which may be less than polite, but I also suspect that they generate at least some interest.  Cast a wide-enough net and you'll catch some fish.

So if I have an emotional response to junk mail that is out of proportion to it's actual cost to me, that's my problem, not the advertisers.  I guess I need to get over it.

Solaris Multicast Deafness Bug

Once again, the mighty Dave Zabel (of two different fames) has found another Multicast-related bug, this time in Solaris.  I think that recent versions of Solaris fix it, and I don't have the energy to track down *when* they fixed it, but if you have Solaris servers that you haven't kept updated for a while, you might have this bug.


DEAFNESS DEMONSTRATED

You'll need Informatica's "mtools" package for Solaris.  These are great tools offered for free in both binary and source form at https://marketplace.informatica.com/solutions/informatica_mtools

And you'll need two hosts: A and B.  Host B should be Solaris 6.10 that hasn't been updated in a long time.  Host A can be anything.

1. On host A, run this:
    msend 239.0.3.13 12000 15

2. On host B, open two windows.  In the first, enter this:
    mdump 239.0.3.13 12000

Admire the printouts of the multicast packets for a while.  Isn't technology wonderful?  :-)

3. In a second host B window, enter:
    mdump 239.128.3.13 12000

Note that the first window continues to print, but the second window is silent.  No surprise; it is listening to a different and unused multicast group!  Of course it is silent.

4. Kill that second mdump.

WHOA!  The first mdump stops printing!  It went deaf to 239.0.3.13.  When trying this same experiment on Linux, or on our Solaris 5.11 machines, it does not go deaf.  But we have several old, non-updated 5.10 machines where the first mdump does go deaf on this step.

5. Enter:
    netstat -g

The OS still thinks it is listening to the multicast group.

6. Enter:
    snoop -P host 239.0.3.13

The packets are still being received!  But they aren't being delivered to the first mdump.

7. Enter:
    mdump 239.128.3.13 12000

WHOA!  The first mdump starts printing again!  The second mdump is still silent since there still isn't any traffic on its multicast group.


MAYBE IT'S MY PROBLEM?

Maybe PEBKAC?  Or a bug in mtools?

Nope.  Let's start over and try it again with a small change in step 3:

1. On host A, run this:
    msend 239.0.3.13 12000 15

2. On host B, open two windows.  In the first, enter this:
    mdump 239.0.3.13 12000

3. In a second host B window, enter:
    mdump 239.64.3.13 12000

See what I did there?  I changed the 128 to 64.  As before, the first window continues to print, but the second window is silent.

4. Kill that second mdump.

Lookie there!  The first mdump continues to print the messages.  No deafness.


WHAT'S GOING ON?

Well, I'm not sure, but I think it's got to be related to multicast group aliasing.  Remember that there are 2**28 different IP multicast groups.  But what about Ethernet?  There are only 2**23 Ethernet multicast MAC addresses allocated for use by IP multicast.  It turns out that 239.0.3.13 and 239.128.3.13 map to the same Ethernet multicast MAC address: 01-00-5E-00-03-13.

The IGMP protocol doesn't care about that; host B still tells the switch which multicast groups are subscribed, and it treats 239.0.3.13 and 239.128.3.13 as different.  But when the IP layer interfaces with Ethernet, it needs to program the NIC with the same multicast MAC address for those two IP groups.  And apparently older versions of Solaris didn't do the book keeping right.

I've tried this experiment on other OSes and they all work as you would expect (no deafness).  Our Solaris 5.11 machine does it right.  And even a recently-installed 5.10 system works right.  But older systems that haven't been updated in a while all have this problem.



THE MORAL OF THE STORY

The obvious moral is to update your systems.

But even then, you should avoid using multicast groups that alias on top of each other.  The whole point of multicast is that you don't receive packets that you aren't interested in.  But if you have traffic published to both 239.0.3.13 and 239.128.3.13,  a host subscribing to only one of them will get data for both.  The IP layer will do the right thing (discard the undesired packets), but it still produces an unnecessary load.


ANY OTHER GOTCHAS?

Sure.  Watch out for well-known and ad-hoc multicast protocols in the range 224.0.0.0 - 224.4.255.255.  Are any of those in use anywhere on your network?  No?  Are you sure they never will be?

Look at the multicast group we tested with: 239.0.3.13.  That aliases on top of 224.0.3.13, which is in an ad-hoc1 range labeled "RFE Generic Service".  I don't know what that is (and Google doesn't seem to know either), but I'm thinking I want to avoid aliasing, even if low probability.

You should be fine if you use multicast groups between 239.0.5.0 - 239.127.255.255.

Oh, and update your systems too.  Good hygiene and all that.


UPDATE: UPGRADING FIXES IT

We've upgraded one of our "problem servers" to the latest Solaris 5.11 and it fixed the deafness problem.

I'm not interested in figuring out exactly which minor release they fixed it in.