[geeks] Weird Linux Multihomed Problem

[Jonathan C. Patschke]

$ork has Linux box multihomed (10.10.0.0/16 on one interface and
10.20.0.0/16 on the other) running SuSE Enterprise Linux v9.  The
10.10.0.0/16 interface has the default route.

We have a network setup[0] like this:

                    +------+
         +------.10=| BSD  |=.20------+
         |          +------+          |
         |                            |
         |          +------+          |
         |  +---.10=|Linux |=.20---+  |
         |  |       +------+       |  |
         |  |                      |  |
       [switch]                  [switch]
           |                        |
           |        +------+        |
           +----.10=|Router|=.20----+
                    +------+
                      .30
                       |
                       |
                    +------+
                    | VPNs |=192.168.0.0/16----->PC
                    +------+

So, we have the Linux server, with parallel connections to one router
that has three interfaces.  We have a BSD server connected similarly.
On the third interface of the router is a VPN concentrator, and on the
other end of that concentrator are PCs.

PC users at home can ping the 10 interface of the Linux box just fine,
they can see services on the 10 interface of the Linux box just fine.
Initially, they cannot see anything on the 20 interface of the Linux
box.  So,

   1) PC user pings 20 interface on Linux box, gets no responses.
   2) PC user pings 10 interface on Linux box in another window, gets
      responses, AND his first ping sesssion starts to see responses.

I first suspected the router (a Dell POS), but the BSD server DTRTs, no
matter what.  In, fact, when the BSD and Linux boxes talk, the BSD box
spews "arp: 10.10.xxx.xxx is on em0 but got reply from
xx:xx:xx:xx:xx:xx on em1", but only when talking to the Linux boxes, and
the addresses involved are always of the Linux boxes.

Proxy-arp is turned off all around, so we shouldn't have Martian packets
surviving anywhere, and I've verified that servers really are plugged
into the ports they should be.

Clue, please?



[0] Yes, it looks overcomplicated and like something you'd see in a
     CCIE abuse lab, but we have a really good reason to have things set
     up this way.
-- 
Jonathan Patschke    )   "A man who never dreams goes slowly mad."
Elgin, TX           (      --Thomas Dolby, "Valley of the Mind's Eye"