[rescue] advice on rescuing an e10k

[Jonathan C. Patschke]

On Mon, 23 Oct 2006, Andrew Gaylard wrote:

> Is Solaris really that much faster on this hardware? And what do
> you mean by "maximum use of the hardware" and "scalability"?

Well, first off, Linux doesn't support that hardware.

In general, open source support of large Sun hardware is a dicey
prospect because they system's don't look like typical workstation
hardware.  Instead of a processor-to-memory bus multiplexed by a
"north bridge" and a processot-to-I/O bus multiplexed by a "south
bridge" and a system controller to manage cache coherency, you tend to
hvae a far more complicated system when you get past 4 CPUs or 2 CPU
boards in RISC hardware.

E-series Suns (and, I would presume, Sun Fire servers) look more like a
cluster running in lock-step with a shared clock.  Linux doesn't support
many systems with an architecture that looks like this (large-ish AXP
and IA64 systems excepted).

> And what in particular makes it faster? (A better kernel?  Better
> compiler?  Something else?)  I'm not disputing this, I'm just curious
> to know.

Well, Sun's compiler beats GCC in terms of both code size and code
efficiency.  I would assume GCC has gotten a little better, but it used
to be that you could look at the assembly output of GCC and laugh at
things like loading dummy values in to registers just before stomping
them with real values, unnecessary NOPs in the branch-delay slot, and
immediate value loads that were obviously generic rather than optimised.

A lot of what "faster" means depends on your point of view.  Solaris is
"fast" on large Sun hardware not because any one task runs faster, but
because the SunOS scheduler is optimised for Very Large hardware (much
the same reason Solaris is so much slower than Linux on tiny SPARC
systems).  Solaris can get more done at once than Linux above a certain
hardware threshold (that used to be nearly any 4-way SMP system)

> If Solaris really is, say, 10%+ faster than Linux, then I'd happily
> use it.

It all depends on how you measure it.  Will a Solaris system support
10% more threads doing the same workload as on Linux, possibly.  Will
any given process run 10% faster?  Probably not.

There are other options to consider, too.  Large systems (Sun, IBM, HP,
etc.) tend to have embedded management hardware (thermal, partitioning,
lights-out management, hot-swap, and component failover) that are very
frequently only working with the vendor's operating system.  If you're
going to run large RISC hardware in such an environment that you lose
all the management features that come with the hardware premium, you'd
be better off running a large fast PC, since that's essentially what
you'll get.

> I was planning to use Gentoo Linux; I've found that compiling for a
> particular CPU does give a noticible speed improvement over vanilla
> distributions.

I've found that any such benefits are largely outweighed by the
inefficient code that runs on so many Unix and Linux systems today.  No
matter how many nanoseconds the compiler trims off a function by
unrolling a loop a few notches, the system still ends up waiting for
things like Mozilla and its runtime-interpreted XUL user-interface,
GNOME/GTK and their propensity for redrawing widgets whenever they want,
and the other mountains of crap we run to slow down our blazing fast
hardware.  As the man from SGI once said:

   "Do you want to be a bloat detective? It's easy; just pick any
   executable. There! You found some!"

That applies more to the modern Unix system than it -ever- applied to
IRIX 5.0.

> My main reason for using Linux is that the app I need isn't ported to
> Solaris, and when I tried to do the port myself, it wasn't simple.
> For instance, Solaris doesn't have
>    fnctl( fd, O_ASYNC, ...)

Well, yes.  The man page on Linux says:

   The use of O_ASYNC, F_GETOWN, F_SETOWN is specific to BSD and Linux.

Hence, they should not be used in portable code.  select() or a blocking
background thread would be a more portable solution.

-- 
Jonathan Patschke   "The ruler demands gifts, the judge accepts bribes,
Elgin, TX            the powerful dictate what they desire--they all con-
USA                  spire together. The best of them is like a brier, the
                      most upright worse than a thorn hedge." --Micah 7:3-4