[rescue] Sun enterprise 10000 stuff

[Tonu Samuel]

Hi!

Sorry I cannot reply to old topic as I just subscribed. I read through
archives and I hope you forgive me posting right after subscription.

This is me who has E10k at home in Estonia. Yes it was bought from ebay and
has nearly everything you might imagine. Full CPUs, mem and such. Most e10k-s
on sale have 8..12 CPUs or similar.

If someone wants to upgrade e10k them main problem is not memory ($4000 per
gig on Sun pricelist) and CPUs ($12000 for 400Mhz CPU in Sun) but board
itself. It costs $58,000 and you need extra $10,000 memory board to use
memory on it. While memory and CPUs are decently prices in ebay then e10k
boards are very rare and expensive. And notice - there is no replacement.
Just e10k boards go.

Meanwhile yes, e10k uses 13,5KW (never measured myself as my house has 20A
main fuse, i.e. around 5KW only.). But e10k can be run in pretty economic
way. It runs right now and wattmeter shows 1200W plus I have A5200 box with
disks and not counted in. e10k consumes some 10W only per board when they are
switched "off" from software. Not so bad. On E6500 they went down to 50% of
usual only.

Back to business. I am already thinking about writing some document "There is
a e10k in ebay, should I buy it?". So I write small FAQ here. I try to answer
all questions I have seen in archive of list.

--- Decision making support questions

Q: Can I make it work?
A: Probably yes. Starfire has nearly all duplicated hardware and most things
even if missing doesn't stop you doing at least something with it.

Q: I have some experience with Sun E3x00..E6x00 computers. Is Starfire same
but bigger?
A: No. Stafire has same CPUs, same memory modules, same disk boxes but lot of
things are different. Other Sun/Sparc/Solaris know-how is useful but be ready
to relearn.

Q: So, what is different?
A: Hardware is incompatible with E3x00...E6x00. No serial port console. You
can run different operating systems on different system board. Generally
everything is better than Exx00 but weights much more and uses more power
too.

Q: Why should I buy one?
A: Starfire has unique features you may want to learn or use. Major difference
from all other Suns of same era is having ability physically separate system
boards and run different operating systems on groups of them. Those are
referred as "domains" in all Starfire documentation. You also can remove
boards from running domain with price of some 1 second suspend only and also
add them back. This computer is very flexible also about power saving and you
have still freedom to add tens of CPUs on the fly when needed. One reason
might be just owning real piece of history. This is one of most powerful and
legendary computers ever made and improved version of Cray at your house.

Q: Why should I do NOT buy it
A: This machine has lot of problems. You may be not able physically fit some
of them. Main troubles:
	1. It is big. Yes you may have room for it but usually it does not fit
through door. Minimum width of door where this machine can go through with
all sidecovers removed is over 90cm.
	2. It is heavy. Weights 1 ton (2000lbs) and by removing all possible parts
from it you get frame with 1/3 of this weight which is still very heavy. It
is nearly impossible to get it in from window if you think about this option.
While it is on wheels it is difficult even to move it a little with one men.
	3. It consumes lot of power. You need minimum 2000W power available. This is
because it takes 2300W initially when switched on. Major part of this are
ventilators. you can switch off some of them but think 1500W as minimum when
not sure. Minimum set with one system board running (4 CPUs, full of memory)
takes around 1200W. Maximum indicated in documentation is 13,5KW
	4. It generates lot of heat. Look up for power consumption and imagine such
powerful electric heater in hour room. If you live in hot climate, this
becomes a problem and you probably need to use 15KW air conditioner just for
this computer.
	5. It makes crazy noise. Really! Forget about using it with same house with
neighbors. Meanwhile you can make it slightly lesser with tuning fans but
problem still exists.

Q: What is SSP?
A: SSP is a controlling computer without which Starfire is a brick. you cannot
use it without this computer. You need to run "power -on" from his command
line for example. Normal Starfire setup had two of them. No problem if you
have just one.


Q: There is a e10k for sale but no SSP included. Can I make it work?
A: Yes! But you need to get another Sun Sparc computer and install some
special software on it. Sparc computer+ Solaris OS+SSP software is all you
need.

Q: So, how they communicate....SSP and Starfire?
A: Ethernet and IP. When switched "on" Starfire boards issue RARP request and
need to get IP for their boards. It is your responsibility to have RARP
daemon. Usually it may run on SSP computer. After that Starfire control
boards have IP address, they try to download firmware over TFTP protocol from
same computer which answered by rarp. Firmware establishes communication with
SSP special daemons after few minutes. From this point you make all tasks
from SSP console commands. No physical access needed for Starfire. BTW, not
all is IP encapsulated there. There are some exceptions - for example
resetting board is done using weird specially crafted ethernet packet.

Q: I heard something that Starfire cannot run without special license or
"keys"
A: This is true. You need keys and usually it is very difficult to learn if
your starfire has them or not. Sellers in ebay are often scrapyard companies
who have not much idea about technical details. Even more - they usually wipe
disks to make sure you cannot get any customer data. Meanwhile I would
suggest you a try buy starfire anyway. Seems like their protection method is
weak and in worst case you can make those keys yourself. Not sure about you
country laws but in many places you are legally allowed to make reverse
engineering if this is necessary to make broken code to work. For example I
got starfire all legal ways with licenses (but no keys) but Sun was too lazy
to speak with me. So I reverse engineered related code and found it trivial
to make keys yourself. Also I wonder if wrong keys work on it. Anyway I think
this problem can be worked off if you really want. Sure I never suggest such
solutions for business users. Sure always try to get keys from seller of
computer first.
You need those keys in stage where you generate EEPROM file to be uploaded
into starfire domain. Every domain needs one key.

Q: Will it run Linux? NetBSD?
A: Maybe. Actually be prepared that they will not run or you have lot of
trouble with it. Currently I have experience running linux on big suns and
only Starfire is place where I never seen it working yet. I get all different
reasons why it crashes and have no workaround yet. I think problems can be
solved but be warned - this is mainly for Solaris only. All other OS-s are
even less supported. I do not want to begin flamewar here but under Solaris
you can add boards on the fly on this platform and under linux there is no
support for such features under Sparc64. You cannot use some coolest features
of this hardware with linux yet.

Q: I may buy one from ebay but not sure how much transport will cost. I heard
about $3000.
A: No, you probably get it for much less. I paid around 500 EUR from UK to
Estonia. This is from one border of EU to another! I know it is difficult to
estimate cost but call into some cargo company and ask for "pallet
price".Notice, not DHL, not UPS but some company which moves big goods with
big trucks. They carry nothing less than pallet. Those companies do lot of
business and always ready to tell price for "pallet". "Pallet" means fixed
footprint goods on wooden pallet and limited to weight of 770kg (1500lbs).
When you know price for pallet, this is good indication. They charge you
little extra for weight over this one but you can live with it. I used
company DFDS Transport and they handled all from pick up in UK till bringing
this box down to my village within one week and decent price.

----- Other stuff

Q: Many CPUs....it should be FAST
A: Not necessarily. Most of jobs just won't scale over many CPUs, so your
application probably runs like on single or dual 400Mhz computer with 8MB
cache on CPU. You get real speed boost only if you have many concurrent users
or your algorithms/apps scale over many CPUs. Most of them don't.  You have
to consider also that big computer in low-end configuration is usually slower
than low end computer in high configuration. Memory latency depends on
maximum amount supported by computer, not by actual amount installed for
example. It is better to use maximum supported amount to get best results.



--- For people who already have starfire

I will make some stuff how to make it work. Some QuickStart. I think for now
it is enough.

Aurora developers who got e10k, please contact me. I think we are in same
boat. My linux running attempts with Aurora 2.0 and Gentoo 2006.x are here:

http://no.spam.ee/~tonu/sparc/gentooe10k.txt
http://no.spam.ee/~tonu/sparc/aurorae10k.txt

I also provide shell access for people with any serious intention. But please
understand that this computer is in my bedroom and making noise of jet
engine. I do not run it 24x7.

   Tunu