http://www.sparcproductdirectory.com/artic-2002-jan-pb.html Unsafe At Any Speed? Looking under the hood at Sun's recent server engine problems Sun's cache memory problem:- What did Sun know? When did Sun know it? And what did Sun do about it? - a critical commentary. Editor's intro:- in 1965 Ralph Nader wrote a book called "Unsafe At Any Speed" to expose design flaws with the Chevrolet Corvair manufactured by GM. In 2001 Sun's cache memory problems caused millions of dollars of wasted time in corporate America and shattered the myth of Sun server reliability. Sun's reported communications about this subject have tried to pass the buck to a major supplier, IBM. But Sun could have designed around this problem, or detected it earlier. The recent comments by Sun Microsystems' Chairman and CEO, Scott McNealy, are extremely interesting and shows the absolute importance of truth and candor by senior executives of any company. Often the first signs of deeper problems within the company come from very simple statements by executives that later appear to be misleading and incorrect. Once corporate credibility has been lost it takes years to recover, and sometimes never does. Lets take a very candid look at the comment (reported in a Computerworld article November 27, 2001) Q: We reported last year about the problem with the external memory cache on UltraSPARC IIs that was causing a lot of Ultra Enterprise servers to crash. Is that something you're still grappling with, or is it history? A. We're no longer buying IBM SRAM [static random-access memory]. They were the biggest source of the problem for us. They knew about it before, and they didn't tell us. But we don't have that issue anymore. We designed IBM out of that and put [error checking and correcting logic] across the entire cache architecture. The Shell Game As we all know the real purpose of a shell game is to distract the viewer from where the REAL situation is. What really WAS the defect: ----Where should we look for the REAL problem. The industry has been rife with rumor and innuendo for three years, not helped by the fact the parts of SUN still today denies there is a problem. No statement has ever been issued by Sun's corporate office to clarify and no clear corporate policy has ever been introduce to rectify the problem.. ScottM seems be pointing at normal day to day QA problems and outside suppliers. Sun takes a risk by subcontracting the majority of its work to third parties, and in doing so bears the responsibility ensuring compliance. This comment seems to say this is really no big deal and not worth talking about. Classic Shell Game. Watch this hand not THAT one. The truth is far more serious. In my view, the real major defect was the complete lack of design of Error Correction in level 1 and 2 of the CPU cache of the UltraSPARC II Sun's flagship CPU processor. Error Detection and Correction Primer Transient errors (as opposed to permanent errors due to device failures) can occur in all digital systems. They can be caused by electrical noise, ambient radiation, clock jitter and other causes. As far back as 1948, R.W. Hamming of Bell Labs developed a general theory for error-correcting schemes in which "check-bits" are interspersed with information bits to form binary words in patterns. These are now reffered to as Hamming codes. The rate of failure, and consequences, if uncorrected, are important parameters in system design. Communications systems always use error correction, because the total environment is not under the control of the designer. Memory and storage systems are particularly vulnerable, because they can capture and store a transient error bit which might have no effect in another part of the system. Error detection and correction codes have been in the Digital Design textbooks for over 3 decades, and got a good press in the early 1980's, as a way of fixing soft errors caused by alpha particle radiation in the 64k bit RAM generation. Chip manufacturers later found that alpha particle emitters were contaminents in commonly used chip packaging materials, and removed this prime cause. ECC stayed in large memory systems because it fixed transient problems from many causes including some types of device failures. As a general rule, in all electronic devices, transfer rates are increasing as time goes on. Also, errors will occur more frequently in any kind of storage device because manufacturers are squeezing more bits into smaller and smaller spaces. As speed and density increase, error correction becomes a necessity because a smaller energy disturbence source can trigger a false bit. Error checking technologies used in processor boards can be categorised as follows: BEST: ECC (Error Correction Code) ECC can detect and correct single- bit errors. It is used in high-end PC's and servers. Medium: Parity This is the most common used method. It can detect errors, but not correct them. Cheapest: Non-Parity Because there has been an increased quality of memory components and an infrequency of errors, more and more manufacturers do no include error checking capabilities. This also lowers the cost of the PC. Only used in lower end hardware. Now, assuming You are the leader in your industry, Marketing says that you make the Rolls Royce, DELUXE product, that's why you command the top $$$$ prices. Which one do you use: Sun's answer on the UltraSPARC II was the cheapest: NONE, no EEC on your primary L1 cache and cheap non-recoverable parity on L2. In my view this is not a minor issue it's a MAJOR design problem. As everyone knows most companies today subcontract the majority of the actual product work to outside original equipment manufacturers and Sun is no exception, BUT, the final quality of the product is the foremost responsibility of the brand manufacturer, in this case SUN. There is even a rumor that extensive email went from both the SUN CPU suppliers to Sun about this design and its potential failure problem.. Let's say you are Ford motor company you assemble most of your products, Dana Assembly sends you complete auto chassis for your top of the line model, but forgets or intentionally does not, because they where directed not to, put brakes on the chassis, Ford assembles and distributes, again without the brakes. Customer has an accident, no brakes. Who is liable. Doesn't take a genius to figure it would be Ford.. But you say " Who would be nuts enough to leave out the brakes?" My point with ECC " Who would..." NO other enterprise vendor uses any of the cheaper options ALL use ECC, it's a cost issue plain and simple, ECC costs 30% more to include and so Suns bean counters killed it in the design faze. OK let's move on, You know you have a problem with the design of you ENTIRE premium product range, what do you do? Be entirely honest and tell your customers what's up and how you are going to fix it. Try and blame someone else. Go into Denial, that anything is wrong. SUN still today is oscillating between 2 and 3 . hoping that the problem will disappear, but as we know it's a SUN engineering design problem and they don't vanish, they are like pesky relatives, they hang around forever. The first people Sun blamed for the problem was its customers and now its suppliers. What would make them do this? Maybe it's the fear of the potential liability? That's speculation of course, but let's look at the kind of impact this problem can have for a customer who encounters the problem. The following is a true story, and like every exciting tale, names and places have been eliminated to protect the innocent.... Take a typical $100 million dollar a year financial company whose entire company operation runs on SUN hardware running Solaris software with Server client desktop Sunrays. The ideal perfect utopia SUN customer. SUN's market pitch when the customer bought the system was, Large, fast rock solid reliable, easy to maintain, central servers running thin desktop clients. The best of the best, the premier UNIX based business system. SUNs marketing argument is that desktop PCs attached to smaller servers have a high maintenance and failure rate, therefore bigger more reliable servers with less intelligent desktop units is much better. Pay big bucks up front and get the best of bread. True argument as far as it goes except when the big SUN server fails more often than its humble PC desktop. At least with the cheaper PC, even when a server fails some form of local work can still be done. Primary SUN server is down and so is your entire company. One day a central tier one primary application server just quits and brings the entire companies business to a grinding halt. The corporate Sysad's are rapidly getting older and white haired trying to trace the problem without success, screams are now being heard from the corporate wing "What the hell have you clowns done" and other juicy comments. Luckily enough they have also bought the best support package possible " GOLD", they call the support hotline and a local support swat team is dispatched, they too cannot find the problem and so they take the entire server apart and reassemble it and guess what, it starts up and runs with no problem. Big sigh of relief and our MIS/IT system heroes go of to fight another battle.