Detailed And Thorough Debunking Of Bloomberg's Sketchy Story About Supply Chain Hack
from the bloomberg-has-some-'splaining-to-do dept
Last week we noted that the general consensus at this point is that Bloomberg screwed up its story about a supposed supply chain hack, in which it was claimed that Chinese spies hacked Supermicro chips that were destined for Apple and Amazon. Basically everyone is loudly denying the story, and many are raising questions about it. In our comments, some of you still seemed to want to believe the article, and argued (without any evidence) that the US and UK governments, along with Amazon and Apple, were flat out lying about all of this. I pointed out a few times that that's not how things work. Also untrue is the idea that many floated that the US government was forcing Apple and Amazon to lie. That also is not how things work (for those who don't believe this, please check your First Amendment case history).
Anyway, over at Serve the Home, Patrick Kennedy has one of the most thorough and comprehensive debunkings of the Bloomberg story, detailing how incredibly implausible the story is. Kennedy's write-up is very detailed, including lots of pictures and detailed drawings of how networks are set up. Here's just a little snippet as an example:
The next inaccuracy to this paragraph is the line describing BMCs as “giving them access to the most sensitive code even on machines that have crashed or are turned off.” That is not how this technology works.
Baseboard management controllers or BMCs are active on crashed or turned off servers. They allow one to, for example, power cycle servers remotely. If you read our piece Explaining the Baseboard Management Controller or BMC in Servers BMCs are superchips. They replace a physical administrator working on a server in a data center for most tasks other than physical service (e.g. changing failed hard drives.)
At the same time, the sensitive data on a system is in the main server complex, not the BMC. When the BMC is powered on, hard drives, solid state drives, the server’s CPU (for decrypting data) and memory are not turned on. If you read our embedded systems reviews, such as the Supermicro A2SDi-16C-HLN4F 16-core Intel Atom C3955 mITX Motherboard Review, we actually publish power figures for when a system is on versus when the BMC only is active. In that review, the BMC powered on utilizes 4.9W of power. SSDs each have idle power consumption generally above 1W and hard drives use considerably more even at idle. The point here is that when the server’s BMC is turned on, and the server is powered off, it is trivially easy to measure that the attached storage is not powered on and accessible.
When a server is powered off it is not possible to access a server’s “most sensitive code.” OS boot devices are powered off. Local storage is powered off for the main server. Further encrypted sensitive code pushed from network storage is not accessible, and a BMC would not authenticate.
This line from the Bloomberg is technically inaccurate because a powered off server’s storage with its sensitive code has no power and cannot be accessed.
There is much, much more in the piece, and it is well worth reading if you still think Bloomberg was on to something with its story.
So far, Bloomberg has stood by its story, even though it increasingly seems clear that its reporters -- Michael Riley and Jordan Robertson -- were in over their heads. It is possible that something questionable happened, but it almost certainly did not happen the way they described it. The fact that Bloomberg has refused to recognize any of these concerns is incredibly damning for Bloomberg's reputation.
Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.
Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.
While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.
–The Techdirt Team
Filed Under: bmc, china, chips, hack, jordan robertson, journalism, michael riley, reporting, supply chain
Companies: amazon, apple, bloomberg, supermicro
Reader Comments
Subscribe: RSS
View by: Time | Thread
1st amendment
[ link to this | view in chronology ]
Re: 1st amendment
[ link to this | view in chronology ]
The fact is, remains, and ever will be the supply chain with electronic devices, especially coming from the PRC, is NOT SECURE and as far as products from the PRC is concerned *cannot* be secured.
Supply chain attacks are not speculation, they are the holy grail of intelligence operations both at the hardware and software levels. In fact, the Snowden files has pointed out very specifically how the US & UK intelligence agencies has gone about it themselves (inserting compromised firmware in transit usually).
I'm sure the reporters did their best at fact checking their personal experience and understanding allowed. This report was years in the making, and it wouldn't have been released unless the authors were very sure of themselves. They both have good reputations for reasonably accurate reporting.
It doesn't matter if some of the details are wrong, what matters is this form of attack is a very real threat to global data security, not just the US. Corporations and governments globally ignore supply chain security at their peril especially with movements growing at holding corporations legally liable for data breaches.
[ link to this | view in chronology ]
Re:
Yes, supply chain attacks exist. That doesn't mean every article about a supply chain attack is accurate.
The "point of the article" was to cover a specific attack, not a general warning.
[ link to this | view in chronology ]
Re: Re:
However, here's the thing about how defamation law works: individual facts can be inaccurate, but if the article is considered substantially correct when taken as a whole, it's considered non-defamatory and essentially true.
And I'm sorry, but one person saying the article misses a few facts, when MANY engineers are saying it's substantially possible as a whole makes me suspect the individual rather than the report.
Bloomberg also wrote a second article that's much more specific about a different supply chain attack and in this case they had a security firm on record as witnessing first hand compromised hardware in action in a telecommunications company USING SUPERMICRO HARDWARE.
https://www.bloomberg.com/news/articles/2018-10-09/new-evidence-of-hacked-supermicro-hardware-found- in-u-s-telecom
Granted it was only one server, so the impact was likely minimal to the overall traffic for the center.
Given the details here, it's not at all surprising nor unbelievable that similar happened to Apple or Amazon even if a few details are wrong in the other article.
[ link to this | view in chronology ]
Re: Re: Re:
[ link to this | view in chronology ]
Re: Re: Re:
Wow, so much wrong packed into one sentence.
First: You're begging the question.
Second: You're arguing that any article which does not result in a defamation lawsuit must be true. That is obviously absurd.
Third: You don't know how defamation works. A story merely being wrong is not defamatory. There is no reason to believe that the Bloomberg reporters were intentionally spreading misinformation with the intention of causing harm. Bad reporting is not the same thing as defamation.
[ link to this | view in chronology ]
Re: Re: Re: Re:
Unfortunately, "causing undeserved harm to the reputation of" pretty much is the colloquial definition of "defamatory".
I'm mildly afraid that the apparently-increasing gap between that colloquial sense of the word and the legal sense of the word is going to eventually force a "correction" in which the legal definition is adjusted to be closer to the commonly understood colloquial one, with all the negative consequences that would involve.
[ link to this | view in chronology ]
Re: Re: Re: Re: Re:
[ link to this | view in chronology ]
Re: Re: Re: Re: Re: Re:
What I mean by that is that if enough people feel for a long enough time that the legal definition is wrong, they will elect enough people who pass enough laws (and/or appoint enough judges who issue enough rulings) which reject the established legal definition that the combined result will be that the legal definition which actually gets used will have changed to more closely matched the colloquial one.
That's basically a description of how democracy is supposed to work: when the law doesn't match what the public thinks the law should be, the public is supposed to get the law changed, by that exact method. It's just that in this case, if the law were what a large part of the public appears to already think it should be, the long-term net negative consequences would be fairly severe.
[ link to this | view in chronology ]
Re: Re:
You drop the bar to ground and bury it.
That every article isn't accurate doesn't mean this attack isn't real.
A general warning is still useful.
[ link to this | view in chronology ]
Re: Re: Re:
Like just because the details of an accusation of murder against someone are shown to be wrong, that doesn't mean that murder isn't real, right? I mean, a general accusation is still useful, right? Besides, if the accused hasn't filed a lawsuit against the accuser it must mean that the accusations are true none the less. Yeah, I see how that works.
[ link to this | view in chronology ]
Re: Re: Re:
[ link to this | view in chronology ]
Re: Re: Re: Re:
[ link to this | view in chronology ]
That the article in question is not accurate does mean the specific attack mentioned in said article is not real, though. If you could prove it was real, well, you would have one hell of a scoop on your hands.
[ link to this | view in chronology ]
Re:
There's an enormous difference between "the supply chain can be compromised" and "the supply chain has been compromised, in this way, at this time, with these targets." The former is, I think, beyond reasonable question (especially since "can be" is a very low bar). The latter? Not so much.
I haven't made it all the way through the STH piece yet, though it does appear to be very thorough. As a counterpoint, this piece purports to explain how something like this would be possible.
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
Re:
Lets start out with Middlemen and corps..
Corps want things as CHEAP as possible When they Buy it, so they can make the best Profit.
Middlemen Balance this, with doing what the Corp says..
Corps had figured that they had Allot of food for the next few years, and they wanted Farmers to Cut prices..(farmers wanted a raise in prices for their goods)
so the Corps STOPPED buying the Fruits and veggies and wheat and most everything ELSE the farmers wanted to raise prices on..
Then they YELLED, that they were SHORT on food(FIRE), and Prices had to go up.
Well, farmers wanted a raise because there hadnt been one in years, and the COST of maintenance and EVERYTHING ELSE had gone up..(this is the time that Farms were not as BIG as the whole state, it was Hundreds of farmers doing THERE OWN THING)..
Well the Farms started Dieing and selling off to PAY LOANS..Loans that were made to Create Crops to sell to the middlemen and to the corps.. Thats HOW CHEAP the food chain was..(even now, 100 pounds of potato's is less then $3 to the farmer, and you pay $1.59(?) at McD's for 1/4 pound, Processing does NOT make this worth 200 times the Cost)( and most other Food stock in this country is about $0.03 per pound)
Between the banks and the corps, the farmers LOST ALLOT of Land to the Corps..
The Corps modernized and made farms HUGE.. And when 1 fails now days its a pretty good hit..Because they dont Diversify the crops..(idaho was the last State to be Food independent, and could supply enough food to feed its own state, NOT anymore)..
with the Corps controlling the farms...Ask monsanto what they are growing this year.. Ask them what experiments they are doing..(yes we can see the Labels out in the fields) they are trying to make things BIGGER, use up LESS ROOM..
The USA tripled its food production..
And the USA EXPORTS over 60% of its grains and Corn, Every year..
(there is more to this, But..)
In the End, it was a lie.
They lie'd to everyone.
Mankind and corporations...the only creature that would sell its mother into Prostitution JUST to make more money, for themselves..
[ link to this | view in chronology ]
Re: Re:
owners/boss..those at the top..
Only want the basics..Because they dont know TECH..
They can ask a general Question of the IT, and get a BASIC EXCUSE.. of WHAT could of happened.
The Guys on top, then give it to the People that make BS sound better..(but dont know tech either)
In the END the IT, MAy not know EXACTLY what happened.. Evne with a Security system moderating and watching everything happening on the system.
Boss reads it and send it out..
90% of these systems are Automated and remote...And 1 person can do most of the work..That took a Good team to do in the past.(that team SAT around allot, but when things went BOOm they were THERE..)
I can get a few sites that SHOW all these systems being Broken into.. And there is no one THERE to Watcvh and monitor whats happening WHILE its happening..and the Servers dont Shut it down..
[ link to this | view in chronology ]
What about the capabilities the BMC does have?
I'm not quite so easily convinced as these guys are. The article on BMC notes that it can be used to position an ISO (equivalent of plugging in a USB with an O/S into the server) and then turn the server on. (Or wait for it to be turned on.)
It seems to me that if one can control the BMC, the server is pwned... even if it does happen to be powered down.
[ link to this | view in chronology ]
Re: What about the capabilities the BMC does have?
PXE boot, is a standard boot technique, and so a standard technique and secured technique is being used as a basis for scare mongering.
[ link to this | view in chronology ]
Re: Re: What about the capabilities the BMC does have?
[ link to this | view in chronology ]
Re: Re: Re: What about the capabilities the BMC does have?
ISO is a disk format, commonly used on DVDs, and PXE includes a network boot option. ISO boot makes no sense, which is about par for the article.
The boot process consists of the BIOS/UEFI mounting a file system, and loading the first stage boot loader from that. The question to be asked is not how is the system booted, but where are the files to boot to system obtained from, and how is the operating system configured to do anything useful after its has booted.
[ link to this | view in chronology ]
Re: Re: Re: Re: What about the capabilities the BMC does have?
The Supermicro BMC (or IPMI) allows the server admin to mount a .iso over the network, so it appears to the machine as a local CD--a very useful capability for installing the OS on a server. I don't know exactly what mechanism the BMC uses to accomplish this, but from the perspective of the main machine, it doesn't have anything to do with PXE.
[ link to this | view in chronology ]
Re: Re: Re: Re: Re: What about the capabilities the BMC does have?
I'm sorry, what did you think PXE boot was again? Because that is literally it. It allows a computer to boot from a network location other than its local drives. And PXE doesn't even require an ISO.
If you are talking about having a running server see an additional drive that is actually elsewhere on the network, well, there's no need for a hardware hack for that because all modern operating systems have that functionality built in, BMC not required.
In addition to that, this is only useful if you've already gained access to the network and have an ISO or other image file ready and waiting, which, if you've already gained access to the network, why do you need the BMC exploit?
If you are PXE booting then you're likely installing a new OS, which means you're wiping whatever is currently on the server. That's going to get noticed by an admin because it's going to take that server down, which will kick off all kinds of alarms and errors and immediately start an investigation.
If you're just using it to attach a "network disk drive" the question still becomes how do you get the OS to do anything with it once it sees the drive? You would have to have a separate exploit that already gained control of the server to actually run something off the network drive. Just connecting it does jack squat.
So again, this is a pretty worthless exploit and technically not plausible as described.
[ link to this | view in chronology ]
Re: Re: What about the capabilities the BMC does have?
UEFI exploits are an actual thing, and PXE boot is based off of UEFI.
The trick here is that some implementations of UEFI (IE from particular manufacturers) do not properly check signatures of the packages they download. Thus, the exploits. UEFI OEMs that follow Intel recommended guidelines have been found not to be vulnerable.
While it is possible to recover from a UEFI rootkit, it's not something the typical user will be capable of, and it takes time. Time == Money. For a corporate consumer, they may find it cheaper buying new hardware. ... from a different vendor.
[ link to this | view in chronology ]
Re: Re: Re: What about the capabilities the BMC does have?
[ link to this | view in chronology ]
Understand the enemy?
In the early days of aircraft production, with the new computer boards, there were some tests on hardware configurations, and software used. It was found that the wiring on the aircraft acted as a antenna. Signals were able to be read from the wiring, and act as actuators for the systems, activating and ceasing or freezing controls. And some of the subsystems broadcasted their location and activity in the aircraft. Why? They changed the boards, same manufacturers, same activity. Similar board, from another factory, problem went away. Why? It seems a nationally known air raft manufacturer used a board from a foreign factory, not their own. Could someone have used a remade device in the aircraft that could have over rode the aircraft controls? Yes, go back and read some military history, not that long ago. Pick up about the development of the f14, and the f16. Early computers, but still applicable, know your supply line. Have trustworthy manufacturers.
[ link to this | view in chronology ]
Re: Understand the enemy?
I suggest you work on it a bit, you know - polish the details and make it more believable and stuff.
"In the early days of aircraft production, with the new computer boards"
I found this one a bit humorous .. in the early days of aircraft as you put it (I assume winged aircraft) would put that in the early 1900's. And then you say they have new computer boards. I realize that computers have been around for some time and they are not necessarily electronic as some are mechanical etc but you said boards which implies it is electronic in nature ... which is obviously incorrect.
"wiring on the aircraft acted as a antenna."
- Humans were well aware of this effect way before the development of wiring in aircraft and have implemented design/test requirements to mitigate such problems. Fault tolerance and graceful degradation are common place among modern aircraft.
But - yeah, gotta watch your supply of incoming parts or you will get screwed.
[ link to this | view in chronology ]
Restoring credibility
Bloomberg could, in one fell swoop, recover all that lost credibility.
How? By providing a detailed description of:
The first two would not reveal their sources; the last ... could, if the hardware was serialized in some fashion. (See, for example, how Reality Winner was caught.)
There is a lot of SuperMicro hardware out there, and I'm confident that Apple, Amazon, etc could track down hardware of the relevant vintage. But without the ability to independently corroborate the story, it is just click-bait.
I've no doubt that plenty of researchers have told Bloomeberg and the reporters precisely these points. That the reporters haven't provided this information is telling. It was excusable in the first day or two of the story - barely. After a week? Not so much.
[ link to this | view in chronology ]
A BMC, and the Redfish API that SuperMicro uses to provide access to their BMC and Remote Management Module, are designed to allow for remote access -- full remote access, as in like having a keyboard and monitor attached to the system -- to a server. It has complete control over the hardware, including power and booting from remote media. While the article from Mr. Kennedy is correct in that the drives, if not powered up and spinning aren't accessible, and there generally isn't enough power to pull data off a solid state drive in a powered off system, that doesn't completely discredit that something has been compromised.
One of the things that makes security easy to compromise is assumptions like this -- it isn't possible to do that. In case you hadn't noticed, Silicon Valley isn't the only place that actual innovative changes in technology take place. China has had a robust economy for decades to provide resources from, has had a state-driven goal to be disruptive in the world political situation, and is full of really, really smart people.
You don't know what they're capable of.
While you can argue that Bloomberg was irresponsible in their reporting and perhaps used overly sensationalized language, here's where all these nay-sayers go wrong -- they haven't proven it ISN'T compromised. They all base their assumptions on what they believe to be possible. It wasn't all that long ago people didn't think it was possible for Stingrays to work they way they do, but here we are. It wasn't all that long ago it wasn't possible that the NSA was listening in on American citizens, yet here we are.
This is how major security failures occur. We assume something isn't possible so we ignore the early warning signs that it is, and disregard the early signs that it indeed has. We ignore the signs that individual cloaking devices are possible until the thief is cloaked in our house and taking our TV, then try to come up with lots of excuses for what's REALLY going on.
Not to overly simplify things, but we created this situation by forcing the supply chain overseas. We knew this was coming, and attacks have happened, so this head in the sand viewpoint is irresponsible. Do we know for a fact that the SM systems are compromised? No, but where there is smoke, there is generally a smoldering that could turn into fire. There is significant reason to suspect something is amiss here.
What is described in the Bloomberg article is NOT impossible just because you think it would be hard to do, don't have enough influence and power or even just enough imagination to make it happen.
[ link to this | view in chronology ]
Re:
Also, isn't that remote access usually tied to a separate internal network, as in make sure it is not directly accessible over the internal data network, never mind the Internet. A compromise of a system is only useful if it can communicate with the outside world. The big data centers take security of management systems seriously, knowing full well what they can be done over such interfaces.
[ link to this | view in chronology ]
Re: Re:
But here's the thing. If the BMC is compromised, we don't know that it isn't doing things on its own to create new paradigms.
The BMC itself doesn't have to be the transport mechanism. If there was a way to ride the BMC control mechanisms and bridge out to the prod network it would be enough. In theory that's possible. If you're working at the PCB layer, connected to the very traces on the board themselves, all manner of things are possible. Just because none of us have figured out how to do that sort of thing doesn't mean it isn't possible.
[ link to this | view in chronology ]
Re: Re: Re:
Also consider that Huawei has had similar accusation made against it by some parties in the US government, and which has been debunked, and this story smells of more of the same, but partly built on what is standard server practice.
[ link to this | view in chronology ]
Re:
Tthe BMC is interconnected a bunch of ways to the system it's in, and the STH guys show this dropdown selector to select between the BMC's own ethernet interface and two others (that are wired in to the board) and declares that there is therefore no way for the BMC to reach the internet if the BMC is on an isolated network. This assumes a bunch of things about the BMC, including that it's running unhacked firmware. The idea that because the BMC's web UI says "hey, I'm using only my dedicated ethernet port" that we can believe that without question is crazy. The BMC also ties into the main host's USB bus directly. We have seen vulnerabilities that leverage just basic USB bus access - again, how do you know the BMC is not doing evil things? How do you know it's not shipped in a compromised state? How do you know that the tiny added component or components aren't being used to quietly reflash exploits onto the BMC?
It all just reads like it's written by someone that has lots of faith in absolutes, and if anything has taught us computers are a fucking mess of unpredictable behavior, it's security folks. The STH person does not sound like a security researcher to me...
[ link to this | view in chronology ]
Not sure why this is such an issue
[ link to this | view in chronology ]
Re: Not sure why this is such an issue
[ link to this | view in chronology ]
Re: Not sure why this is such an issue
OK, let's agree nuclear weapons are (potentially) very very dangerous to lots of people. And let's agree agree that vicious dictatorships from Korea to Pakistan have the capability of exploding nuclear weapons.
So what would you say if Fox News insisted the Russians had exploded nuclear bombs over New York City, San Fransisco, and Dallas? and then, when the mayors of NYC, SF, and D. broadcast outraged denials, suppose Fox insisted that it was really the Pakistanis that had bombed Boston, Wichita, and Spokane. And when homeland security carefully investigated and said that those cities were perfectly intact, Fox announced that it was "standing by its story, based on 100 years of investigations" and "it had one witness that a nuclear device had exploded in some other unknown city" ---
WHAT WOULD YOU SAY?
"Well, it's OK to say something happened because it could have happened, and 100 interviewees could be found who were too ignorant to know whether Boston was actually standing"?
Or, would you say Fox was a liar, harming the credibility of the entire network and undercutting the credibility of anyone who was opposed to nuclear weapons?
This is exactly the same scenario. Bloomberg is unquestionably saying something happened which did not happen, but which was vaguely similar to something else which could conceivably be made to happen by a sufficiently wealthy and motivated party. That's exothermically-oxidizing-trousers prevarication.
Bloomberg is now on my list of sources that shouldn't be trusted to confirm the direction of sunrise or the color of the sky.
[ link to this | view in chronology ]
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
So the argument is that...
Somehow, to me, this is not a convincing counter argument.
[ link to this | view in chronology ]
Re: So the argument is that...
[ link to this | view in chronology ]
Re: Re: So the argument is that...
[ link to this | view in chronology ]
This was persuasive in the days before secret courts. Now, much less so.
[ link to this | view in chronology ]
Bloomberg's incentives are bad
[ link to this | view in chronology ]
Free Classified Site in India
[ link to this | view in chronology ]
Free Indian Classified Site in India
[ link to this | view in chronology ]