Seattle Public Radio Station Manages To Partially Brick Area Mazdas Using Nothing More Than Some Image Files
from the may-we-offer-you-a-tote-bag-bearing-our-logo.null dept
Welp. This isn't going to help future fundraising drives. Not when a public radio station is negatively affecting, you know, driving.
Drivers of certain vehicles in Seattle and other parts of Western Washington are shouting at their car radios this week. Not because of any particular song or news item that’s being broadcast, but because an apparent technical glitch has caused the radios to be stuck on public radio station KUOW.
The impacted drivers appear to all be owners of Mazda vehicles from between 2014 and 2017. In some cases the in-car infotainment systems have stopped working altogether, derailing the ability to listen to the radio at all or use Bluetooth phone connections, GPS, the rear camera and more.
Behold the collision of OTA and IO(car)T. This unique situation -- limited solely to drivers in the Upper West -- presented a host of new problems and a lot of speculative answers. The radio station had absolutely no idea why this was happening. One local dealership told a customer it had something to do with 5G, which apparently meant affected Mazdas were now infected with a car-borne form of COVID, presumably necessitating plenty of expensive diagnostics and what have you.
Fortunately, the cars' manufacturer was actually able to pinpoint the cause of the malfunction -- which left some drivers staring at in-car systems stuck in a perpetual "loading…" loop. The answer arrived roughly a week after the problem presented itself. The problem -- discussed in this entertaining Reddit thread -- had nothing to do with network upgrades or an unexplained bug in Mazda software.
Instead, the public radio station had done something completely unexpected, sending affected vehicles into in-car entertainment purgatory. This is the statement Mazda gave to Geekwire.
“Between 1/24-1/31, a radio station in the Seattle area sent image files with no extension, which caused an issue on some 2014-2017 Mazda vehicles with older software,” the Mazda statement said. “Mazda North American Operations (MNAO) has distributed service alerts advising dealers of the issue.”
While it's somewhat troubling to note that Mazdas manufactured within the last eight years are running what Mazda considers to be outdated software, the good news is that it can be fixed. The bad news follows the good news: due to shipping constraints affecting goddamn everything, drivers affected by this oddity shouldn't expect to see a fix anytime soon. "Part delays" cited by Mazda could put permanent fixes months off.
On the other hand (good news!), even older models will be covered by these repairs, whether or not they're still under warranty. The company has instructed dealers to honor "goodwill requests" for free repairs of affected vehicles. Back to the bad news: the part that apparently needs to be replaced is the ominous-sounding "connectivity master unit," which indicates a whole lot of connectivity will be affected until dealers get the part in stock and start dealing with the backlog of semi-bricked Mazdas. Some users have reported in-car entertainment systems stuck in permanent loops, non-functioning GPS systems, and bricked back-up cameras.
This isn't going to go well for Mazda, considering it's the only manufacturer whose systems have been rendered useless by a misconfigured file distributed by a radio station. While this situation is certainly an outlier, there's likely a reason other in-car entertainment systems weren't similarly affected, which suggests a crucial shortcoming in the tech installed in those models -- one that could be exploited by entities far more nefarious than local public broadcasters.
Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.
Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.
While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.
–The Techdirt Team
Filed Under: images, public radio, radio, seattle, washington
Companies: kuow, mazda
Reader Comments
Subscribe: RSS
View by: Time | Thread
""Part delays" cited by Mazda could put permanent fixes months off."
While I'm happy that it seems that no actually vital car functionality was affected
[ link to this | view in chronology ]
Re:
Going off what I read on this a few days ago it corrupts some critical part of the firmware and the devices don't have an easily access port to allow them to use something like JTAG to force load good firmware so that at least dealers could fix it in the field..
Pretty epic fail.
[ link to this | view in chronology ]
Re: Re:
Damn... yeah if that can happen anyway it's pretty bad design, but from what I understand here all that caused it was a file being sent without an explicit extension specified? Wow...
[ link to this | view in chronology ]
Re: Re: Re:
All of this. Wow. Depending on file extensions. Files without extensions wreck firmware, requiring hardware replacement. What the actual hell?
[ link to this | view in chronology ]
Re: Re: Re: Re:
So what these idiots forgot to install the magic database and file utility?
Quick someone try sending it some random binary executable. Chances are it will just run it anyway....
/s
In all seriousness, why would this even cause such a huge failure to begin with? It's unvetted untrustworthy data coming into the system from a remote source via an interceptable wireless transmission. Who the fuck is depending on the file extension for anything here?!?!? And why would they save it to the non-volatile storage for re-use when it hasn't been checked yet!?!?!?!? (It obviously hasn't been as the whole system fails when trying to check the damn thing.) Multiple someones at Mazda need to loose their jobs over this. I wonder if someone can find a unsigned code execution exploit here. (Or just in general to get these "faulty" parts working again during a parts shortage....)
[ link to this | view in chronology ]
Re: Re: Re: Re: Re:
"Who the fuck is depending on the file extension for anything here?!?!?"
I've worked for a number of companies where tight deadlines, overworked developers and management being willing to turn a blind eye to obvious problems to hit a profit level have led to some bad outcomes. All this really takes is developers not taking into account a use case they didn't think of, and an underfunded testing department (who are always considered the enemy but people trying to release a new revenue stream) not returning those tests or have their tests ignored in favour of gambling that they can make a lot of money before it's exposed.
[ link to this | view in chronology ]
Re:
Not true. I've worked in this industry, and know that US regulators consider the rear-view camera to be vital (regardless of how well any individual driver thinks they can reverse without it). New cars are required to have them, and they have to be working within 2 seconds of entering reverse—even if "entering reverse" happens right when you turn the car on. That makes for some interesting programming: the camera may need to run before the whole operating system has booted, and on some systems runs with no OS involvement at all.
Some affected person, therefore, should make a complaint to the regulator, especially if they're getting any kind of runaround. It would not be at all surprising for them to force a safety recall in response.
[ link to this | view in chronology ]
It takes a few lines of code, at most, to validate input, Mazda. That's programming 101. Embarrassing.
[ link to this | view in chronology ]
Re:
"Validation" seems the completely wrong thing to do. Do we even know that the radio station sent something invalid, i.e. violating some specification? It's certainly not invalid, in general, for a computer file to have no extension, or to have a percent sign in the name, and there's no good reason to reject these. We might be lucky they didn't send a name like "../../etc/passwd", or a malformed image file.
The software developers could have just hashed the frequency and the provided filename, and the call letters if known, and used a hex/base64 version of that as the filename.
"A few lines of code", though? I don't think you work in this field. It's a few lines of code for every piece of incoming data, and they have to be the right lines of code. I've worked with developers who could hardly write such code to save their lives; it's like banging my head against a wall trying to get them to see the problem, let alone fix it after I've explained in detail. It's pretty obvious that Mazda does not have the best people working on this.
[ link to this | view in chronology ]
Absolutely hilarious! I would love to see videos of people screaming at their radios/entertainment centers.
Is that mean?
[ link to this | view in chronology ]
Re:
Depends. Are they doing it stopped at a red light, or while they're speeding? It's not the fault of the car owner that they bought a car with incompetent software developers, but it's also not the fault of the people they drive in to...
[ link to this | view in chronology ]
Auto(lol)ated systems
Love the thought.
BUT WHY?
Computerized radio?
1 standard FM station can screw up allot of things?
takes me back to a person in germany being able to do basic controls in the Same model of car in Australia.
But from an FM station?
[ link to this | view in chronology ]
Re: Auto(lol)ated systems
"BUT WHY?
Computerized radio?"
In theory, it's nice to see the name of the song being played. Some stations have already apparently been abusing this to show ads instead of track information, but you'd also expect the radio to do some basic checks on the inputs.
[ link to this | view in chronology ]
Re: Re: Auto(lol)ated systems
They could've done something like TV's "closed captioning", which worked in the 1980s without requiring full computer operating systems in the televisions; or caller ID which could send names in the '90s. What they actually did is called "HD radio", being short for "hybrid digital" while making people think of "high definition". It's a totally proprietary way (incompatible with pre-existing international standards) to cram one or two extra low-bandwidth channels next to the ordinary and unenhanced radio station, along with metadata like song names. Being overcomplicated, then, is a feature: it lets the patent-holder collect licensing fees from everyone involved.
[ link to this | view in chronology ]
Re: Re: Auto(lol)ated systems
But its affecting everything in the car.
So a 2nd system is getting it and sending to the Radio.
If its only coming from the FM, why isnt the radio the only one affected?
To much Embedded/integrated automation. Think of what happens if you CHANGE the radio. The secondary system is still there, Doing WHAT?
Cops wanted a remote control system years ago, to shut down cars in a chase.
[ link to this | view in chronology ]
Re: Re: Re: Auto(lol)ated systems
"But its affecting everything in the car."
No, it's affecting things that are tied to the central console and the entertainment/GPS functions. From what I understand, the car is still functional, you just lose those extra features that people didn't have at all 30 years ago.
"If its only coming from the FM, why isnt the radio the only one affected?"
Because multiple services run on the same OS that's been affected.
[ link to this | view in chronology ]
Re: Re: Re: Re: Auto(lol)ated systems
You need an OS for a radio?
Forget buying a new car as they can Tune you to anything they want.
Lets hack the car over FM.
[ link to this | view in chronology ]
Re: Re: Re: Re: Re: Auto(lol)ated systems
"You need an OS for a radio?"
No, but if you do if you have a multifunction device in the centre console that does things like GPS, phone syncing, display parking cams, and various other applications on top of just accessing an AM/FM broadcast.
But, this still has bugger all to do with the operation of the car overall. I'm sorry if your obsession over fixing the dashboard to a primitive state is confusing you when people are relating to new tech.
[ link to this | view in chronology ]
Re: Re: Re: Re: Re: Re: Auto(lol)ated systems
Actually, you need at least a CS, a controller system, to tune digital radio.
[ link to this | view in chronology ]
"The problem... had nothing to do with... an unexplained bug in Mazda software."
Proceeds to discuss the newly discovered bug in Mazda software which caused the problem.
While it's somewhat troubling to note that Mazdas manufactured within the last eight years are running what Mazda considers to be outdated software
How is that troubling? While Tesla produces cars which constantly phone home to request software updates, few other car manufacturers do so. Based on techdirt's articles on both IoT devices and ownership issues related to software updates (including Tesla's recent woes), I'm surprised to see techdirt disapproving of a company not following their lead.
[ link to this | view in chronology ]
Re:
I saw that bit about the "had nothing to do" as well, but you beat me to it.
For my money, "over the air" software updates to cars is a catastrophe waiting to happen. You can hack a phone? Your car effectively has a phone, calling in for updates. Whatever you can update over the air can be hacked. If nothing else, a supply chain hack.
How much would you like to wager that the NSA does not already deploy some vehicular surveillance through entertainment systems already?
[ link to this | view in chronology ]
Re:
On my end, it's troubling that they're making excuses for their fuckups. Are they implying it's the fault of the customers for not updating their software? Apparently even Mazda dealers can't even do that without replacing the whole hardware unit.
If people should somehow know that running "old" software will cause their cars to break, then Mazda and their dealers should have known that too, and made it a regular service item. "Change your oil every X distance, and your software every Y years." Since they didn't, all they should be saying is "that's a defect on our part and we'll fix it".
Given that it breaks a mandatory safety feature (the backup camera), they should be recalling all vehicles with the faulty software, not just fixing it for the people affected by this instance of an unexpected radio transmission. Per Mazda, "This device complies with Part 15 of the FCC Rules. Operation is subject to the following two conditions: (1) this device may not cause harmful interference, and (2) this device must accept any interference received, including interference that may cause undesired operation." (Note that while "undesired operation" may be cool with the FCC, the NHTSA is likely to feel differently, especially when the "harmful" transmission didn't, as far as we know, violate any regulations.)
[ link to this | view in chronology ]
Re:
The problem is that most modern cars, Teslas or not, probably phone home anyway! If companies are going to phone home, then they should install security updates too.
[ link to this | view in chronology ]
Re:
Sounds legit. It's not an unexplained bug if you explain it, after all.
[ link to this | view in chronology ]
Re: How is that troubling?
Because, in the real world of hardware, it's not appropriate to release products this shit. Car's have never required an over the air update mechanism because their safety systems are designed to work and tested before release. There's no reason to treat car software any differently.
Holding up Musk, maker of the car that accelerates into the concrete lane divider on an off ramp killing it's drivier, as an example of how to make cars safe is ridiculous. A Tesla is a very expensive battery wrapped in a shit car running beta software.
[ link to this | view in chronology ]
Timely.
"Tesla investigated over 'phantom braking' problem" https://www.bbc.co.uk/news/technology-60432351
"Coming up next on WWTF's Non-Stop Long-drive replays, it's Phantom of the Stopera, followed by Stop! in the Name of Love, Don’t Stop Me Now, Who’ll Stop the Rain, Don’t Stop Believin’, I Can’t Stop Loving You, Can’t Stop This Thing We Started, Can't Stop till you get Enough, You can't Stop the Music, and Stop right now thank you very much "
[ link to this | view in chronology ]
And then there is this: by 2026, vehicles sold in the US will be required to automatically and silently record various metrics of driver performance, and then make a decision, absent any human oversight, whether the owner will be allowed to use their own vehicle. Even worse, the measure goes on to require that the system be "open" to remote access by "authorized" third parties at any time.
This was in the so far not passed infrastructure bill. One can imagine the ramifications of such a requirement in the future.
[ link to this | view in chronology ]
"Back to the bad news: the part that apparently needs to be replaced is the ominous-sounding "connectivity master unit," which indicates a whole lot of connectivity will be affected until dealers get the part in stock and start dealing with the backlog of semi-bricked Mazdas."
Ah, yes. Mazda. Hiroshima's revenge.
Just imagine what they could do to us if we ever go to war with them again.
[ link to this | view in chronology ]
Poe's law?
I hope you aren't seriously trying to imply that this was on purpose somehow.
[ link to this | view in chronology ]
Re: Poe's law?
I think some people are still very bitter about how the US car industry had its ass handed to it decades ago
[ link to this | view in chronology ]
Re: Re: Poe's law?
Well, the fact that Subaru, Kia and Mazda are starting to run into the same problems of bad design and obsoletion the US car industry ran into in the 70's may be a good indicator of yet another paradigm change when it comes to which country gets to host the biggest carmakers.
I honestly hope these OEM's get their shit together and fundamentally rethink the fact that if they want to add computerization to their product then that needs the same attention they'd give to designing new engines.
[ link to this | view in chronology ]
That is one reason to have a phone with FM radio in it, and than bluetooth it to the stereo that way. I would never use the radio on modern cars, I would either use FM radio feature on my phone, or listen to the Internet stream.
[ link to this | view in chronology ]
I'm sorry Dave, it's gif not jif.
[ link to this | view in chronology ]
Re:
I don't know, that gif has quite a lot of nuts in it..... :P
[ link to this | view in chronology ]
If you stare hard enough
IO(car)T looks like a racecar facing left.
[ link to this | view in chronology ]
Modern cars are often hard to repair (i.e. defective) by design.
Perhaps fixing this issue wouldn't be as hard as it is if car manufacturers had designed their cars to be repairable by independent drivers and third-party repair garages. In other words, if car companies had respected the right to repair in the first place, then this Mazda radio fiasco might not have happened in the first place. Granted, Mazda is a Japanese company, so I don't know how exactly the US government should go about passing a law protecting the right to repair.
[ link to this | view in chronology ]
Weighted fault
Wonder who is actually legally responsible here.
Mazda, that didn’t implement an error routine for damaged files?
Or the station that sent out a damaged (mis/mail-formed file) file that bricked car computers?
Given the timely example, there’d be hell to pay in class actions and corporate suits if TurboTax bricked systems with a bad icon.
As a side pondering, what OS/CS bricks on a missing file extension?
[ link to this | view in chronology ]
Re: Weighted fault
I'd imagine that Mazda would be the ones taking the blame. It's unlikely that the radio station did anything maliciously by not formatting the files correctly. Even if they did, then the fact that no other manufacturer's cars were affected still puts the onus on Mazda for not having a way to record from the error or properly sanitise inputs.
"Given the timely example, there’d be hell to pay in class actions and corporate suits if TurboTax bricked systems with a bad icon."
It's again worth noting that from what I understand the car wasn't bricked, it's just that you'd have to drive it around as if it were a car from the 80s till it gets fixed.
"As a side pondering, what OS/CS bricks on a missing file extension?"
Lots of OSes, especially embedded ones, can be vulnerable to all sorts of things that were missed during testing or that some project manager decided to let slide to meet a deadline because they were unlikely to happen. Most of them aren't vulnerable to something as basic or from an FM broadcast, but there's a lot of things going on in all sorts of devices that make you wonder how anything actually works long term.
[ link to this | view in chronology ]
Re: Re: Weighted fault
My guess is if the two companies filed against each other the judge would find equal fault.
It’s ultimately how the system is designed. Most companies use third party radios that are secondary systems.
Howeve if the radio is directly built into the primary control computer (modern Vehicles have multiple “computers”) we have a major safety issue. As that’s also the telematics system. Air bags, ABS… it depends on how much is part of that specific system.
Bingo.
What surprises me is this requires parts replacement. And not an OTA update to the OS.
For a modern car you should at the very worst be able to download a USB or SD update.
[ link to this | view in chronology ]
Re: Re: Re: Weighted fault
"Most companies use third party radios that are secondary systems."
Yes, but again, modern central consoles often have a lot more functionality than simple playing the radio. How much they tie into the other safety features I don't know, but it should be the case that such features operate even if that console panel is not present, you just won't have as much detail on them. Although, this is not the time or place to start doing in depth research on exactly how Mazda design their systems.
"What surprises me is this requires parts replacement. And not an OTA update to the OS."
It depend on what actually broke. It's claimed that the part was "partially bricked", which suggests to me that either the OTA update function no longer works (or the OS doesn't start at all), or that it's designed to require physical intervention in the case of catastrophic failure. In the case of physical intervention, it's prudent to replace the unit, rather then just do a local fix and discover in 6 months that it caused another problem that wasn't identified before.
It's like if you discover that your server has been compromised by a rootkit or ransomware - you don't just remove the files you know were infected, you do at minimum a complete wipe and reinstall, and preferably replace the discs completely. Anything short of that leave you open to further infection by what you missed.
[ link to this | view in chronology ]
Re: Re: Re: Re: Weighted fault
… I’m agreeing with you: btw.
Your closer to my realm of experience here: old, custom, and embedded.
Many have no clue just how integrated the modern computer systems in vehicles are.
Take Crs/Fiat. 100 percent of the system is software. There’s no manual override for any of the infotainment or comfort offerings.
Heated items are controlled by apps. Seat positions. Security settings. Safety. All apps.
Blow/brick anything in the system and ftw. Out of luck.
GM is a bit better with more secondary controllers and separated manual options and overrides for basic functions.
I don’t know about Mazda systems myself. But from my dodge and GM experience… if it’s closer to the Dodge method those car fixes are going to be expensive. And painful for drivers until a fix is done.
[ link to this | view in chronology ]
Re: Re: Re: Re: Re: Weighted fault
"Many have no clue just how integrated the modern computer systems in vehicles are."
Sure, but again it's down to which part fails. There shouldn't be any reason why a car would fail on any of its fundamental system because the central console dies, since the main purpose of that is to provide feedback to the driver and provide optional functions.
"There’s no manual override for any of the infotainment or comfort offerings.
Heated items are controlled by apps. Seat positions. Security settings. Safety. All apps."
Some of those things are not like the others. I haven't driver any of the cars you mention so it seems weird to me that things like seat positions can't be controlled manually, and that's is a safety concern. But, there's a vast difference between someone not being able to heat their seat or a mirror until they get a service and actual safety features going down because a screen is no longer available.
[ link to this | view in chronology ]
Re: Re: Re: Re: Re: Re: Weighted fault
Once again we are agreeing.
Personally I have a serious dislike for the consolidation to a touch interface a single controlling computer. — when there is no manual alternative or override!
Let’s take some current gm products, in the higher level packages.
The primary infotainment computer also handles the car’s internal and external sensors. Such as seat weight sensors. This could change how airbags are deployed!
Etc.
[ link to this | view in chronology ]
Re: Re: Re: Re: Re: Re: Re: Weighted fault
Again I'll defer to you on the cars I haven't driven. All I know is that in the car I have, the console itself was an optional extra, I still have manual seat, windows, mirror, etc. controls and the safety warnings appear on the dashboard display, not the central console (which covers GPS, rear parking cameras, CarPlay/Android Auto, etc. - all optional).
[ link to this | view in chronology ]