CERIAS - Center for Education and Research in Information Assurance and Security

Skip Navigation
CERIAS Logo
Purdue University
Center for Education and Research in Information Assurance and Security

Customer (dis)service

Share:

As our technology becomes more complex, it is often shipped with flaws and missing features. The evolution of the Internet coupled with a “must ship” attitude has led to a number of interesting business practices. One in particular, remote updates/patching, presents some interesting reliability issues.

One of the best known versions of remote patching is the software update function, currently found in many computer applications, and in most common operating systems. In its usual form, this is a system that can download patches or whole new software artifacts to address a newly-discovered security vulnerability. Some systems are automated, but most require manual intervention. The current systems generally only involve security fixes and no functionality improvements—the functionality improvements may or may not be bundled in less frequent updates (service packs), or they may be deferred into a major revision that requires additional payment.

Many of us working in security and reliability have expressed concern about these updates, because if the update mechanism is somehow hijacked by an attacker, it can be used to quickly distribute malware to large numbers of systems at once. There have also been examples where updates accidentally deleted critical files or provided faulty configurations, thus disabling or degrading many, many machines at once (for example this one and this one). Most vendors have elaborate systems in place to test and verify such patches, and they have plans in place to quickly respond if something goes wrong.

Now, we’re seeing the same concerns begin to occur with consumer goods that aren’t primarily intended as interactive computers. I can speak from personal (unfortunate) experience that at least one major vendor appears clueless and not customer-friendly.

I recently purchased 2 Samsung Blu-Ray DVD players: a BD-P2500, and a BD-P1500. Both have Internet connections for firmware updates and Blu-Ray Live. The BD-P2500 also supports live streaming of Netflix content.

A couple of days after Christmas, the 2500 froze up. I could not get it to respond to anything, including the factory reset code. I contacted Samsung and was given information to send the player in for service—it was still within warranty. They’ve had it for nearly 2 weeks with a status of “waiting for parts.” It has now been broken longer than it was working, with still no prognosis about when it might be returned.

No problem—I still have the other player I can use, right?

The 1500 came up with an on-screen message early in the week that a firmware update was available. Having had experience with downloads and upgrades of OS components, I waited a couple of days before doing anything. When I initiated the download, it completed without error, according to the display. However, after completion, it too was dead—no response to anything, including reset codes. So, I called Samsung again. The problem was escalated in customer service. This is what I was told:

     
  1. There was a bad update put on the servers, and many players that got the download have frozen up
  2.  
  3. They do not have a fix for it at the current time and have no idea when one will be available
  4.  
  5. I should check their WWW site once a week to see when an update is available. “It should almost certainly be within a month.”
  6.  
  7. Even though it is their fault for putting up a bad firmware update, if I am required to send in the player, it is now out of warranty for service so it is my own expense.

It seems fairly clear that Samsung has a major problem in testing and assurance, and a surprising lack of concern for customer support. It also sounds like they don’t have much of a handle on what it will take to fix a locked-up player.

I wonder how many other people around the world are stuck with non-functional players and a vague answer about the fix? It could well be in the thousands. And the best they can offer us is to check the WWW site once a week to see when they are ready for us to pay to install a fix to a problem they caused in the first place!

As someone who works in security and reliability, I can see all sorts of interesting problems here involving updates to consumer appliances. They problems are magnified with incomplete or incompetent responses from the vendors. It certainly suggests that consumers should press vendors to issue things that work correctly and don’t require updates—or at least have a fail safe state that allows recovery! Imagine losing use of your TV, phone, refrigerator or car indefinitely because of a faulty update caused by the vendor, with an indefinite fix. For those with malice in mind, this would be a great thing to do to harm a company—and maybe to extort some money as “protection.”

As a consumer, I’m rather angry. I don’t expect to buy anything else made by Samsung, and I certainly won’t recommend them to anyone else. You may choose to use this as a cautionary tale in your own pursuit of consumer items and choose another vendor that is more careful with their updates, and more considerate of customers who have paid for their products. And if you have one of the frozen players with some idea how to recover it to working condition, I’d be interested in hearing about it.

Sadly, caveat emptor.




Update 01/19/09: Samsung is shipping me a replacement for my bricked P2500. It left their plant on Friday, surface UPS. So, that will be a 3-week turnaround.

Meanwhile, I called the service number again about the P1500 and pressed until they escalated me to “executive response.” (Third or fourth level customer service, I guess.) I kept reminding them that it was their firmware update that caused the problem. After 30 minutes on the phone, I must have worn them down sufficiently: they extended the warranty through this week, and are providing me the shipping information to send it in for service under warranty. Hooray!

Unlike last week, the personnel I talked with today were uniformly helpful and informative. I wonder if they have had enough complaints that there has been a change in policy? Or did I just get two really bad service reps in a row last week?

Nonetheless, the bad updates and the lack of a failsafe are really poor design.

 

Comments

Posted by CCD
on Sunday, January 11, 2009 at 10:46 AM

I have the same problem with iTunes and Quicktime.  They have released several updates over the past two years that have broken my system’s ability to play THEIR video content that I paid for.  They deny all accountability on the message boards, but it is readily apparent that there is a problem affecting thousands of people.  Usually within 1 to 3 weeks they come out with another update to fix the problem.  In the mean time I have to roll back to vulnerable versions if I want access to my paid content.

Posted by Milo Velimirovic
on Monday, January 12, 2009 at 10:12 AM

Thanks for the heads up on Samsung. One less vendor to consider. As much as I’m not a fan of litigation to solve problems, this situation sounds like it’s ripe for a class action suit.

Posted by Carlie J. Coats, Jr.,Ph.D.
on Monday, January 12, 2009 at 10:33 AM

IANAL but my wife is (and lots of things seep in by osmosis).  For the BD-P1500, I don’t see how (1)-(4) can fail to be a tort—legally actionable active misbehavior on Samsung’s part.  And this is the size of claim which should be adjudicable in small claims court (details vary from state to state), and I seriously doubt they could/would defend it successfully.  They should be made to pay… and the whole process documented on the web.  That should get their attention. excaim

Posted by stu
on Monday, January 12, 2009 at 11:06 AM

Are you telling me, as someone who works in security and reliability, this never occured to you with other brands nor many years ago in a far far away long ago finally admitted vendor statement nor, more specifically, ever occured behind the scenes un-noticed until attempt to burn or review your firewall logs?

I’ve yet to have such failure thru updating hardware but the pile of hardware which failed and subsequent firmware updates never ever released, or failed to fix the hardware or got the hardware running properly again but only for a very short period of time has grown high in my closet over the years.

Hence why I have learned Linux is capable of backing up your firmwares (should come on the included with purchase cd in a pro security reliability vendor environment) and I wish I had at the tips of my fingers the url for the .exe capable of doing so within windows.  Many many years ago I asked a few sites about such software and years later found it was created and then learned from a Linux person, someone who works in security and reliability and fixing things field, of its ability should I be capable of learning the process.

I’m just happy that Purdue.edu seems to have gotten its act together as I no longer see   entires from Purdue within my firewall log unless I actually visit a Purdue site.  Took em forever!  And what is with all the bad linux images mirrored at Purdue or more specifically why does it take so long for Universities to realize their files have been altered?

For example, over a year after the Love ?? worm/whatever was finally discovered, university pages still contained and distributed it.  Cudos they didn’t take the Microsoft approach instead responding with thank you’s and appologizies rather than denials, lies or not contact back.

Posted by Steve
on Monday, January 12, 2009 at 04:16 PM

I also have a Samsung BD player. I was faced with the “update?” prompt, and answered yes.

My player hung up on the fourth step, “Do not turn off while update is proceding”.

I waited a couple of hours, and then was forced into killing the power.

Luckily, the update failed, and the player will still work.

I’ll be unplugging that network drop for a while!

Posted by JSK
on Monday, January 12, 2009 at 04:46 PM

In addition to thorough QA testing of updates, embedded systems like this should always have a base level of functionality that allows them to store enough intelligence to recover from a bad firmware update. Memory is relatively cheap now, so giving devices either 1) the ability to revert back to a backup firwmware version, or better yet, to at least be able to perform a download of a newer version when it becomes available should be built into the system.

Posted by Ben
on Monday, January 12, 2009 at 05:59 PM

The creed “if it ain’t broke, don’t fix it” used to be the domain of the less-motivated; it’s now becoming 21st century wisdom.

Posted by Peter
on Tuesday, January 13, 2009 at 12:55 PM

> Even though it is their fault for putting up a bad firmware update, if I am required to send in the player, it is now out of warranty for service so it is my own expense.

Can’t you argue that the warranty period is restarted by the update?

Posted by Trevor
on Tuesday, January 13, 2009 at 02:34 PM

A friend once found a performance tweak for his Honda sports car. It let him turn off the A/C while the dehumidifier was running, which he liked because the A/C robs the engine of a small amount of horsepower.  The procedure to turn off the A/C goes like this:

1) Turn vehicle off.
2) Turn fan knob off.
3) Turn temperature knob all the way to the left (blue area).
4) Turn vent knob to panel only setting (next to max a/c).
5) Hold down both air recirculation button and a/c button while starting car.
6) Continue holding for 7 seconds until air recirculation button stops flashing.

This has nothing to do with DVD players, of course; but the fact that this is a user-accessible ‘hidden feature’ is what I’m getting at.  There are no special tools required, only obscure information.  It is essential that the procedure for repealing a failed firmware update on a DVD player or TV be 1) in existence and 2) easier than the procedure detailed above.

Posted by gruvenwagon
on Tuesday, January 13, 2009 at 03:20 PM

Hewlett Packard has had a similar problem with their IPAQ 310 GPS navigation system.  Apparently the device is designed without a baseline version of the firmware in non-volatile storage.  So, if a firmware update goes awry, there is absolutely no recourse but to send it into the factory - even a manual reset from the “engineering menu” is insufficient to resuscitate the unit.

This problem has been exacerbated by excessive bandwidth-limiting on their firmware-update site such that no single TCP connection could do more than about 10KB/s - for a download that is often over 100MB.  Of course the download had to occur during the firmware update, you could not transfer it ahead of time and then just schlep it over via the micro-SD card reader built into the unit.  Thus the typical user has a big multi-hour window to accidentally bork the firmware ugprade process.

In the USA at least, HP has been pretty good about providing free shipping both ways to repair these units, but word is that people in other countries have not been so well cared for.

Posted by Chris
on Monday, January 19, 2009 at 09:16 PM

I’m all for self updating, as long as their is an “undo button”.  I like the motherboards with two bios chips, (apparently, I’ve never tried) they can boot from either chip, so you can install a firmware update to one and if it fails you still have something bootable to revert your changes with.  It would be nice if they did exactly the same thing with general hardware, with a readonly chip that has the factory defaults and not upgradeable chip, and a readwrite chip that will be automatically updated.  It would also need a button on the back to reflash the read chip to the readwrite chip without having to fully boot into the device.

Posted by Jeff
on Wednesday, January 21, 2009 at 12:42 PM

I also purchased a Samsung bd-2500 Blu-Ray on Christmass eve. The player worked great. I updated the firmware that came with the unit and did the updates that were promted by the player.The player just quit playing blu-ray discs.
It would however play a regular dvd. I called Samsung and they wanted me to also send back. I returned the player to the store where purchased and got a new one. I loaded the firmware that came with the unit again but have not done any additional updates. I am afraid to do so. The player will not play some bluray discs from netflix. Is there any known problem with netflix?
Is there any updated information from samsung about the problem? Is it safe to upgrade the firmware now?

Posted by Larry J. Hughes, Jr.
on Monday, February 2, 2009 at 02:06 PM

In the general case I have sympathy.  It’s hard to regression test the universe.  Not in this case.

I’m willing to bet money on three things:

(1) The software that was released has no release notes, or if it does, there is one documented bug on it. Never trust software that doesn’t have a moderately long list of (non-critical) documented bugs. Having such a list demonstrates diligence and calculated risk.

(2) There is no release process, or if there is, nobody in management is required to sign off on it.

(3) If anybody in upper management ever actually cares about this, an inquisition will be held in which heat passes downward in the org chart. This is a classic sign of a company that releases bad software. A CEO that thinks otherwise hasn’t grasped the fact that s/he has effectively made the developers the company’s spokespersons.

My own story, while a little more dramatic, at least had reasonable downtime.  http://www.riskbloggers.com/ljh/2007/04/the-hell-with-oops-from-security-updates/

Posted by Bob
on Friday, February 13, 2009 at 11:23 PM

I couldn’t resist - had to have the bd-p2500. It worked well (I suspect) until I upgraded the firmware. The upgrade completed without errors, but now it doesn’t power on without first pulling the plug. Quite annoying, but I’m not fooling around with customer service. As a software engineer I disapprove of the quality of Samsung’s lack of testing.

Spaf - funny I should find your post while searching for a fix!

Posted by Lana
on Saturday, February 21, 2009 at 09:02 PM

Nice article thanks for sharing

Posted by Houston Texas Pressure Washing
on Sunday, February 22, 2009 at 04:00 PM

I think it’s curious how most technology is created to make our lives easier, but it always seems to make things more difficult.  If only technology had an “easy button” like Office Max!

Daniel Simmons

Posted by dafa
on Sunday, March 15, 2009 at 10:33 PM

nice info.. thanks!

Posted by john rabe
on Wednesday, March 18, 2009 at 02:14 AM

smirk

technology is created to make us lazy

Posted by Mariage Marocain
on Thursday, March 19, 2009 at 06:52 AM

Nice informative post. thank you for sharing

Posted by msn nickleri
on Monday, March 23, 2009 at 11:54 PM

Memory is relatively cheap now, so giving devices either 1) the ability to revert back to a backup firwmware version, or better yet, to at least be able to perform a download of a newer version when it becomes available should be built into the system

Posted by LED Television
on Saturday, October 17, 2009 at 03:34 AM

Yep, untested updates are a definite problem. Reading your blog was the first time I’ve heard of it happening with blu-ray dvd players and so that is obviously something I’ll have to keep an eye out for with mine.

What’s somewhat more surprising to me is the response you received when trying to fix a problem they caused…horrendous. So terribly frustrating dealing with CS Reps that don’t seem to have a grasp of what customer service is all about. They (the company) break your product that put money in their pocket with their lack of quality control and testing, and then they want you to pay to have it fixed. I just shake my head.

That lack of customer service is usually attached to a lack of work ethic as well as a lack of seniority with the company. I’ve worked off and on for a pizza company for around 14 years, and we’ve gone through a few incarnations of what to put in the box to prevent the pizza from sticking. Wax paper, tin foil, freezer paper, and now we don’t use anything. But one time while working I overheard a co-worker angrily arguing with a customer on the phone, telling them how “we never used wax paper in pizza boxes”, telling the customer that we could put tin foil in the box for them or nothing, that is their only two choices, and that is all we’ve ever done. I just shake my head. The pizza restaurant has been open for 25 years, the worker had been there for 3 and she’s arguing with a long time customer about our past practices, just ignorant.
If the customer wants wax paper in their pizza box, put it in their pizza box, we’ve got plenty in the restaurant since we use it in fry boxes. That’s what customer service is about!

But yah, your post is why I’m in the habit of waiting a while before installing an update for my computer or iphone (which has also recently seen some major problems with an update).

About the only thing not worth waiting to test out is antivirus software, that’s one update you want done as quickly as it comes out to prevent much worse consequences.

Great informative post, thanks for the lookout.

Posted by ero
on Friday, February 19, 2010 at 07:11 AM

i lost a nice patch of hair because of Sony’s Playstation 3 tech support..

Posted by Glen "Loves Good Technical Support" Parker
on Sunday, March 28, 2010 at 10:17 PM

Samsung seems to have a name for itself in this regard… I bought a Panasonic plasma TV about 2 years ago, and at that time, was looking also at the Samsung model of equivalent specs.

When I asked about the price difference (Samsung being considerably cheaper), the sales assistant stated “well sure it’s cheaper, but don’t expect it to last”...!

Panasonic was the only choice smile

Have heard similar things about Samsung devices, including terrible technical support and customer service, and won’t spend any significant money on one of their products.

Shame companies like Samsung don’t understand the implications of bad tech support / customer service in an age where a post like yours can be seen by thousands of people around the world!

Better luck next time.

Leave a comment

Commenting is not available in this section entry.