How Toy Story 2 Almost Got Deleted... Except That One Person Made A Home Backup

from the did-it-break-the-rules? dept

Fri, May 18th 2012 5:01am — Mike Masnick

Here's a random story, found via Kottke, highlighting how Pixar came very close to losing a very large portion of Toy Story 2, because someone did an rm * (non geek: "remove all" command). And that's when they realized that their backups hadn't been working for a month. Then, the technical director of the film noted that, because she wanted to see her family and kids, she had been making copies of the entire film and transferring it to her home computer. After a careful trip from the Pixar offices to her home and back, they discovered that, indeed, most of the film was saved:
Now, mostly, this is just an amusing little anecdote, but two things struck me:

How in the world do they not have more "official" backups of something as major as Toy Story 2. In the clip they admit that it was potentially 20 to 30 man-years of work that may have been lost. It makes no sense to me that this would include a single backup system.
I wonder if the copy, made by technical director Galyn Susman, was outside of corporate policy. You would have to imagine that at a place like Pixar, there were significant concerns about things "getting out," and so the policy likely wouldn't have looked all that kindly on copies being used on home computers.

The Mythbusters folks wonder if this story was a little over-dramatized, and others have wondered how the technical director would have "multiple terabytes of source material" on her home computer back in 1999. That resulted in an explanation from someone who was there that what was deleted was actually the database containing the master copies of the characters, sets, animation, etc. rather than the movie itself. Of course, once again, that makes you wonder how it is that no one else had a simple backup. You'd think such a thing would be backed up in dozens of places around the globe for safe keeping...

Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.

Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.

While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.

–The Techdirt Team

Filed Under: backups, mythbusters, toy story
Companies: pixar

50 Comments

If you liked this post, you may also be interested in...

Reader Comments

Subscribe: RSS

View by: Time | Thread

Anonymous Coward, 18 May 2012 @ 5:07am

It's called Godzilla-proofing. Know it. Live it.
[ link to this | view in thread ]
Lazureus, 18 May 2012 @ 5:16am

And this is why the Canadian/American governments should stop trying to make the act of backing up videos illegal.

You could say it's a different thing all together, but it really isn't.
[ link to this | view in thread ]
silverscarcat (profile), 18 May 2012 @ 5:22am

There's a trope for that!
http://tvtropes.org/pmwiki/pmwiki.php/Main/KeepCirculatingTheTapes
[ link to this | view in thread ]
Anonymous Coward, 18 May 2012 @ 5:23am

Shame on this horrible woman for making an illegal backup of this movie. By doing so, she cost so many workers over a year of salary for restoring it, all for her own selfish greed. Pathetic. When will these freetards think of the workers and the customers?
[ link to this | view in thread ]
Anonymous Coward, 18 May 2012 @ 5:29am

This story gave me a woody
[ link to this | view in thread ]
The eejit (profile), 18 May 2012 @ 5:29am

Re:
No, it certainly is. Without her, we'd have never seen the dreck from after the first one. SHE TOTALLY RUINED IT FOR ME!!!!
[ link to this | view in thread ]
Anonymous Coward, 18 May 2012 @ 5:29am

so, no back ups allowed, even when your copy of the movie is deleted means all the work for the whole thing is lost. good strategy, guys! i suppose that the one 'illegal' copy, regardless of who had it and that it saved some serious arse kicking from happening and a hell of a lot of money is irrelevant? as it applies to any download by anyone for any reason, all i can say is 'but but but piracy'!
[ link to this | view in thread ]
Hans B PUFAL (profile), 18 May 2012 @ 5:53am

Reminds me of ....
Some decades ago I was called to a customer site, a bank, to diagnose a computer problem. On my arrival early in the morning I noted a certain panic in the air. On querying my hosts I was told that there had been an "issue" the previous night and that they were trying, unsuccessfully, to recover data from backup tapes. The process was failing and panic ensued.

Though this was not the problem I had been called on to investigate, I asked some probing questions, made a short phone call, and provided the answer, much to the customer's relief.

What I found was that for months if not years the customer had been performing backups of indexed sequential files, that is data files with associated index files, without once verifying that the backed-up data could be recovered. On the first occasion of a problem requiring such a recovery they discovered that they just did not work.

The answer? Simply recreate the index files from the data. For efficiency reasons (this was a LONG time ago) the index files referenced the data files by physical disk adresses. When the backup tapes were restored the data was of course no longer at the original place on the disk and the index files were useless. A simple procedure to recreate the index files solved the problem.

Clearly whoever had designed that system had never tested a recovery, nor read the documentation which clearly stated the issue and its simple solution.

So here is a case of making backups, but then finding them flawed when needed.
[ link to this | view in thread ]
Anonymous Coward, 18 May 2012 @ 6:00am

Re: Reminds me of ....
That's why, in the IT world, you ALWAYS do a "dry run" when you want to deploy something, and you monitor the heck out of critical systems.
[ link to this | view in thread ]
Chuck Norris' Enemy (deceased) (profile), 18 May 2012 @ 6:17am

Re:
...and a buzz?

/lameness
[ link to this | view in thread ]
john, 18 May 2012 @ 6:30am

Yeah good point, you think that they would be more secure, whats to stop a employee from "saving" a film on their home computer then selling it to someone who would release it online or something? That must have been a wake up call.
[ link to this | view in thread ]
Rich Kulawiec, 18 May 2012 @ 6:30am

Two notes on backups
1. Everyone who has worked in computing for any period of time has their own backup horror story. I'll spare you mine, but note that as a general observation, large organizations/corporations tend to opt for incredibly expensive, incredibly complex, incredibly overblown backup "solutions" sold to them by vendors rather than using the stock, well-tested, reliable tools that they already have. (e.g., "why should we use dump, which is open-source/reliable/portable/tested/proven/efficient/etc., when we could drop $40K on closed-source/proprietary/non-portable/slow/bulky software from a vendor?"

Okay, okay, one comment: in over 30 years of working in the field, the second-worst product I have ever had the misfortune to deal with is Legato (now EMC) NetWorker.

2. Hollywood has a massive backup and archiving problem. How do we know? Because they keep telling us about it. There are a series of self-promoting commercials that they run in theaters before movies, in which they talk about all of the old films that are slowly decaying in their canisters in vast warehouses, and how terrible this is, and how badly they need charitable contributions from the public to save these treasures of cinema before they erode into dust, etc.

Let's skip the irony of Hollywood begging for money while they're paying professional liar Chris Dodd millions and get to the technical point: the easiest and cheapest way to preserve all of these would be to back them up to the Internet. Yes, there's a one-time expense of cleaning up the analog versions and then digitizing them at high resolution, but once that's done, all the copies are free. There's no need for a data center or elaborate IT infrastructure: put 'em on BitTorrent and let the world do the work. Or give copies to the Internet Archive. Whatever -- the point is that once we get past the analog issues, the only reason that this is a problem is that they made it a problem by refusing to surrender control.
[ link to this | view in thread ]
Anonymous Coward, 18 May 2012 @ 6:33am

Re: There's a trope for that!
And there went my morning. I can't resist that place....
[ link to this | view in thread ]
saulgoode (profile), 18 May 2012 @ 6:38am

Re: Two notes on backups
"Real Men don't make backups. They upload it via ftp and let the world mirror it." - Linus Torvalds
[ link to this | view in thread ]
Anonymous Coward, 18 May 2012 @ 7:02am

What I suspect is that she was copying the rendered footage. If the footage was rendered at a resolution and rate fitting to DVD spec, that'd put the raw footage at around 3GB to 4GB for a full 90min, which just might fit on the 10GB HDD that were available back then on a laptop computer (remember how small OSes were back then). Even losing just the rendered raw footage (or even processed footage), would be a massive setback. It takes a long time across a lot of very powerful computers to render film quality footage. If it was processed footage then it's even more valuable as that takes a lot of man hours of post fx to make raw footage presentable to a consumer audience.
[ link to this | view in thread ]
W Klink (profile), 18 May 2012 @ 7:07am

Backups
Everyone knows to make backups, but not everyone thinks to validate and test restoring them.
[ link to this | view in thread ]
You, 18 May 2012 @ 7:08am

Re: Re: Two notes on backups
I should make stickers for hard drives/USB drives/disks:
Proud to be a backup

Let people print 'em for free and let them go wild on their media collection of choice (and I'll never lose the sticker image either). Go ahead, add some character to that aging collection of DVDs, they'll all mold over eventually!
[ link to this | view in thread ]
McCrea (profile), 18 May 2012 @ 7:13am

Re: Human Trumps Radioactive Dinosaur
Godzilla-proofing is far easier than Idiot-proofing.
[ link to this | view in thread ]
David (profile), 18 May 2012 @ 7:14am

Re: Re:
/lameness

I doubt it :)
[ link to this | view in thread ]
Jeremy, 18 May 2012 @ 7:31am

Re:
Nah, she says she made the copies because she needed to "work" from home. You don't work on rendered footage, it's basically done. Also, the other guy doesn't say that the movie was disappearing from the drive, he says characters or parts of characters were vanishing.

I'm guessing she's a texture artist who was making backups of the character/scene models and she was copying them home so she could see final result with her artwork included.

Though, when I think about more, does Toy story even have complicated textures? I have a strong memory of mostly solid colors.
[ link to this | view in thread ]
The eejit (profile), 18 May 2012 @ 7:43am

Re: Backups
Indeed. I'm reminded of the guy in my Intro to OS class accidentally removing everything from the Lecturer's computer.
[ link to this | view in thread ]
New Mexico Mark, 18 May 2012 @ 8:12am

Re:
You are a sad, strange little man, and you have my pity.
[ link to this | view in thread ]
Overcast (profile), 18 May 2012 @ 8:18am

Human complacency and dependance on computers is scarey sometimes...
[ link to this | view in thread ]
aldestrawk (profile), 18 May 2012 @ 8:34am

a retelling by Oren Jacob
Oren Jacob, the Pixar director featured in the animation, has made a comment on the Quora post that explains things in much more detail. The narration and animation was telling a story, as in storytelling. Despite the 99% true caption at the end, a lot of details were left out which misrepresented what had happened. Still, it was a fun tale for anyone who had dealt with backup problems. Oren Jacob's retelling in the comment makes it much more realistic and believable.
The terabytes level of data came from whoever posted the video on Quora. The video itself never mentions the actual amount of data lost or the total amount the raw files represent. Oren says, vaguely, that it was much less than a terabyte. There were backups! The last one was from two days previous to the delete event. The backup was flawed in that it produced files that when tested, by rendering,
exhibited errors. They ended up patching a two-month old backup together with the home computer version (two weeks old). This was labor intensive as some 30k files had to be individually checked.
The moral of the story. Firstly, always test a restore at some point when implementing a backup system. Secondly, don't panic! Panic can lead to further problems. They could well have introduced corruption in files by abruptly unplugging the computer. Thirdly, don't panic! Despite, somehow, deleting a large set of files these can be recovered apart from a backup system. Deleting files, under Linux as well as just about any OS, only involves deleting the directory entries. There is software which can recover those files as long as further use of the computer system doesn't end up overwriting what is now free space.
[ link to this | view in thread ]
Anonymous Coward, 18 May 2012 @ 8:56am

Re:
Your such a Ham
[ link to this | view in thread ]
polypusher, 18 May 2012 @ 9:40am

Re: Re:
Toy Story 2 had very complicated shaders for it's time. They don't just use solid-color textures. Even shaders that appear solid in texture to you have very subtle variations that insinuate reality. This is part of what made and makes Pixar so special.
[ link to this | view in thread ]
I-Blz, 18 May 2012 @ 9:46am

Re: Re:
Not so much complicated as detailed. In Toy Story 2, Woody's arm has athread pattern, and the texture works with this. The stuffing in his arm is also very detailed. Just a few examples
[ link to this | view in thread ]
Mason Wheeler, 18 May 2012 @ 10:01am

Re: a retelling by Oren Jacob
Panic can lead to further problems. They could well have introduced corruption in files by abruptly unplugging the computer.

What's worse? Corrupting some files or deleting all files?
[ link to this | view in thread ]
Anonymous Coward, 18 May 2012 @ 10:27am

who the hell punched in RM* to begin with?
[ link to this | view in thread ]
aldestrawk (profile), 18 May 2012 @ 10:38am

Re: Re: a retelling by Oren Jacob
In this case they were not dealing with unknown malware that was steadily erasing the system as they watched. There was, apparently, a delete event at a single point in time that had repercussions that made things disappear while people worked on the movie. I'll bet things disappeared when whatever editing was being done required a file to be refreshed. A refresh operation would make the related object disappear when the underlying file was no longer available. Apart from the set of files that had already been deleted, more files could have been corrupted when the computer was unplugged. Having said that, this occurred in 1999 when they were probably using the Ext2 filesystem under Linux. These days most everyone uses a filesystem that includes journaling which protects against corruption that may occur when a computer loses power. Ext3 is a journaling filesystem and was introduced in 2001.

In 1998 I had to rebuild my entire home computer system. A power glitch introduced corruption in a Windows 95 system file and use of a Norton recovery tool rendered the entire disk into a handful of unusable files. It took me ten hours to rebuild the OS and re-install all the added hardware, software, and copy personal files from backup floppies. The next day I went out and bought a UPS. Nowadays, sometimes the UPS for one of my computers will fail during one of the three dozen power outages a year I get here. I no longer have problems with that because of journaling.
[ link to this | view in thread ]
Danny (profile), 18 May 2012 @ 10:49am

I've gotta story like this too
Ive posted in athe past on Techdirt that I used to work for Ticketmaster. The is an interesting TM story that I don't think ever made it into the public, so I will do it now.

Back in the 1980s each TM city was on an independent computer system (PDP unibus systems with RM05 or CDC9766 disk drives. The drives were fixed removable boxes about the size of a washing machine, the removable disk platters about the size of the proverbial breadbox. Each platter held 256mb formatted.

Each city had itts own operations policies, but generally, the systems ran with mirrored drives, the database was backed up every night, archival copies were made monthly. In Chicago, where I worked, we did not have offsite backup in the 1980s. The Bay Area had the most interesting system for offsite backup.

The Bay Area BASS operation, bought by TM in the mid 1980s, had a deal with a taxi driver. They would make their nightly backup copies in house, and make an extra copy on a spare disk platter. Tis cabbie would come by the office about 2am each morning, and they'd put the spare disk platter in his trunk, swapping it for the previous day's copy that had been his truck for 24 hours. So, for the cost of about two platters ($700 at the time) and whatever cash they'd pay the cabbie, they had a mobile offsite copy of their database circulating the Bay Area at all times.

When the World Series earthquake hit in October 1988, the TM office in downtown Oakland was badly damaged. The only copy of the database that survived was the copy in the taxi cab.

That incident led TM corporate to establish much more sophisticated and redundant data redundancy policies.
[ link to this | view in thread ]
aldestrawk (profile), 18 May 2012 @ 11:30am

Re: I've gotta story like this too
I like that story. Not that it matters anymore, but taxi cab storage was probably a bad idea. The disks were undoubtedly the "Winchester" type and when powered down the head would be parked on a "landing strip". Still, subjecting these drives to jolts from a taxi riding over bumps in the road could damage the head or cause it to be misaligned. You would have known though it that actually turned out to be a problem. Also, I wouldn't trust a taxi driver with the company database. Although, that is probably due to an unreasonable bias towards cab drivers. I won't mention the numerous arguments with them (not in the U.S.) over fares and the one physical fight with a driver who nearly ran me down while I was walking.
[ link to this | view in thread ]
silverscarcat (profile), 18 May 2012 @ 11:55am

Re:
Obviously Disney did.
[ link to this | view in thread ]
The eejit (profile), 18 May 2012 @ 11:57am

Re: Re:
Well, it was pretty well-known that Disney wanted the partnership to fail at the time.
[ link to this | view in thread ]
Anonymous Coward, 18 May 2012 @ 12:19pm

So, making copies of an entire copyrighted film isn't Piracy?
[ link to this | view in thread ]
Anonymous Coward, 18 May 2012 @ 12:29pm

Re: I've gotta story like this too
Very interesting story, Danny...

& I Don't mean to nit-pick but the quake was Oct. '89. After reading the sloppy coding entries, I've been paying attention 2 details way more than I'd like to. I've been seeing 0's & 1's in my sleep. Damn nerds
[ link to this | view in thread ]
Anonymous Coward, 18 May 2012 @ 4:12pm

Re: Re: There's a trope for that!
Tere's a trope for that too
[ link to this | view in thread ]
Anonymous Coward, 18 May 2012 @ 4:12pm

Re: Re:
And a zerg!
[ link to this | view in thread ]
Anonymous Coward, 18 May 2012 @ 4:18pm

This is why limiting copying of a work isn't "protecting" it.

It may protect to company's monetary interest in it, but once the product is out there it hinders the growth of the work and in some cases cause it to be lost forever if it isn't "stolen" by those damn dirty pirates
[ link to this | view in thread ]
That One Guy (profile), 18 May 2012 @ 5:00pm

What I want to know is this...
She copied bits of a movie to her home system... how hard did they have to pull in the leashes to keep Disney's lawyers from suing her to infinity and beyond after she admitted she'd done so(never mind the fact that he doing so saved them apparently years of work...)?
[ link to this | view in thread ]
Anonymous Coward, 18 May 2012 @ 11:05pm

Mike, 'deleting' a file does not delete it from HDD unless it's overwritten.
[ link to this | view in thread ]
Anonymous Coward, 18 May 2012 @ 11:06pm

BTW once again "Linux" is mentioned.
[ link to this | view in thread ]
Huw Davies, 19 May 2012 @ 1:20am

Re: Re: I've gotta story like this too
RM05s are removable pack drives. The heads stay in the washing machine size unit - all you remove are the platters.
[ link to this | view in thread ]
Leget�j, 13 Nov 2012 @ 9:06pm

Fake
I really don belive this, cant be :)
[ link to this | view in thread ]
HisNameWasDoug, 28 Mar 2014 @ 6:19pm

Not nearly as exciting but i saved my boss's job one year when he accidentally wiped our database. I had been making copies to save my work in case I quit. I hated this boss more than anything but I couldn't find it in me to allow him to be fired, so I lied and said I had been making regular backups.
[ link to this | view in thread ]
Lance, 3 May 2014 @ 8:53am

http://thenextweb.com/media/2012/05/21/how-pixars-toy-story-2-was-deleted-twice-once-by-technology-a nd-again-for-its-own-good/

Evidently, the film data only took up 10 GB in those days. Nowhere near TB...
[ link to this | view in thread ]
Lance, 3 May 2014 @ 8:53am

http://thenextweb.com/media/2012/05/21/how-pixars-toy-story-2-was-deleted-twice-once-by-technology-a nd-again-for-its-own-good/

Evidently, the film data only took up 10 GB in those days. Nowhere near TB...
[ link to this | view in thread ]
DELETE TOY STORY 3 MOVIE, 25 Dec 2016 @ 10:55pm

TOY STORY 1 YES AND TOY STORY 2 YES NOW MOVIES
[ link to this | view in thread ]
DELETE TOY STORY 3 MOVIE, 25 Dec 2016 @ 11:03pm

TOY STORY 1 YES AND TOY STORY 2 YES NOW MOVIES
DELETE TOY STORY 3 MOVIE
YES
NOW
NO TOY STORY 3 MOVIE DELETE NOW
YES
AND ANDY BOY SMALL YES
ANDY NO MAN BIG DELETE
NOW YES
GO NOW
TOY STORY 1 YES AND TOY STORY 2 YES
NOW
ANDY BOY SMALL YES TOYS
TOYS ANDY TOY STORY 1 YES
TOYS ANDY TOY STORY 2 YES
NOW YES
GO NOW
DELETE TOY STORY 3 ANDY MAN
ANDY NO MAN BIG DELETE
ANDY BOY SMALL YES TOYS
GO NOW
DELETE TOY STORY 3 MOVIE
TOY STORY 1 MOVIE YES
TOY STORY 2 MOVIE YES
NOW
NETWORK DELETE TOY STORY 3 MOVIE
TV CHANNEL DELETE TOY STORY 3 MOVIE
ANDY BOY SMALL YES TOYS
ANDY LOVE MUMMY WOMAN
ANDY BOY YES
TOY STORY 1 MOVIE YES
TOY STORY 2 MOVIE YES
GO NOW
YES
OK
[ link to this | view in thread ]
Bridget (profile), 17 Oct 2017 @ 6:56pm

They're no better at looking after stuff nowadays
Whether it's accidental deletion or hacking or an inside job - or simply losing millions of dollars of bitcoins! I guess the thing is no one thinks it will happen to them. Till it does.
[ link to this | view in thread ]