How Toy Story 2 Almost Got Deleted... Except That One Person Made A Home Backup
from the did-it-break-the-rules? dept
Here's a random story, found via Kottke, highlighting how Pixar came very close to losing a very large portion of Toy Story 2, because someone did an rm * (non geek: "remove all" command). And that's when they realized that their backups hadn't been working for a month. Then, the technical director of the film noted that, because she wanted to see her family and kids, she had been making copies of the entire film and transferring it to her home computer. After a careful trip from the Pixar offices to her home and back, they discovered that, indeed, most of the film was saved:Now, mostly, this is just an amusing little anecdote, but two things struck me:
- How in the world do they not have more "official" backups of something as major as Toy Story 2. In the clip they admit that it was potentially 20 to 30 man-years of work that may have been lost. It makes no sense to me that this would include a single backup system.
- I wonder if the copy, made by technical director Galyn Susman, was outside of corporate policy. You would have to imagine that at a place like Pixar, there were significant concerns about things "getting out," and so the policy likely wouldn't have looked all that kindly on copies being used on home computers.
Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.
Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.
While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.
–The Techdirt Team
Filed Under: backups, mythbusters, toy story
Companies: pixar
Reader Comments
Subscribe: RSS
View by: Time | Thread
[ link to this | view in chronology ]
Re: Human Trumps Radioactive Dinosaur
[ link to this | view in chronology ]
You could say it's a different thing all together, but it really isn't.
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
There's a trope for that!
[ link to this | view in chronology ]
Re: There's a trope for that!
[ link to this | view in chronology ]
Re: Re: There's a trope for that!
[ link to this | view in chronology ]
[ link to this | view in chronology ]
[ link to this | view in chronology ]
Re:
/lameness
[ link to this | view in chronology ]
Re: Re:
I doubt it :)
[ link to this | view in chronology ]
Re: Re:
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
[ link to this | view in chronology ]
Reminds me of ....
Though this was not the problem I had been called on to investigate, I asked some probing questions, made a short phone call, and provided the answer, much to the customer's relief.
What I found was that for months if not years the customer had been performing backups of indexed sequential files, that is data files with associated index files, without once verifying that the backed-up data could be recovered. On the first occasion of a problem requiring such a recovery they discovered that they just did not work.
The answer? Simply recreate the index files from the data. For efficiency reasons (this was a LONG time ago) the index files referenced the data files by physical disk adresses. When the backup tapes were restored the data was of course no longer at the original place on the disk and the index files were useless. A simple procedure to recreate the index files solved the problem.
Clearly whoever had designed that system had never tested a recovery, nor read the documentation which clearly stated the issue and its simple solution.
So here is a case of making backups, but then finding them flawed when needed.
[ link to this | view in chronology ]
Re: Reminds me of ....
[ link to this | view in chronology ]
[ link to this | view in chronology ]
Two notes on backups
Okay, okay, one comment: in over 30 years of working in the field, the second-worst product I have ever had the misfortune to deal with is Legato (now EMC) NetWorker.
2. Hollywood has a massive backup and archiving problem. How do we know? Because they keep telling us about it. There are a series of self-promoting commercials that they run in theaters before movies, in which they talk about all of the old films that are slowly decaying in their canisters in vast warehouses, and how terrible this is, and how badly they need charitable contributions from the public to save these treasures of cinema before they erode into dust, etc.
Let's skip the irony of Hollywood begging for money while they're paying professional liar Chris Dodd millions and get to the technical point: the easiest and cheapest way to preserve all of these would be to back them up to the Internet. Yes, there's a one-time expense of cleaning up the analog versions and then digitizing them at high resolution, but once that's done, all the copies are free. There's no need for a data center or elaborate IT infrastructure: put 'em on BitTorrent and let the world do the work. Or give copies to the Internet Archive. Whatever -- the point is that once we get past the analog issues, the only reason that this is a problem is that they made it a problem by refusing to surrender control.
[ link to this | view in chronology ]
Re: Two notes on backups
[ link to this | view in chronology ]
Re: Re: Two notes on backups
Proud to be a backup
Let people print 'em for free and let them go wild on their media collection of choice (and I'll never lose the sticker image either). Go ahead, add some character to that aging collection of DVDs, they'll all mold over eventually!
[ link to this | view in chronology ]
[ link to this | view in chronology ]
Re:
I'm guessing she's a texture artist who was making backups of the character/scene models and she was copying them home so she could see final result with her artwork included.
Though, when I think about more, does Toy story even have complicated textures? I have a strong memory of mostly solid colors.
[ link to this | view in chronology ]
Re: Re:
[ link to this | view in chronology ]
Re: Re:
[ link to this | view in chronology ]
Backups
[ link to this | view in chronology ]
Re: Backups
[ link to this | view in chronology ]
[ link to this | view in chronology ]
a retelling by Oren Jacob
The terabytes level of data came from whoever posted the video on Quora. The video itself never mentions the actual amount of data lost or the total amount the raw files represent. Oren says, vaguely, that it was much less than a terabyte. There were backups! The last one was from two days previous to the delete event. The backup was flawed in that it produced files that when tested, by rendering,
exhibited errors. They ended up patching a two-month old backup together with the home computer version (two weeks old). This was labor intensive as some 30k files had to be individually checked.
The moral of the story. Firstly, always test a restore at some point when implementing a backup system. Secondly, don't panic! Panic can lead to further problems. They could well have introduced corruption in files by abruptly unplugging the computer. Thirdly, don't panic! Despite, somehow, deleting a large set of files these can be recovered apart from a backup system. Deleting files, under Linux as well as just about any OS, only involves deleting the directory entries. There is software which can recover those files as long as further use of the computer system doesn't end up overwriting what is now free space.
[ link to this | view in chronology ]
Re: a retelling by Oren Jacob
What's worse? Corrupting some files or deleting all files?
[ link to this | view in chronology ]
Re: Re: a retelling by Oren Jacob
In 1998 I had to rebuild my entire home computer system. A power glitch introduced corruption in a Windows 95 system file and use of a Norton recovery tool rendered the entire disk into a handful of unusable files. It took me ten hours to rebuild the OS and re-install all the added hardware, software, and copy personal files from backup floppies. The next day I went out and bought a UPS. Nowadays, sometimes the UPS for one of my computers will fail during one of the three dozen power outages a year I get here. I no longer have problems with that because of journaling.
[ link to this | view in chronology ]
[ link to this | view in chronology ]
Re:
[ link to this | view in chronology ]
Re: Re:
[ link to this | view in chronology ]
I've gotta story like this too
Back in the 1980s each TM city was on an independent computer system (PDP unibus systems with RM05 or CDC9766 disk drives. The drives were fixed removable boxes about the size of a washing machine, the removable disk platters about the size of the proverbial breadbox. Each platter held 256mb formatted.
Each city had itts own operations policies, but generally, the systems ran with mirrored drives, the database was backed up every night, archival copies were made monthly. In Chicago, where I worked, we did not have offsite backup in the 1980s. The Bay Area had the most interesting system for offsite backup.
The Bay Area BASS operation, bought by TM in the mid 1980s, had a deal with a taxi driver. They would make their nightly backup copies in house, and make an extra copy on a spare disk platter. Tis cabbie would come by the office about 2am each morning, and they'd put the spare disk platter in his trunk, swapping it for the previous day's copy that had been his truck for 24 hours. So, for the cost of about two platters ($700 at the time) and whatever cash they'd pay the cabbie, they had a mobile offsite copy of their database circulating the Bay Area at all times.
When the World Series earthquake hit in October 1988, the TM office in downtown Oakland was badly damaged. The only copy of the database that survived was the copy in the taxi cab.
That incident led TM corporate to establish much more sophisticated and redundant data redundancy policies.
[ link to this | view in chronology ]
Re: I've gotta story like this too
[ link to this | view in chronology ]
Re: Re: I've gotta story like this too
[ link to this | view in chronology ]
Re: I've gotta story like this too
& I Don't mean to nit-pick but the quake was Oct. '89. After reading the sloppy coding entries, I've been paying attention 2 details way more than I'd like to. I've been seeing 0's & 1's in my sleep. Damn nerds
[ link to this | view in chronology ]
[ link to this | view in chronology ]
It may protect to company's monetary interest in it, but once the product is out there it hinders the growth of the work and in some cases cause it to be lost forever if it isn't "stolen" by those damn dirty pirates
[ link to this | view in chronology ]
What I want to know is this...
[ link to this | view in chronology ]
[ link to this | view in chronology ]
[ link to this | view in chronology ]
Fake
[ link to this | view in chronology ]
[ link to this | view in chronology ]
Evidently, the film data only took up 10 GB in those days. Nowhere near TB...
[ link to this | view in chronology ]
Evidently, the film data only took up 10 GB in those days. Nowhere near TB...
[ link to this | view in chronology ]
TOY STORY 1 YES AND TOY STORY 2 YES NOW MOVIES
[ link to this | view in chronology ]
TOY STORY 1 YES AND TOY STORY 2 YES NOW MOVIES
YES
NOW
NO TOY STORY 3 MOVIE DELETE NOW
YES
AND ANDY BOY SMALL YES
ANDY NO MAN BIG DELETE
NOW YES
GO NOW
TOY STORY 1 YES AND TOY STORY 2 YES
NOW
ANDY BOY SMALL YES TOYS
TOYS ANDY TOY STORY 1 YES
TOYS ANDY TOY STORY 2 YES
NOW YES
GO NOW
DELETE TOY STORY 3 ANDY MAN
ANDY NO MAN BIG DELETE
ANDY BOY SMALL YES TOYS
GO NOW
DELETE TOY STORY 3 MOVIE
TOY STORY 1 MOVIE YES
TOY STORY 2 MOVIE YES
NOW
NETWORK DELETE TOY STORY 3 MOVIE
TV CHANNEL DELETE TOY STORY 3 MOVIE
ANDY BOY SMALL YES TOYS
ANDY LOVE MUMMY WOMAN
ANDY BOY YES
TOY STORY 1 MOVIE YES
TOY STORY 2 MOVIE YES
GO NOW
YES
OK
[ link to this | view in chronology ]
They're no better at looking after stuff nowadays
[ link to this | view in chronology ]