In May of this year, the Stanford Digital Repository accomplished another first: a complete media migration.
What do we mean by “media migration”? Media migration involves making copies of stored digital files onto new storage media. Because storage media – like CDs, DVDs, hard drives, and digital tape – have short useful lifetimes and are subject to failure, media migration is one of the more fundamental operations of a digital archive.
In the case of the SDR, this media migration was especially important, because it involved copying every file stored in the SDR– all 2,673,229 of them (representing some 30,640 objects) – from one data storage system onto a whole new data storage system.
It was time to decommission the storage equipment used in the repository -- a Sun L700 tape library (both LTO2 and LTO4 tapes) and Sun Honeycomb hard disk devices – to be replaced by the current configuration of Sun Thumpers and an IBM 3584 tape library (LTO4). With the new storage configuration, the SDR will be better positioned to scale its operations, capacity, and services as well as to sustain and manage the repository well into the future.
Kudos go to SDR team members Xinlei and Donald who worked hard on this project from start to finish. It is critical to carefully monitor and verify the migration process in order to maintain control over the contents of the repository as they are being shuttled from one place to another. One needs to be sure that all files are copied (“no file left behind”), and that they are copied completely and without error.
The project took about 18 months altogether. After several weeks of planning and development of computer scripts to assist in the process, the migration began in July 2009 and completed in February 2010. Then, a quality assurance audit was conducted, correlating all objects and files from all available sources (storage systems, logs, and administrative databases) before and after migration. The project was not without its challenges, but in the end, it was a clear success and we learned useful lessons along the way.
It’s exciting to have a fresh copy of the repository’s contents written on the new storage system, just as we are developing the newly-designed SDR 2.0! Good timing.