How to Preserve Digital TV
“By nature and necessity, public broadcasting is a hodgepodge of media types and formats. A documentary might include moving and still images, speeches and voice-overs, sound effects, or a song. Children’s programming might include a combination of live action, cartoons, musical numbers, and kaleidoscopic effects. Source material for any of these production elements might be analog (a strip of film, a track from a 78-rpm phonograph record) or digital (panoramic portraits, credit rolls, logos). In whatever manifestations these objects previously existed, they become bits and bytes before they reach the public eye. That is an enormous amount of digital information to manage over time. As we move into the increasingly complex digital world, those charged with preserving our television heritage have the opportunity to develop and establish better coordinated and standardized preservation policies and practices to ensure what television programs and related assets survive.”
Mary Ide, Dave MacCarn, Thom Shepard, and Leah Weisse
”Understanding the Preservation Challenge of Digital Television”
Building a National Strategy for Preservation: Issues in Digital Media Archiving
Council on Library and Information Resources and the Library of Congress, April 2002
In a relatively rapid period of time, digital technology has radically transformed the nature of television program production — from a linear, sequential analog process, to a non-linear, random access, totally digital environment. Consequently, standard preservation practices that have served (and continue to serve) to protect analog productions on videotape no longer apply.
The public television institutions participating in this project are responsible for producing more than 60% of the program content aired on public television, and they are currently working together on several projects related to shifting from analog to digital program distribution methods.
But they have not yet come to terms with the technical standards, content selection criteria, and operating structures necessary to design and operate a repository for program preservation. This project, grounded directly on work that has already been done, will help public television take the next steps necessary to save our digital productions well into the future.
Public Television Needs a Digital Preservation Strategy
This initiative could not have come at a better time for public television. Just as the Library of Congress has been strongly committed to preserving America’s public television programs, up to this point, the efforts have naturally been focused on preserving tens of thousands of analog videotapes where these programs reside.
Even as the need to preserve these tapes remains critical, they are no longer our only preservation challenge. The changes in television production in the last few years have been profound, and we are rapidly approaching the “tapeless environment” – where programs will live solely as “disembodied” assets, attached to their metadata, distributed and stored in a totally digital environment. This introduces an entirely new set of issues and problems relating to long-term program preservation, for which no coordinated strategy yet exists in public television.
Thirteen, PBS and WGBH, the public television partners on this project, bring an unparalleled capacity, expertise, technical resources, facilities and personnel to the arena of digital video production and preservation. Working with NYU, this project will take the first important steps to establish standards, procedures and structures to preserve major public television assets – both complete programs and program elements – which are being created, produced, distributed and preserved completely in digital forms, and which will have long term historical and cultural value to the public. The system is just now grappling with adopting technical standards and procedures for such operational functions as digital distribution of programs for broadcast and is already looking at some of these questions. It is a logical extension of this ongoing process to plan a preservation strategy at the same time.
Program Production and Distribution Have Been Transformed
In a rapid period of time, digital technology has radically transformed the nature of television program production from a linear, sequential process to a non-linear, random access, totally digital environment. The impact is that many operational procedures in place for years no longer work. The standard preservation procedures that have served (and continue to serve) to protect analog productions on videotape cannot be used for the long-term preservation of digitally-produced broadcast programs.
Developing a new preservation approach is crucial –
- Most programs are now being shot, assembled and edited in digital form
- Programs themselves are created as digital files
- Broadcast playout is moving rapidly to a tapeless environment, where programs are stored only as files in a variety of formats which are then assembled and aired directly from a server
- With broadband networks finally able to handle large video files cost effectively and with integrity, PBS is on the verge of retiring its requirement of sending multiple versions of tapes for broadcast distribution and replacing it with an operational system that transfers digital files instead.
Not only has the process been altered, but the marketplace itself has changed. Previously, the standard for preserving programs was primarily based on saving the “national broadcast version” of a completed show. But such a definition today is open to broad interpretation. Soon, PBS is going to require producers to deliver discrete program elements rather than packaged programs, and the elements for a given program will be produced and assigned a unique asset identification. These elements may include opening credits, underwriter announcements, program segments, closing credits and similar components. In PBS’ future vision, an automated playback system would assemble the elements specifically for a particular broadcast. Future broadcasts that require changes will simply reassemble the elements as needed, so that whole segments could be customized and replaced with something else. When finished, all the various files return to their ‘disembodied’ state, with only an automation log indicating how the program actually aired.
Thus, when this plan is implemented, there will no longer be a definitive “broadcast version” of a given program, making the formerly accepted concept of a “broadcast version” of a program no longer valid – a completely new definition is needed. Such new definitions are needed network-wide, with broad implications for all the producers and distributors across the whole public television system.
The paradigm shift from preservation of a “definitive version” to preservation of assembled elements necessitates viewing public television programs at a finer level of granularity. When coupled with a digital asset management system, this offers enormous opportunities for solving long-standing problems in access and reuse, as well as in preservation.
The full life-cycle management of these assets will allow the stations to capture and save metadata regarding source and rights restrictions for every clip, still image, or sound recording, at the point they enter the station. (For example, under current practice, very little source and rights information is maintained on materials that don’t go into the final production, making it difficult to use those elements later.)
At the same time, companion websites for national and local programming are now required extensions for every featured public television program, containing far more content and audience engagement than any single broadcast can possibly squeeze onto the air. All of this is generating huge amounts of digital materials.
Once the broadcast rights to a program have expired, the public television system is generally uninterested in its disposition. By default, implementing policies for long-term archival preservation reverts back to those institutions with an investment in the longevity of the work. In this case, the interests rest with PBS, as the agent for all public television producers under its existing agreement with the Library of Congress, and with Thirteen/WNET and WGBH, which between them, produce a large proportion of the regular national program series seen on public television.
Within public television, WGBH has also been at the very forefront in developing a system for digital asset management. Since the advent of digital broadcasting (DTV) mandated by the FCC, Thirteen and WGBH have been working closely together on a number of projects to share aspects of local program operations and distribution. Both stations have also been working with PBS for more than two years to plan trials for transferring large digital files over broadband networks, to streamline national program distribution.
These tests are just getting off the ground, and they demand that these institutions come to agreement with each other on common standards for file formats and related specifications that will eventually be used system-wide. But common agreements on definitions, file formats, protocols and metadata have not yet been reached, and many standards still have to be worked out.
Apart from common distribution interests, Thirteen and WGBH are also program producers with different expectations regarding program preservation from PBS, which focuses its efforts on distribution. Working together to tackle the similar questions of long-term program preservation is a logical extension of the other problems related to digital production and distribution that we are struggling to resolve. It is imperative that we come to terms now to establish common definitions, standards and procedures, in tandem with and building on the efforts already underway.
The Library of Congress has a Primary Interest In Preserving Public Television
The Library of Congress has a legal agreement with the Public Broadcasting Service that stipulates that PBS shall act as an agent for all public television producers by allowing the LOC to have a copy of any program it wants, one year after the broadcast rights have expired. However, neither the LOC nor PBS has had enough funding to keep the program donations flowing as regularly as imagined when the agreement was signed in the early 90’s. Consequently, the costly transfer of analog tapes has been slow. PBS simply does not have the resources to create new plans for transferring digital materials as well.
In its strategic planning for NDIIPP, the Library of Congress recognized a special interest in preserving digital television productions, and it has supported considerable planning to address the particular problems that are now clearly emerging as predicted.
Public television station WGBH, with its well-established Archives, has been an active participant and prominent leader in this planning. A recent newcomer to the field of video preservation, Thirteen has turned to WGBH for guidance and direction to create its own archives, and is now in a position to work alongside WGBH in this effort.
Groundwork Is In Place, But Standards Need to Be Adopted
This project is based on the considerable planning and technical work which has been done in this field to date and the genuine need for these organizations to engage in a collective process that will lead to adopting standards for digital preservation and for designing models for video repositories.
NYU partners, with their own critical experience supporting processes to set standards and practices for preserving digital content, will lend particular authority to discussions on selection criteria, on defining the metadata critical for preservation, on procedures and best practices for operationalizing the workflow and preservation process, and in demonstrating how a video preservation repository can be run. Once standards are drafted, the project will test them using practical applications and models for how a preservation archive for America’s digital public television programming might actually operate.
Public television digital activity is relatively new, but solid groundwork is already in place for outlining functional procedures to manage the entire life-cycle of public television’s digital assets.
- Major efforts have been made through the Association of Moving Image Archivists (AMIA) and other organizations to develop cataloging standards and guidelines for preservation purposes, (such as AMIM2 Archival Moving Image Materials cataloging manual). Public television can use this work to adopt standard practices of its own.
- File formats, asset management systems and storage environments are being used, but they rely on a variety of protocols and applications that use different architectures and approaches to solving complex problems tied together. The project will sort out the options on standards for compression, file formats, and other key characteristics that have to be put in place.
- Within public television, the institutions partnering in this proposal have worked together for years to solve technical problems in program production and distribution as they evolve. But until now, discussions about long-term program preservation have been informal. Each organization is ready to participate in a formal planning process that will lead to an appropriate organizational design for creating and operating a digital repository, as well as for procedures and structures for submitting material to that repository.
A significant amount of work has already been done by these institutions. WGBH in particular has taken the lead in helping to design a reference architecture for digital asset management that is appropriate for public television, and this model is also being adopted by Thirteen.
WGBH has also developed the concept and plans for creating a Universal Preservation Format, a format-neutral container for digital assets as a long-term preservation strategy. [Development of the UPF was funded by National Historical Publications and Records Commission, Grant #97-029 ] The concept of the UPF is a data file format which utilizes a container or wrapper structure to avoid dependence on proprietary hardware and/or software. It was one of the first initiatives to explore the use of the wrapper/container approach as a preservation format. The idea has been well received as it has been presented over the last few years to librarians and archivists. The improvements in storage devices and other technology improvements, coupled with the system transition to a tapeless environment, point to this being a timely moment for a practical demonstration of this concept.
Organizational And Technical Capacity Of Partners
Educational Broadcasting Corporation, Thirteen/WNET, New York
EBC, one of America’s premier public television institutions. will be the lead institution in this partnership and will have primary responsibilities for managing and administering the grant. Its flagship station Thirteen/WNET reaches more than 4.5 million viewers in New York, New Jersey and Connecticut. Thirteen has an operating budget of more than $150 million dollars and a staff of 450, making it the second largest station in the public television system.
EBC inaugurated broadcast service in 1962. In the forty years since, the station has earned a strong reputation for producing thousands of hours of award-winning cultural affairs and community programming for both local and national broadcast. Today, Thirteen produces many of the signature arts and cultural affairs programs aired over public television, including such important series as American Masters, Great Performances, Dance in America, EGG The Arts Show, NATURE, and Cyberchase.
In 1999, the station moved into new studios from a building it had occupied for 25 years. Working closely with WGBH, up to then the only public television station with an archive, EBC used the move as an opportunity to transform a chaotic collection of 60,000 videotapes into a genuine videotape archive of its own. Now a function of Thirteen’s Reference Library, the Tape Archive houses Thirteen’s rich but fragile video legacy of nearly 30,000 tapes and is the official department designated to collect, classify, protect, and preserve the station’s program and production assets.
In 2003, EBC acquired public television station WLIW Channel 21, Long Island’s premier public television station and now operates both stations from a joint master control studio in mid-town Manhattan. With this acquisition, EBC also became responsible for the unique programs produced by WLIW, including the popular travel series “Visions of …..” . Many of these programs, as well as programs within Thirteen’s own production departments such as NATURE and American Masters, are produced expressly for public television in High Definition (HD) format. It is especially important to include HD as a format considered at-risk, because its future is uncertain as a robust format that will be widely adapted by commercial producers.
WGBH: Media Archives & Preservation Center
WGBH in Boston is the largest station in the public television system. It traces it origins back more than a century and a half, when a bequest from John Lowell, Jr. in 1836 created the Lowell Institute to provide free public lectures for Boston’s citizens. In the mid-1940’s, the Lowell Institute spearheaded a cooperative venture to broadcast the Institute’s lectures over the radio. By 1951 WGBH/FM Radio was launched, followed by WGBH/TV in 1955.
Today WGBH holds the licenses for six broadcast operations; three radio and two television stations and is the largest station within the public broadcasting system. It provides one-third of all prime-time programs seen on PBS nationwide and is a major supplier of programming to the public radio system as well. Major national television series produced by WGBH include: Antiques Roadshow, The American Experience, Arthur, Between the Lions, Frontline, Masterpiece Theatre, Mystery! The New Yankee Workshop, NOVA, Victory Garden and Zoom. WGBH also provides a diverse menu of educational services for the public including on-line and print classroom curriculum materials, support for interactive multimedia series and companion web sites to its major series.
WGBH Media Archives & Preservation Center was established in 1979 to serve in-house productions by maintaining the master and element programming materials from radio and television productions. Today, the Archive manages a collection of nearly 500,000 editorial assets, administrative documents, computer discs, tapes and graphic materials. During 2002 the Archives circulated more than 26,000 items and acquired more than 18,000 new items from productions. Television programming in the Archives dates back to the 1960’s including such series as: Prospects of Mankind, a political discussion series hosted by Eleanor Roosevelt and Course of Our Times, a lecture series featuring faculty from Harvard, MIT and other Boston universities. The collection also contains Julia Child’s first cooking series, The French Chef.
WGBH currently maintains three major archival collection guides, embedded with streaming video clips, available via the Web. The first is New Television Workshop, a video art series produced by WGBH between 1967 and 1994. The second is Say Brother, an African American public affairs program produced by WGBH between 1968 and the mid 1990s. The newest collection is “Ten O’Clock News”, a local nightly news program that aired between 1974 and 1991. [All three sites can be reached at http://main.wgbh.org/resources/archives.
In addition to its Archive, WGBH has had a long involvement developing a comprehensive digital asset management system, designed to capture information from the very beginning of the production cycle. They have been working in close partnership with Sun Microsystems, Artesia Software and Sony to bring such a system to its current stage of implementation, and the new system called Reference Architecture for Digital Asset Management is now being rolled out. [Complete documentation outlining this and related presentations is available on-line from their DAM seminar in the fall of 2002 at http://daminfo.wgbh.org/maccarn2.pdf.
PBS — The Public Broadcasting Service
PBS, headquartered in Alexandria, Virginia, is a private, non-profit media enterprise owned and operated by the nation’s public television stations. Founded in 1969, PBS provides quality TV programming and related services to 349 noncommercial public television stations serving all 50 states, Puerto Rico, the U.S. Virgin Islands, Guam and American Samoa. In its mandate, PBS oversees program acquisition, distribution and promotion; education services; new media ventures; fundraising support; engineering and technology development; and video marketing.
As the primary repository of the nation’s public television programs, the Public Broadcasting Service maintains a tape and film collection dating back to the mid-1950s, including National Educational Television (NET) as well as one or more broadcast versions of nearly every program distributed by PBS. The current collection includes over 115,000 videotapes, the bulk of which are 1-inch or 2-inch analog or in a variety of digital formats. To date, PBS has transferred approximately 22,000 2-inch videotapes or 16mm film reels—representing the entire NET collection and the bulk of the PBS 2” format era–to the Library of Congress as part of a 1993 donation agreement.
The archival agreement specifies that PBS will donate the “best copy” of every public television program to the Library after its broadcast rights have expired. Given the wide variety of program versions collected and maintained by PBS, each potential donation candidate must be evaluated in relation to its siblings, requiring a complete re-inventory of all affected programs and a sometimes tortuous decision concerning which version constitutes the best copy.
The introduction of digital distribution, in which there are no physical master tapes or “fixed” program versions, represents both an opportunity and a challenge for both PBS and the Library of Congress. The retention and assembly of programs as a series of discrete elements minimizes the need for redundant long-term storage but adds a new requirement to develop or adopt standard definitions for the archival preservation of program elements and metadata. Consequently, the need to adopt standards for long-term retention of digital program elements and metadata has is a critical priority.
New York University
Founded in 1831, New York University is one of the largest private universities in the United States. From an initial student body of 158 and a faculty of 14, today NYU has more than 3,100 full time faculty members, more than 48,000 students, programs in 14 schools and colleges, six major centers in Manhattan, and study abroad centers in over 20 countries.
NYU has a leading academic research library which supports a strong and innovative Digital Library program, a joint initiative between the NYU Divisions of Libraries and NYU’s Information Technologies Services. NYU is an active member of the Digital Library Federation, and NYU’s Digital Library is engaged in the development and partnerships necessary to ensure that scholarly digital content will be available and readily useful for teaching, learning and research.
NYU is also home to the renowed Tisch School of the Arts, a center for the study of film and television. Tisch has recently established its ground-breaking program in Moving Image Archiving and Preservation, a new Masters Degree program which is one of only two such programs in the United States. Launched in Fall 2003, this two-year course of study will train future professionals to manage preservation-level collections of film, video, new media, and other types of moving image works. The NYU Libraries has partnered with the new Tisch program to create a model moving image preservation program for the Libraries collections. Funded by a grant from The Andrew W. Mellon Foundation, the joint initiative will develop models and best practices for such collections in academic libraries.
NYU Libraries was well positioned for the partnership with Tisch because it is home to an innovative and distinguished preservation program, to one of the largest audio visual media collections and services in an academic library, and to archival collections that include significant moving image materials. These strengths also have lead to a focus of its digital library program to address the technical, cultural and legal issues in supporting audio-visual and complex hypermedia works in digital library systems.
Through these intersecting programs, NYU is positioning itself as one of the major academic players in the field of digital preservation of moving images. High level staff and faculty have been participating in national and international standard-setting committees, including those convened by the LOC, and the school has successfully acquired funds from both private and public sources to support preservation projects. NYU is enthusiastic about lending their considerable scholarly expertise to facilitate standard-setting for public television’s digital archives. http://library.nyu.edu/diglib/
Staffing and Key Personnel
Kenneth Devine – Chief Technology Officer & VP, Educational Broadcasting Corp
Mr. Devine is the overall supervisor of the project. He is involved in a number of parallel activities involving planning and collaborations with both WGBH and PBS, and he has key responsibilities for facilitating coordination between technical personnel and between this project and others that are underway.
David MacCarn — Chief Technologist & Asset Management Architect, WGBH
Mr. MacCarn is currently responsible for long-term planning, investment and adoption of new technologies at WGBH, where he co-authored “Universal Preservation Format” He is also one of the principle design architects of the digital asset management application and will have primary oversight for completing the UPF design and related issues.
Dr. Howard Besser – Professor and Director, NYU Moving Image Archive & Preservation Program
Dr. Besser is a Professor at New York University, where he is Director of a new Master’s Degree program in Moving Image Archiving and Preservation. He has published several articles on digital preservation, and has worked internationally on issues relating to multimedia, image databases, digital libraries, metadata standards, digital longevity, web design, information literacy, distance learning, intellectual property, and the social and cultural impact of new information technologies. He is overseeing research on standards and other activities related to comparative practices in digital video preservation.
Ms. Rubin has more than twenty years of experience managing technical and communications projects for pubic radio and television, including ten years at Thirteen. She planned and coordinated the creation of the Thirteen Tape Archives and has been instrumental in assisting Thirteen in developing an initiative for the long-term preservation of its program assets. She is responsible for managing all aspects of project work, including overseeing progress, maintaining relationships with partner organizations, and keeping the project accountable to all parties. She works under the direct supervision of Mr. Devine, who is providing the primary fiscal and technical oversight for the work.
As Director of the WGBH Media Archives, Ms. Ide has a long history of providing leadership in designing and maintaining television archives. She has primary responsibility for overseeing the process of preparing inventories of at-risk holdings and activities related to appraisal practices.
Mr. Kutzner is directly engaged with planning and implementing the next generation interconnection project at PBS, which will be based on adopting standard protocols for file transfers between PBS, PBS producers, and PBS member stations. He is ensuring that standards and procedures are compatible with the needs of the PBS system.
Ms. Morse oversees the PBS Broadcast Operations, Technical Services and Media Library departments. Her responsibilities include updating systems for managing and archiving the PBS media library’s wide range of content, and she is one of the people at PBS most familiar with their efforts in videotape preservation and relations with the Library of Congress.
Mr. Liroff is a leader in developing long-term technical innovations in public television. He has been central in supporting WGBH’s preservation activities and in taking a strong role advocating for and designing a digital asset management plan for public television stations.
He is a member of the Corporation for Public Broadcasting Task Force on Digital Funding, the public broadcasting Digital Advisory Group, the CPB TV Future Fund Advisory Panel, the PBS New Technology Committee, the PBS Interconnection Committee, the Association of Moving Image Archivists, MIT Council for the Arts, and the MIT Museum Advisory Board.