dB Broadcast Delivers Landmark Project for BBC Wales

New IP Infrastructure Required Extensive System Testing, and was Delivered Despite Coronavirus

Overview and Background dB Broadcast has completed a landmark project for BBC Wales – a state of the art broadcast centre at Central Square in Cardiff. Not only is it the first BBC facility to use IP technology across both its production and broadcast operations, but it is also one of the largest SMTPE 2110 IP routing systems for broadcast deployed anywhere.

The vision for the project from the start was to design and implement new technology to support production and creativity based on a Live IP approach. As a result, it delivers new agile ways of working through the open plan, collaborative style and flexible production spaces. The system is central to the broadcaster’s new TV playout and radio operations, supporting BBC1 Wales, BBC2 Wales, Welsh language channel S4C, Radio Wales and Radio Cymru.

BBC Wales is responsible for content production in Welsh and English across all platforms, and has the largest BBC newsroom outside London. In addition to BBC1, BBC2 and S4C TV channels and BBC Radio Wales, BBC Radio Cymru & Network radio production, the BBC Wales hub produces flagship content for BBC Studios and runs online services for BBC News, BBC Sport, BBC iPlayer and Cymru Fyw.

The broadcast centre is housed in a new 150,000 sq ft building - around half the size of the previous HQ in Llandaff, which suffered from dated technology and a poor building configuration. It has space for around 1,200 staff, and the technology deployed can adapt quickly to new video formats, higher bandwidth and better video resolution. It also offers simpler production for new platforms, with less time spent converting between different video or audio formats. The new BBC Wales HQ extends over five floors, with a rooftop garden and public spaces pre-wired as shooting locations. The new facilities provide the best possible tools with a greater level of creative freedom for content makers, while audience engagement is enhanced. dB Broadcast was appointed, through a competitive process, as system integrator in recognition of its significant prior experience in broadcast IP systems. There was collaboration with the BBC from a very early stage providing knowledge and expertise on project planning, system design and architecture. A Proof of Concept (PoC) analysis phase followed at dB Broadcast’s premises in Ely for interoperability testing for 2110 and AMWA/NMOS networks and equipment compliance.

Broadcast Infrastructure dB Broadcast worked closely with contract holder Grass Valley, who installed their IP orchestration and routing system, capable of carrying around 6Tbps of uncompressed live media traffic throughout the building. IP enabled Sirius routers were installed alongside a fully redundant Morpheus and ICE Panoplay system, with Morpheus deployed as a completely virtualised system. The 2110 IP network is based on a spine/leaf architecture utilising Cisco networking hardware with a Cisco DCNM layer for device authentication and management of flow permissions. The IP Core was designed with the highest resiliency in mind utilising A & B networks, enabling dual path resiliency across the network supporting the SMPTE 2022-7 protocol. The Software Defined Network (SDN) was provided by Grass Valley with GV Convergent which enabled interoperability with commercial off the shelf switches (COTS) and support of SMPTE/EBU/AMWA standards.

Audio Network integration with the IP Core was paramount with dedicated Audio Over IP (AoIP) islands for Radio (AES67) and Post-Production (Dante). Tight integration between BNCS, the principal control platform, and Convergent allows users and operators to utilise the IP core routing with familiar, easy to use interfaces.

TV Studios

dB Broadcast integrated two News Studios and a General Purpose Studio, all Live IP facilities, as well as a further versatile space equipped with a lighting rig so that it can be another shooting space. There are three studio galleries that can be assigned to any of the studio spaces.

The largest 3,500 sq ft studio includes a fully motorised lighting grid and is equipped with Sony IP cameras and a Vizrt AR graphics system. The second 1,600 sq ft news studio is fully automated and equipped for AR and a third studio is a full green-screen VR space. The News studios benefit from Robotic cameras and pedestals, augmented and virtual reality (AR/VR) graphics systems, whilst the larger News Studio boasts a 70m2 video wall, integrated and driven by gallery automation.

The building has 20 ‘inject points’ where cameras can be connected, and the IP network can route the signals from them directly to any of the galleries.

Media production

Elsewhere in the building, there are state of the art Post- Production facilities, including 27 Video Editing Suites, three Dubbing Theatres, two Tracklaying suites, three Voice Over Booths and a Colour Grading Suite. All take advantage of the Live IP Core and facility wide IHSE KVM system to access a shared pool of resources. The Audio Editing Suites were designed with Dolby Atmos capability, enabling the BBC to work with the latest and most impressive audio standard. Avid Interplay was chosen as the Post Production asset management tool, with fast access Nexus storage. While facility asset management was provided by Dalet, which meant dB Broadcast and the BBC engaging in workflow analysis and consultancy to optimise the file-based workflows between sub-systems such as EVS, Avid, VizRT and NRCS systems. dB Broadcast

worked closely with EVS and VizRT so their equipment natively supported 2110 directly onto the IP Core network.

TV Playout dB Broadcast delivered automated Playout and Continuity systems for BBC1W, BBC2W and S4C to viewers in Wales across linear TV (DTT, DSAT) and VOD Platforms, with a mixture of Network and Local programming, both live and pre-recorded. A flexible approach to de-risk this critical element of the project included a phased deployment, with dB Broadcast delivering live and POC systems in the original Llandaff facility.

Radio

dB Broadcast delivered five flexible IP radio studios clustered around three control rooms for main services, with one larger studio to accommodate live music guests. A further seven self-op radio studios used for a variety of programming including news were also delivered. A total of nine production spaces had visualisation and lighting installed, enabling the radio studios to deliver multimedia based services particularly to online platforms. Further to this, ten Audio Edit Suites based on SADie editing were installed, as well as a dedicated Drama Studio, Control Room and Edit Suite.

Benefits of Live IP

A key feature of the project has been the focus on a Live IP infrastructure throughout, requiring liaison with multiple equipment vendors and extensive system testing before handover.

The benefits of this approach are substantial. It is a highly flexible solution, supporting new ways of working with more dynamic assignment of resources, allowing facilities to be scaled up more easily.

While IP standards are increasingly being adopted, this was not the case at the start of the project, and dB Broadcast had to maintain as much flexibility as possible. The result is a system with a good deal of future proofing, with the ability to adopt new formats as they emerge.

Efficiency has been a key focus. The system supports exceptional utilisation of facilities due to the adoption of virtualisation, COTS hardware and the ability to move broadcast functions onto a more generic platform with a distributed routing core.

And the challenges…

Achieving these benefits did not come easily of course: moving site and introducing new technology at the same time was always going to be a challenge!

“The most significant issue was achieving interoperability along with reliable device control, because at the time IP standards were very new and evolving rapidly”, says Mike Bryan, Programme Manager at dB Broadcast. “We have found from the many large scale IP projects that dB Broadcast has delivered recently, that having access to our own test lab here at dB is hugely beneficial, not only for our own engineers’ access, but we are able to provide remote access to the BBC team and to third party manufacturers to a standalone version of the core system. This allowed extensive testing and promoted collaborative supplier working to prove multi-vendor interoperability in a controlled way.” Accordingly, a greater level of resource was required to concentrate efforts on this aspect. dB Broadcast used a formalised approach using IT techniques (TestRail) to manage regression testing, and to track progress between test runs.

Fault finding can become a huge issue on large ST 2110 systems, and the testing approach itself can unexpectedly break the system under test. Security is another major challenge with IP systems as a network can be disrupted e.g. by denial of service attacks. So a lot of time was spent designing in safeguards such as segregating the network from general office traffic.

With Live IP, equipment refresh cycles are much faster, and don’t necessarily align well with traditional capital planning models. New skills were required to deliver and support Live IP, both at dB Broadcast and at the BBC. This implied an increased attention to training.

Delivery 2020

On top of the inherent challenges of delivering the BBC’s first large IP facility with Post- Production and Radio integration came Covid-19! This of course meant a radical change to ways of working, with colleagues required to work either remotely - connected to the dB Broadcast server and administration systems by a secure VPN - or under more controlled conditions with adequate PPE etc. Everything possible was done to protect employees and the client while continuing the project, allowing dB Broadcast and its partners to successfully deliver the system which went on air under lockdown conditions.

Roger Crothers, Head of Technology at BBC Cymru Wales, has commented: “The worlds of broadcasting and media are changing at an unprecedented rate; our audience wants more content than ever before across an increasingly diverse range of devices. We need a facility that can remain up-to-date for many years to come, allowing us to adapt easily to new formats and services. IP is the only logical choice: it is format agnostic and meets our need for scalability and agility.”

A final word from Mike Bryan: “With any programme of this scale, especially taking into account this being based on an IP infrastructure, it had some significantly challenging moments, but ultimately it was extremely rewarding to see the finished installation come together, and BBC Wales migrate live services into Central Square. It has been a huge team effort, bringing together BBC Wales, wider BBC teams, suppliers and ourselves; everyone had a part to play and did so. The system gives BBC Wales an excellent technology platform to build on, and to produce high quality content for its audiences for many years to come.”