By Thomas R. Ray, III CPBE, AMD, DRB
Tom Ray Consulting
NEW YORK — We have been discussing, on and off over the past couple of weeks, IP and IP codecs in our “broadcast” environments. This was a discussion that quickly escalated when Verizon quietly started telling customers in the northeast that ISDN was soon to be history.
At the recent NAB Show in Las Vegas, nearly everything I saw was IP based in some form or another. So the truth of the matter is, IP is here to stay and it’s up to us to make it work.
First and foremost, there was a comment by someone who stated, and I’m paraphrasing here, “Would you trust your high dollar sports remote to an IP connection over the public internet?” Apparently, CBS television has answered that question.
How many reading this watched the Final Four coverage and the games leading up to it? Normally, a company – and sometimes it is the venue itself – is contracted to provide video production of the game. This is normally a company that is familiar with the venue and that has the equipment locally to properly cover the game. This feed is sometimes distributed to several outlets – perhaps a local television station, the in-house video production at the venue, to CBS television for satellite uplink to New York for inclusion on the network. CBS also normally provides a second crew that handles the announcers – normally a three camera shoot in the booth for between period, halftime and pre- and post-game coverage, as well as courtside coverage after the game.
As you can imagine, it is expensive to send a second crew with equipment, possibly renting equipment locally, and then having to uplink this separately to New York. Satellite time for television is not inexpensive.
This year, CBS sent robotic cameras and only a few crew members. The camera shots were switched in New York, not on location. This was not done by satellite. It was done via IP, including what you saw on television! CBS has stated that they have had little to no problems with the public internet. They find that they have problems with the local “drop.” In other words, they have trouble with the local connection and internet service provider (ISP) on either end. Once the data stream hits the public internet, it is transported from point A to point B without issue.
Last time I checked, the Final Four and the games leading up to such were a very high dollar remote for CBS. And they were relying on an internet connection for all of the shots of the announcers. That speaks volumes to me.
So, let’s look at some issues surrounding IP codec connectivity for radio remotes and a life without ISDN.
One big ticket item is interoperability – if you have a Brand X IP codec, you typically cannot “talk” to a Brand “Y” IP codec. The reason for this, simply, is that different manufacturers transport their data differently across the internet. You and I may think of an internet connection as a simple connection to transport 1’s and 0’s from one place to another – and that is what it is. But there is a lot more to it than that. There are different “flavors,” if you will, of packaging up and transporting those 1’s and 0’s over the internet, much the same way you can come up with numerous ways to pack up a dozen eggs to ship via the US Mail. Some methods are much more effective at having the payload, or item, reach its destination unscathed.
And that’s not to say that any one manufacturer is right or wrong in their chosen method of transport. They all have their reasons, and each method works – but, much like in the movie Ghost Busters where you don’t “cross the streams,” you can’t mix and match transport protocols. The outcome is trash at the other end that the device won’t recognize as valid data.
To counter this interoperability problem, a standard is presently being formulated that will allow interoperability between different boxes made by different manufacturers. This means that, if you have a Brand X codec, it will be able to talk to a Brand Y codec, much in the same way ISDN codecs of different brands now connect and produce audio. While there is no definitive timeline on this, know that it is in the works.
So from that standpoint, what is a broadcaster to do?
First, determine your needs. If you do a lot of out of house ISDN work – ie, if you connect with many different places for many different needs, start asking around to see if these places have IP codecs and if so, what they are using.
If your needs are simply that your station needs to do remote broadcasts, you need to look at – and ask for – demos of the different brands that are available and see which one suits you best. They all have different features and you need to look at the pros and cons of each unit.
The big issue with broadcasters is that we need the audio in real time. I’ve said this before – the digital stream we receive from an IP codec is quite different from the digital stream we receive from our online streaming operations. With an audio stream such as your listeners would get from your station, the player on their computer determines the internet conditions and forms a buffer – a place to store the data before it turns it back into audio for their listening pleasure. This buffer can be from 10 seconds to well over a minute depending on many factors.
While this continuous data stream is heading in their direction, many things can happen. The data is “packaged” in what we call packets. This makes transport more efficient. Every so often, a packet doesn’t reach its destination either in a timely fashion or at all. The player on the computer then shouts out to the server that it’s missing packet umpteesquat, so the server resends this packet, as both the server and the player keep track of what is being sent and received in a somewhat complex mathematical balancing act that is constantly going on in the background. Once the errant packet is received, the computer just quietly inserts it where it belongs. The result is the listeners don’t know that something went awry – they hear a continual stream of audio rather than a glitch, pop or silence.
In our world, we don’t have the luxury of large buffers. Can you imagine trying to talk up a record or interact with a phone call with 25 seconds of buffer time? It won’t work. So we need to keep the buffering time very small, usually considerably less than one half second. This doesn’t leave much time – or any time – for a missing packet to be resent. And typically, the receive end does not “talk” to the sending end in what we call synchronous real time data. The receive end is constantly doing math in the background. If a packet turns up missing, and let’s call it packet B, the receiver knows that it should have received packets A, B and C. It also knows that the mathematical sum of A plus B plus C equals Z. Well, if B is missing, it knows the sum total, and knows the values of packets A and C. Solving for the missing packet reveals the value of that packet, and the receiver can recreate it as best as it can by looking not only at the mathematical sum of the three packets, but also at how the other packets in the stream were structured. Normally, this results in an audio stream that has no audible problem. Sometimes there is a slight pop. But the audio gets there with as little delay as possible.
So how do we get around the “missing packet” problem? The manufacturers are all working on different schemes to try to recover lost packets better than they do at present. For extremely mission critical applications, several manufacturers use diverse paths; in other words, they send the same data stream out two or more paths (let’s say a DSL link, a cable internet link, a 4G link and perhaps a satellite link) to the other end. The other end also has two or more paths connected to it. The chances of the different paths missing the same packet at the same time is small. The receive end constantly monitors the packets and quality of such received and makes sure the best quality packets in the correct order are presented to the decoder.
One manufacturer has had a system connected between the US and the UK for over two years on two different internet providers. In that time, the amount of packets actually dropped is phenomenally small. It is less than 0.1%… which is about the reliability of our microwave STL units.
If that is overkill for your typical remote, then I would suggest you have a chat with the internet provider at your studio facility to see what they can do to make your pathway from “the cloud” to your site as transparent as possible. Talk to your IT persons about adding Quality of Service (QoS) priority to the packets presented by the IP codecs. QoS lets all the internal networking equipment know that these packets are of utmost importance and MUST get through and take priority over all other traffic. If your studio audio system is IP, I would be careful with QoS if the codec, console system and automation system are all on the same network – you don’t want your remote to accidentally take your station off the air briefly because its packets take priority over anything else.
And get a demo of an IP codec and try it out. You may be quite surprised at the quality and simplicity of your remote audio and setups. I am involved with the Ron Ananian: The Car Doctor radio program. Since we took the program into independent syndication on January 26, 2013, we have been sending the show to the satellite uplink via IP codec. I think we’ve had one minor glitch which resulted in a one second silent gap. Listening to it off the bird, you can’t tell it’s an IP codec. As a matter of fact, it sounds better than an ISDN codec would, as we’re sending it in at a higher data rate and using the AAC algorithm.
I also consult WJMJ radio in Connecticut, the radio station of the Archdiocese of Hartford. They use a digital microwave STL to get audio from their studios in Prospect to the transmitter in Burlington. They use an IP codec as a backup STL and, when their microwave unit failed a while back, they were on the IP codec for almost 3 weeks. Not one glitch, nor did any listener call in to complain – and they run classical music most evenings.
For now, if you have ISDN lines installed, leave them installed. If you find you cannot get ISDN for your remote, you will need to venture into the IP codec world. Get demo units. Play with them. Find out their capabilities (I think you will be surprised), and you will most likely find that they sound better than comparable ISDN units.
If your station is in an area where the internet is shaky, you may need to look around to see if there are alternatives to “the big” company in the area. If not, you may need to get an audience with the company president or, as a last resort, the state regulatory body that the company falls under to see what they can do to make your Internet connection reliable. I had one person contact me regarding IP codecs – the company he was with was a small company that didn’t even have a backup generator for their headend facility. I find this unacceptable in this day and age, particularly when even FEMA demands that we monitor them for emergency messages through our EAS equipment via an IP connection. The internet is no longer a novelty – it has become a necessity in most businesses. It’s time the providers stepped up to the plate – in every size market – and make it as reliable as the telephone system has been. But I fear the only way this will happen in some areas is if we constantly hammer these providers to make that happen.
The unfortunate part here is that I cannot produce any concrete answers as of yet. Only time will bring these answers to us. But being afraid of the internet? We can’t afford that fear any longer. When I started in this business, we utilized RPU links (the ubiquitous Marti unit) to get remote audio to the studio and, if that wouldn’t work, we got an equalized service from Ma Bell. This has changed several times over that 36 year period, and is changing yet again. We will survive this – but we can’t back away from it. As I hear of more developments, I will report them. Until then, get a demo, try it out, see what the capabilities are. The manufacturers do have our backs on this – and they run into the same limitations some of us have. They are working on alternatives to the problems that have developed. And as this progresses, it will only get better – until the next thing comes along.
Thomas R. Ray, III CPBE, AMD, DRB is president of Tom Ray Consulting and Technical Editor of TALKERS. He can be phoned at 845-418-5065 or emailed at email@example.com. His website is www.tomrayconsulting.com. Meet Tom Ray at TALKERS New York 2013 on Thursday June 6.