Hands on at the DIRECTV Los Angeles Broadcast Center (LABC)
It's one of the coolest places I've ever seen. Last week, I was invited to an all-access tour of DIRECTV's Los Angeles Broadcast Center (LABC), where DIRECTV controls national satellite channels, live sports, and produces Audience Network programming. I was expecting to be blown away... I wasn't expecting to be this blown away.
LABC is located in an industrial neighborhood near LAX Airport. You'd never see it from the street... during my visit the fences were blacked out and the only way I could find it at all was by GPS. It's true, you can see the DIRECTV logo from the street, if you're looking for it of course. It looks, not surprisingly, like a building built in the 1990s; that was when DIRECTV started broadcasting.
I wasn't permitted to take pictures of security or of the lobby, and that's no surprise. This facility controls content that is seen by over 33 million homes across the Western Hemisphere, and then some; they're naturally a little squirrely about letting just anyone in. I was lucky enough to get a guided tour from a key DIRECTV executive, who took a lot of time out of his day to walk me around.
It all starts with power
It's a lot of power. I don't know how much power it takes to light up a small city. Every time you read one of these articles they want to tell you that it takes enough power to light up a small city. I don't know about that. I know this... behind these nondescript grey walls is a lot of power. About 20 megawatts of power. I was told that the bill is over a quarter of a million dollars per month.
It's not just power, either. It takes 7 backup generators to make sure that LABC is protected against any sort of power outage. Here's where you begin to understand the DIRECTV way -- it's not enough to have one of anything. There's a backup part for everything. One isn't good enough... in fact in most cases two aren't good enough. They need maximum overkill, whenever possible. 33 million people demand it.
Take a look at this room. You think you have a powerful UPS? This was about a 1,000 square foot room filled with batteries. And keep in mind, all they have to do is keep the lights on long enough to fire up the generators.
Processing the incoming signals
There are six massive main equipment rooms used for processing incoming signals. These racks just go on and on and on. I couldn't get too close... a lot of this equipment is proprietary. But, I did get a feeling of what happens in these rooms.
It starts with getting the signal through two sources. Whenever possible, DIRECTV doesn't just rely on one method to get a signal. If there's a satellite feed, they use that. If there's a fiber optic feed, they use that too. If you can get it over the air, that's part of the mix too. All in all over 4,000 input streams come in, both SD and HD. Every feed is processed before being converted to IP streams. That's right, this isn't really a television facility, it's a computer facility. And it's a big one.
These rooms go on and on and on. This is just one part of the redundant, backup encoding system,. Everything you see here is just used if something else goes down. There are rack upon rack upon rack of these things. That's why everything is converted to computer data. It's just impossible to scale up all the routing switchers you would need for 1,000 streams and keep it all in video format. IP format is much easier to scale. Of course, you have to know what you're doing, and you have to keep it neat.
At left you see just one of the hundreds of equipment racks. Everything is carefully labeled. Imagine what a mess it would be if it weren't. To the right you see these fairly pedestrian-looking servers. It's true, they look like the kind of thing you'd find everywhere. The cool part of these servers is what they do. They provide the "secret sauce..." adding guide data, interactive features, conditional access data, and the like to every video and audio stream.
What's the final result? This is a closeup of one of the two redundant systems that brings you VersusHD. Every HD Channel has two of these setups, and there are separate systems for SD. No wonder they need so much rack space.
I have a few demands...
This is one small part of the racks that make up DIRECTV's on-demand systems, DIRECTV Cinema and DIRECTV Everywhere. All on-demand and streaming programming is stored here. I was told they started in 2007 with 30 terabytes (that's 30,000 gigabytes) of storage and kept adding. Each one of those grey boxes is 30 terabytes and they have around 150 of them. That's 4.5 petabytes of storage. Put it this way, it's enough to store 1,000,000 DVDs. Yes really.
Look, up in the sky
Once all the signals are processed, checked, proper commercials added, rechecked again, modulated, upcoverted and mixed, all that's left is getting them up in the sky. There are two signal feeds for each satellite, corresponding to the left and right circular polarization. A lot of the hard work is done here because there are so many different dish installs, the signal has to be really perfect.
This meant I was headed outside to the dish farm. This is one of the world's largest dish farms. I mean seriously, look at these things. The only way you can really appreciate them is from an aerial shot. You can see the two dishes I photographed near the center of the picture. They are... simply huge.
In fact, take a look at that "torus" dish that's at the left in the top picture. This is a massive, massive piece of machinery. There are only about 20 on the planet and DIRECTV owns three of them. Here you can see the 32 LNBs used to receive signals from 32 satellites at the same time. Without this dish, there would have to be one dish for every satellite and the dish farm would be the size of a small town.
Now, that's just the "down" part of the dish farm. Dishes are also used to send signals up, of course. The four dishes you see here are just a few of the broadcast dishes used to get that programming up to the satellite, where it bounces down to you. DIRECTV pioneered the use of amplifiers on the dish with some super-secret "traveling wave-tube" technology. Amps right on the dish are a lot more efficient, but they run hot. That's why every dish has an air-conditioner right on it, as you see in the picture on the right.
While you're walking around, it's easy to miss these 1-meter dishes (still twice the size of the ones on your home.) They're almost visible at the center left of the aerial shot. They pick up the satellite signals that the big dishes transmit, so that engineers inside the building can analyze and monitor it for quality control, 24 hours a day, 7 days a week.
Here's where the fun begins
If you're an NFL fan, you recognize the red zone channel studios. Well, they don't always look so exciting. When I toured the studio (picture at right) all the monitors were off and the stage was quiet. DIRECTV is very proud of this studio, as well as the other studios they use for original programming. A lot of content on the Audience Network, as well as all the mix channel content, is all produced here at the LA Broadcast Center.
Studio facilities like this one may not look like much when they're not in use, but they are the absolute latest in high-tech. There are more studios and edit rooms than I could count and they all look like this. There is a bunch of custom equipment in racks and just tons of monitors.
When I was touring, they were mixing and producing "The Dan Patrick Show" for the Audience Network. The cool thing is that Mr. Patrick was in his studio in the northeast. All the signals were converted to data and transmitted cross country, where the final production was done in a big room that looked more like a stock brokerage than a traditional TV studio.
Everything in the LA Broadcast Center is digital. That means when someone comes in with video tape (and unbelievably that still happens) it needs to be digitized right away. This photo shows two of the dozens of machines of different formats that can be used for real-time digitization of video tapes. Even though my tour guide wasn't really enthusiastic about a room full of VCRs, I thought it was pretty cool... it was the only part that really looked like a broadcast center!
Mr. Spock, you have the conn
Finally, at the end of the tour, I got to see the part I expected. This is the SD control room. Every one of those monitors is tuned to one of the SD channels on DIRECTV service. Several operators are on hand at all times to check that everything is going right.
Every SD feed is labeled. The SD Control room is still a very impressive part of the operation but.. it's nothing compared to...
This is HD master control. I couldn't even fit it all in the frame actually... there's even more to it. A main operator sits in that forward position, while a team behind the glass monitors all the HD signals. You can see that there are just tons of feeds here. It's server after server, rack after rack, and a truly incredible number of monitors.
Remember when I said "Hands On?"
Yes, I called this a "hands on" review. Thanks to DIRECTV, it really was. While I was in the HD control room, I asked the question you might be asking: Where are all the people? I had imagined one person per channel, just sitting their monitoring the signal with an oscilloscope and a TV. Well, if it were 1985 I would have been right.
In fact, there were a lot of people but for security reasons I didn't include them in the pictures. You can see a few here and there.
Still, a lot of the heavy lifting is done by computers. DIRECTV has spent millions upon millions developing complex software that can analyze a TV signal. It "knows" what a good signal should look like, what would affect its quality, and what things it should have. These computers constantly check things and alert the engineers when something doesn't look right. Remember, there's a backup for everything, a second piece of equipment that can pick up the slack at a moment's notice. This software even keeps a history of when devices lock up or have errors, so it can predict when that device should be replaced.
At the heart of it is this workstation. Here, the operator can check to make sure the right commercials are programmed and that everything in the signal chain is working properly. He can make changes as needed, too. This is really like operating a thousand TV stations at the same time!
And the hands on part... At roughly 11:30AM Pacific time, on the day of my visit, I TOOK CONTROL. The power! The power! Ok, it wasn't that dramatic. I did change the active monitoring to Food Network HD and checked every aspect of the signal chain. It was all "green lights." For that one brief shining moment, everything about DIRECTV was in my hands. Good thing the software was actually doing the hard work!
My tour ended at the live sports production center. Here, over 70,000 live sporting events are handled each year. The team was actually working on a Nascar race at the time, coordinating the driver cameras and creating the mix channel. They were also gearing up for NFL Football. It takes a team of 350 people to produce NFL Sunday Ticket every week and a lot of it is done here.
There was one last surprise in my tour of the sports center. You would think DIRECTV engineers would have their hands full with US Sports. This part of the facility also handles sports for a lot of the other countries where DIRECTV operates as well. Take a look at the closeup of the rack and you'll see the rare LHR22 Sky+ HD DVR. It may not look all that different from our HR22, but it's only for Latin America.
I walked away thoroughly impressed. The sheer scope of this place, the sheer amount of equipment... at some point you look at it and say, "oh yeah, another 100 monitors..." it's crazy. But, it's all necessary. Remember, everything has a backup, everything has a plan, and the amazing thing is that with all this equipment, DIRECTV puts 1,000 channels up in the sky with practically no downtime. When something happens, it's rare, and it gets handled quickly.
I'd like to thank the senior management at the Los Angeles Broadcast Center for such a comprehensive tour, and also thank the staff for being so friendly and accommodating. (Sorry I flashed some of you in the eyes.)
If you ever had any question where your money goes when you pay your DIRECTV bill... now you know.