The 49ers’ plan to build the greatest stadium Wi-Fi network of all time

When the San Francisco 49ers’ new stadium opens for the 2014 NFL season, it is quite likely to have the best publicly accessible Wi-Fi network a sports facility in this country has ever known.

The 49ers are defending NFC champions, so 68,500 fans will inevitably walk into the stadium for each game. And every single one of them will be able to connect to the wireless network, simultaneously, without any limits on uploads or downloads. Smartphones and tablets will run into the limits of their own hardware long before they hit the limits of the 49ers’ wireless network.

Jon Brodkin

Until now, stadium executives have said it’s pretty much impossible to build a network that lets every single fan connect at once. They’ve blamed this on limits in the amount of spectrum available to Wi-Fi, despite their big budgets and the extremely sophisticated networking equipment that largesse allows them to purchase. Even if you build the network perfectly, it would choke if every fan tried to get on at once—at least according to conventional wisdom.

But the people building the 49ers’ wireless network do not have conventional sports technology backgrounds. Senior IT Director Dan Williams and team CTO Kunal Malik hail from Facebook, where they spent five years building one of the world’s largest and most efficient networks for the website. The same sensibilities that power large Internet businesses and content providers permeate Williams’ and Malik’s plan for Santa Clara Stadium, the 49ers’ nearly half-finished new home.

“We see the stadium as a large data center,” Williams told me when I visited the team’s new digs in Santa Clara.

I had previously interviewed Williams and Malik over the phone, and they told me they planned to make Wi-Fi so ubiquitous throughout the stadium that everyone could get on at once. I had never heard of such an ambitious plan before—how could this be possible?

Today’s networks are impressive—but not unlimited

An expansive Wi-Fi network at this year’s Super Bowl in the New Orleans Superdome was installed to allow as many as 30,000 fans to get online at once. This offloaded traffic from congested cellular networks and gave fans the ability to view streaming video or do other bandwidth-intensive tasks meant to enhance the in-game experience. (Don’t scoff—as we’ve noted before, three-plus-hour NFL games contain only 11 minutes of actual game action, or a bit more if you include the time quarterbacks spend shouting directions at teammates at the line of scrimmage. There is plenty of time to fill up.)

Superdome officials felt a network allowing 30,000 simultaneous connections would be just fine, given that the previous year’s Super Bowl saw only 8,260 at its peak. They were generally right, as the network performed well, even for part of the game’s power outage.

The New England Patriots installed a full-stadium Wi-Fi network this past season as well. It was never used by more than 10,000 or so people simultaneously, or by more than 16,000 people over the course of a full game. “Can 70,000 people get on the network at once? The answer to that is no,” said John Brams, director of hospitality and venues at the Patriots’ network vendor, Enterasys. “If everyone tried to do it all at once, that’s probably not going to happen.”

But as more fans bring smart devices into stadiums, activities like viewing instant replays or live camera angles available only to ticket holders will become increasingly common. It’ll put more people on the network at once and require bigger wireless pipes. So if Williams and Malik have their way, every single 49ers ticket holder will enjoy a wireless connection faster than any wide receiver sprinting toward the end zone.

“Is it really possible to give Wi-Fi to 68,500 fans at once?” I asked. I expected some hemming and hawing about how the 49ers will do their best and that not everyone will ever try to use the network at once anyway.

“Yes. We can support all 68,500,” Williams said emphatically.

How?

“How not?” he answered.

Won’t you have to limit the capacity each fan can get?

Again, absolutely not. “Within the stadium itself, there will probably be a terabit of capacity. The 68,500 will not be able to penetrate that. Our intentions in terms of Wi-Fi are to be able to provide a similar experience that you would receive with LTE services, which today is anywhere from 20 to 40 megabits per second, per user.

“The goal is to provide you with enough bandwidth that you would saturate your device before you saturate the network,” Williams said. “That’s what we expect to do.”

Fans won’t be limited by what section they’re in, either. If the 49ers offer an app that allows fans to order food from their seats, or if they offer a live video streaming app, they’ll be available to all fans.

“The mobile experience should not be limited to, ‘Hey, because you sit in a club seat you can see a replay, but because you don’t sit in a club seat you can’t see a replay,'” Malik said. “That’s not our philosophy. Our philosophy is to provide enhancement of the game experience to every fan.” (The one exception would be mobile features designed specifically for physical features of luxury boxes or club seats that aren’t available elsewhere in the stadium.)

It’s the design that counts

Current stadium Wi-Fi designs, even with hundreds of wireless access points distributed throughout a stadium, often can support only a quarter to a half of fans at once. They also often limit bandwidth for each user to prevent network slowdowns.

The Patriots offer fans a live video and instant replay app, with enough bandwidth to access video streams, upload photos to social networks, and use the Internet in general. Enterasys confirmed to Ars that the Patriots do enforce a bandwidth cap to prevent individual users from overloading the network, but Enterasys would not say exactly how big the cap is. The network has generally been a success, but some users of the Patriots app have taken to the Android app store to complain about the stadium Wi-Fi’s performance.

According to Williams, most current stadium networks are limited by a fundamental problem: sub-optimal location of wireless access points.

“A typical layout is overhead, one [access point] in front of the section, one behind the section, and they point towards each other,” he said. “This overhead design is widely used and provides enough coverage for those using the design.”

Williams would not reveal the exact layout of the 49ers’ design, perhaps to prevent the competition from catching on. How many access points will there be? “Zero to 1,500,” he said in a good-natured attempt to be both informative and vague.

That potentially doubles or quadruples the typical amount of stadium access points—the Super Bowl had 700 and the Patriots have 375. But this number isn’t the most important thing. “The number of access points will not give you any hint on whether the Wi-Fi is going to be great or not,” Malik said. “Other factors control that.”

If the plan is to generate more signal strength, just adding more access points to the back and front of a section won’t do that.

The Santa Clara Stadium design “will be unique to football stadiums,” Williams said. “The access points will be “spread and distributed. It’s really the best way to put it. Having your antennas distributed evenly around fans.” The 49ers are testing designs in Candlestick Park and experimenting with different access points in a lab. The movement of fans and the impact of weather on Wi-Fi performance are among the factors under analysis.

“Think of a stadium where it’s an open bowl, its raining, people are yelling, standing, how do you replicate that in your testing to show that if people are jumping from their seats, how is Wi-Fi going to behave, what will happen to the mobile app?” Malik said. “There is a lot that goes on during a game that is hard to replicate in your conceptual simulation testing. That is one of the big challenges where we have to be very careful.”

“We will make great use of Candlestick over the next year as we continue to test,” Williams said. “We’re evaluating placement of APs and how that impacts RF absorption during the game with folks in their seats, with folks out of their seats.”

Wi-Fi will be available in the stands, in the suites, in the walkways, in the whole stadium. The team has not yet decided whether to make Wi-Fi available in outdoor areas such as concourses and parking lots.

The same could theoretically be done at the 53-year-old Candlestick Park, even though it was designed decades before Wi-Fi was invented. Although the stadium serves as a staging ground for some of the 49ers’ wireless network tests, public access is mainly limited to premium seating areas and the press box.

The reason Wi-Fi in Candlestick hasn’t been expanded is a practical one. With only one year left in the facility, the franchise has decided not to invest any more money in its network. But Williams said 100 percent Wi-Fi coverage with no bandwidth caps could be done in any type of stadium, no matter how old. He says the “spectrum shortage” in stadiums is just a myth.

With the new stadium still undergoing construction, it was too early for me to test anything resembling Santa Clara Stadium’s planned Wi-Fi network. For what it’s worth, I was able to connect to the 49ers’ guest Wi-Fi in their offices with no password, and no problems.

The 2.4GHz problem

There is one factor preventing better stadium Wi-Fi that even the 49ers may not be able to solve, however. Wi-Fi works on both the 2.4GHz and 5GHz bands. Generally, 5GHz is better because it offers more powerful signals, less crowded airwaves and more non-overlapping channels that can be devoted to Wi-Fi use.

The 2.4GHz band has 11 channels overall and only three that don’t overlap with each other. By using somewhat unconventionally small 20MHz channels in the 5GHz range, the 49ers will be able to use about eight non-overlapping channels. That’s despite building an outdoor stadium, which is more restricted than indoor stadiums due to federal requirements meant to prevent interference with systems like radar.

Each 49ers access point will be configured to offer service on one channel, and access points that are right next to each other would use different channels to prevent interference. So even if you’re surrounding fans with access points, as the 49ers plan to, they won’t interfere with each other.

But what if most users’ devices are only capable of connecting to the limited and crowded 2.4GHz band? Enterasys said 80 percent of Patriots fans connecting to Wi-Fi this past season did so from devices supporting only the 2.4GHz band, and not the 5GHz one.

“You have to solve 2.4 right now to have a successful high-density public Wi-Fi,” Brams said.

The iPhone 5 and newer Android phones and tablets do support both the 2.4GHz and 5GHz bands, however. Williams said by the time Santa Clara Stadium opens in 2014, he expects 5GHz-capable devices to be in much wider use.

When asked if the 49ers would be able to support 100 percent of fans if most of them can only connect to 2.4GHz, Williams showed a little less bravado.

“For those 2.4 users we will certainly design it so that there’s less interference,” he said. “It is a more dense environment if you are strictly constrained in 2.4, but we are not constrained in 2.4. We’re not trying to answer the 2.4 problem, because we have 5 available.”

“It’s 2013, we have another year and a half of iteration,” he also said. “We’ll probably be on, what, the iPhone 7 by then? The move to 5GHz really just makes us lucky. We’re doing this at the right time.”

Building a stadium in Facebook’s image

Williams and Malik both joined the 49ers last May. Malik was hired first, and then brought his old Facebook friend, Williams, on board. Malik had been the head of IT at Facebook, while Williams was the website’s first network engineer and later a director. They both left the site, basically because they felt there was nothing left to accomplish. Williams did some consulting, and Malik initially planned to take some time off.

Williams was a long-time 49ers season ticket holder, but that was far from the only thing that sold him on coming to the NFL.

“I had been looking for something challenging and fun again,” Williams said. “Once you go through an experience like Facebook, it’s really hard to find something that’s similar. When Kunal came to me, I remember it like it was yesterday. He said, ‘If you’re looking for something like Facebook you’re not going to find it. Here’s a challenge.'”

“This is an opportunity to change the way the world consumes live sports in a stadium,” Malik said. “The technology problems live sports has today are unsolved and no one has ever done what we are attempting to do here. That’s what gets me out of bed every day.”

Williams and Malik have built the 49ers’ network in Facebook’s image. That means each service—Wi-Fi, point-of-sale, IPTV, etc.—gets its own autonomous domain, a different physical switching system to provide it bandwidth. That way, problems or slowdowns in one service do not affect another one.

“It’s tribal knowledge that’s only developed within large content providers, your Facebooks, your Googles, your Microsofts,” Williams said. “You’ll see the likes of these large content providers build a different network that is based on building blocks, where you can scale vertically as well as horizontally with open protocols and not proprietary protocols.

“This design philosophy is common within the content provider space but has yet to be applied to stadiums or venues. We are taking a design we have used in the past, and we are applying it here, which makes sense because there is a ton of content. I would say stadium networks are 10 years behind. It’s fun for us to be able to apply what we learned [at Facebook].”

The 49ers are still evaluating what Wi-Fi equipment they will use. The products available today would suit them fine, but by late 2014 there will likely be stadium-class access points capable of using the brand-new 802.11ac protocol, which allows greater throughput in the 5GHz range than the widely used 802.11n. 11ac consumer devices are rare today, but the 49ers will use 802.11ac access points to future-proof the stadium if appropriate gear is available. 11ac is backwards compatible with 11n, so supporting the new protocol doesn’t leave anyone out—the 49ers also plan to support previous standards such as 11a, 11b, and 11g.

802.11ac won’t really become crucial until 802.11n’s 5GHz capabilities are exhausted, said Daren Dulac, director of business development and technology alliances at Enterasys.

“Once we get into 5GHz, there’s so much more capacity there that 11ac doesn’t even become relevant until we’ve reached capacity in the 5GHz range,” he said. “We really think planning for growth right now in 5GHz is acceptable practice for the next couple of years.”

Santa Clara Stadium network construction is expected to begin in Q1 2014. Many miles of cabling will support the “zero to 1,500” access points, which connect back to 48 server closets or mini-data centers in the stadium that in turn tie back to the main data center.

“Based on service type you plug into your specific switch,” Williams said. “If you’re IPTV, you’re in an IPTV switch, if you’re Wi-Fi you’re in a Wi-Fi switch. If you’re in POS [point-of-sale], you’re in a POS switch. It will come down to a Wi-Fi cluster, an IPTV cluster, a POS cluster, all autonomous domains that are then aggregated by a very large fabric, that allows them to communicate lots of bandwidth throughput, and allows them to communicate to the Internet.”

Whereas Candlestick Park’s network uses Layer 2 bridging—with all of the Wi-Fi nodes essentially on a single LAN— Santa Clara Stadium will rely on Layer 3 IP routing, turning the stadium itself into an Internet-like network. “We will be Layer 3 driven, which means we do not have the issue of bridge loops, spanning tree problems, etc.,” Williams said.

Keeping the network running smoothly

Wireless networks should be closely watched during games to identify interference from any unauthorized devices and identify usage trends that might result in changes to access points. At the Patriots’ Gillette Stadium, management tools show bandwidth usage, the number of fans connected to each access point, and even what types of devices they’re using (iPhone, Android, etc.) If an access point was overloaded by fans, network managers would get an alert. Altering radio power, changing antenna tilt, or adding radios may be required, but generally any major changes are made between games.

Enlarge / Dashboard view of Patriots’ in-game connectivity.
Enterasys

“In terms of real-time correction, it depends on what the event is,” said John Burke, a senior architect at Enterasys. “Realistically, some of these APs are overhead. If an access point legitimately went down and it’s on the catwalk above 300 [the balcony sections] you’re not going to fix that in the game. That’s something that would have to wait.”

So far, the Patriots’ capacity has been enough. Fans have yet to overwhelm a single access point. Even if they did, there is some overlap among access points, allowing fans to get on in case one AP is overloaded (or just broken).

The 49ers will use similar management tools to watch network usage and adjust access point settings in real time during games. “We expect to overbuild and actually play with things throughout,” Williams said. “Though we are building the environment to support 100 percent capacity, we do not expect 100 percent capacity to be used, so we believe we will be able to move resources around as needed [during each game].”

The same sorts of security protections in place in New England will be used in Santa Clara. Business systems will be password-protected and encrypted, and there will be encrypted tunnels between access points and the back-end network. While that level of protection won’t extend to the public network, fans shouldn’t be able to attack each other, because peer-to-peer connections will not be allowed.

What if the worst happens and the power goes out? During the Super Bowl’s infamous power outage, Wi-Fi did stay on for at least a while. Williams and Malik acknowledged that no system is perfect, but they said that they plan for Wi-Fi uptime even if power is lost.

“We have generators in place, and we’ll have UPS systems, so from a communications standpoint our plan is to keep all the communication infrastructure up and online [during outages],” Williams said. “But all of this stuff is man-made.”

A small team that does it all

Believe it or not, the 49ers have a tech team of less than 10 people, yet the organization is designing and building everything itself. Sports teams often outsource network building to carriers or equipment vendors, but not the 49ers. Besides building its own Wi-Fi network, the team will build a carrier-neutral distributed antenna system to boost cellular signals within the stadium.

“We are control freaks,” Williams said with a laugh. He explained that doing everything themselves makes it easier to track down problems, accept responsibility, and fix things. They also feel the need to take ownership of the project because none of the existing networks in the rest of the league approach what they want to achieve. There is a lot of low-hanging fruit just from solving the easy problems other franchises haven’t addressed, they think.Not all the hardware must be in-house, though. The 49ers will use cloud services like Amazon’s Elastic Compute Cloud when it makes sense.

“Let’s say we want to integrate a POS system with ordering,” Malik said. “If you have an app that lets you order food, and there’s a point of sale system, all the APIs and integration need to sit in the cloud. There’s no reason for it to sit in our data center.”

There are cases where the cloud is clearly not appropriate, though. Say the team captures video on site and distributes it to fans’ devices—pushing that video to a faraway cloud data center in the middle of that process would slow things down dramatically. And ultimately, the 49ers have a greater vision than just providing Wi-Fi to fans.

When I toured a preview center meant to show off the stadium experience to potential ticket buyers, a mockup luxury suite had an iPad embedded in the wall with a custom application for controlling a projector. That provides a hint of what the 49ers might provide.

“Our view is whatever you have at home you should have in your suite,” Williams said. “If that means there’s an iPad on the wall or an application you can use, hopefully that’s available. Your life should be much easier in this stadium.”

And whatever applications are built should be cross-platform. As Malik said, the 49ers are moving away from proprietary technologies to standards-based systems so they can provide nifty mobile features to fans regardless of what device they use.

Williams and Malik are already working long hours, and their jobs will get even more time-intensive when network construction actually begins. But they wouldn’t have it any other way—particularly the longtime season ticket holder Williams.

When work is “tied to something that you love deeply, which is sports, and tied to your favorite team in the world, that’s awesome,” Williams said. “I’m crazy about it, man. I get super passionate.”

Source:  arstechnica.com

Tags: , , , ,

Leave a Reply

You must be logged in to post a comment.