All Quiet on the Network Front

The building at 4500 Southgate is indistinguishable from the cookie-cutter offices that dot the outskirts of Dulles Airport in Virginia. But beyond a phalanx of security cameras, behind doors controlled by retina scanners and handprint readers, sits a room resembling the Norad command center depicted in the movie WarGames. This is a Secure Operations Center run by Counterpane Internet Security, one of a growing number of companies that monitor clients' computer networks -- from e-commerce sites to internal servers -- in search of malicious intruders.

With cybercrime paranoia soaring in the wake of several high-profile incidents -- from last spring's "I Love You" fiasco to late September's Disney World intrusion -- wired companies increasingly rely on third-party experts to keep a digital eye peeled for miscreants. According to the Gartner Group (IT) , $7.1 billion will be spent on security services this year, and that figure will grow by 40 percent annually for the near future.

It all sounds quite scary. Ninety percent of the respondents to a recent Computer Security Institute survey reported "computer security breaches" last year. But that figure includes such banal transgressions as employees downloading porn, exchanging bawdy e-mail jokes and pirating software. The real headline-grabbers -- stolen credit card numbers, pilfered trade secrets -- are frightening yet rare.

Hence, Secure Operations Centers like Counterpane's are the new-economy equivalent of the Alaskan radar stations that once scanned the skies for incoming Soviet ICBMs. Still, clients pay outfits like Counterpane, RIPTech and Pilot Network Services (PILT) as much as $12,000 per month for the peace of mind that comes with knowing their systems are being constantly monitored for unauthorized uses.

Curious to witness the action on the front lines of network security, The Standard spent 24 hours monitoring the monitors at Counterpane. Founded in 1999 by cryptographer Bruce Schneier, inventor of the still-unbroken Blowfish algorithm (and a contributor to The Standard), Counterpane was among the first companies to offer around-the-clock human surveillance.

On their watch, the monitors sift through a constant tidal wave of information looking for the minuscule anomalies -- a failed log-in attempt, a malfunctioning router -- that can indicate that a nascent attack is under way. A cyberattack is nearly impossible to detect with an untrained eye; a massive denial-of-service onslaught can appear to be nothing more than a few jargony command lines. Thus the culture of surveillance is one of patience. A typical 24-hour stretch includes a fair share of alerts, but also long stretches of thumb-twiddling. Sealed in a sterile, windowless room, SOC employees play digital-age voyeurs, scanning the horizons for the next calamity.

9: 00 am

Arrayed behind desks in a quasi-lecture hall arrangement, the three-person morning shift sits in near darkness facing a wall adorned with several massive screens. One displays a never-ending loop of shots from the 13 security cameras strewn throughout the facility. Another shows a continuous stream of data culled from the "sentries," Counterpane-speak for the PC installed behind a client's firewall to monitor network activity. The scrolling data resembles the hypnotic thicket of characters that Keanu Reeves gawked at in The Matrix.

Each sentry emits a regular "heartbeat," a signal that indicates it's up and running. A dormant heartbeat is cause for alarm; it indicates that Counterpane's surveillance abilities are temporarily crippled, and therefore a customer's network is ripe for exploitation -- whether by a disgruntled insider, a precocious preteen armed with hacking scripts or an evil mastermind in search of digitized loot. The latter is the most dangerous adversary, the one SOC analysts live to combat. But for every mastermind, there seems to be a million kiddies. And, even worse, a billion false alarms.

9:48 am

Eyes slightly straining in the SOC's dim, blue glow, Rob Jamison pores over a chronicle of the past week's activity for a client. The Web-hosting firm that Jamison is vetting experienced 1,642 so-called tickets in the previous seven days. "Ticket" is shorthand for an incident, and they are given one of four classifications. The lowest grade is "interesting," which refers to elementary glitches such as "printer out of toner" messages or brief traffic spikes. The next level is "security relevant," which can be something as minor as a mistyped password. Above that is "suspicious," which includes activities that can be preludes to attacks, such as scans that can detect weak firewalls or pliable backdoors. Finally, there is "critical," an attack in progress, something that requires immediate attention.

Only two tickets merit the suspicious label. Both are related to malfunctioning sentries that lost their heartbeats for over 10 minutes. But alas, it's nothing to get the heart racing; Jamison simply chalks up the downtime to hardware problems on the customer's end.

10:02 am

Don DeBolt may be a technogeek, but he has the solid build and close-cropped hair of a newly minted Marine. He terms himself an "ethical hacker" and did his fair share of penetration testing while at Ernst & Young. Now, as a senior SOC engineer, he decides when to "escalate" a ticket -- that is, to notify a client of a security issue and provide counsel on how to react. DeBolt calls his cramped office the War Room.

11:41 am

DeBolt emerges from the War Room, brow furrowed. One sentry has been down for 14 hours, reportedly because of a hardware problem. But the continued lack of a heartbeat is worrying him. He studies the ticket on a console and makes a fateful judgment call: "Let's go on and escalate this one."

Jamison calls the client to assess the situation. But the tech person he reaches there isn't exactly in the loop. "The guy doesn't have root access and he doesn't have physical access, so there's not a lot they can do," groans Jamison.

Clients may be willing to let Counterpane log and analyze their most sensitive network traffic, but they stop short of giving them the power to hit the kill switch. DeBolt calls this look-but-don't-touch access "limited keys to the kingdom."

1:59 pm

The boredom is beginning to take its toll as the morning watch draws to a close. MSNBC has been playing over the PA system for hours, and the umpteenth story about VP Al Gore is starting to grate on people's nerves.

2:17 pm

Differing musical tastes cause a mini ruckus. John Glasscock, the newest member of the SOC team, says "I like country, I like ska, I like all sorts of stuff. Except for what he likes." He points at a grinning Jason Van Brecht, the team's resident Unix snob and a rabid industrial music fan.

The sterility of the SOC contributes to the stir-craziness. There are none of the typical accoutrements of "wacky" new-economy culture. The only sign of color is a lone mousepad shaped like a pizza.

3:24 pm

Second-shifter Kathy Wang plows through a bowl of microwaved ravioli and mulls over her career anxiety. "I feel like I'm running out of time," sighs the 27-year-old. Eager for a slice of computer-world glory, she's developing an intrusion-detection system for ISPs but has found it difficult to get research done on the job, especially since laptops are verboten at the SOC to ensure that analysts won't walk away with sensitive data.

11:27 pm

Techie to the core, graveyard-shifter Rodney Mitchell pontificates on a future in which computer chips will be embedded in everything from rocking chairs to neckties. "Soon you'll be able to walk up to a Coke machine and put your watch up to it, or your cell phone, and you'll get a Coke and it'll all be paid for," he predicts. "But of course, that increases the vulnerabilities. Somebody can go up and hack a Coke machine." That insecurity partly explains why Mitchell joined Counterpane; as a man who takes pride in trendspotting, he foresees a rosy future for companies aimed at keeping technothugs from swiping sodas.

2:27 am

Dozing behind his console, Mitchell is startled awake by a series of four bings. He scrambles to make sense of the incoming tickets, which indicate that a system error has occurred on a client's network. "They've been doing maintenance since Friday, but this looks different from a maintenance event," says DeBolt, emerging slightly bleary-eyed from the War Room.

Mitchell gets on the horn to the sister SOC in Mountain View, Calif. "You guys looking at these tickets? ... They're having some adjacency problem? ... Things quiet on that end? ... OK, take care."

4:59 am

Jamison arrives a minute early for his shift to find Mitchell studying a Cisco manual. Van Brecht arrives shortly and immediately opens a cable cabinet to retrieve his stealthily concealed wool blanket. "It gets so cold in here in the morning," he says, bundling up as he flicks on his console's monitors. "The air conditioning gets a little crazy."

These are the slowest hours at the SOC, when even the hardiest cybercriminals are probably fast asleep. Bings are rare, and the crew occupies themselves with debating the relative merits of sourdough vs. whole-wheat toast. ZDTV's programming has turned from late-night infomercials to a show on international business, featuring a segment on the Bahamian communications minister. "Oh, that must be a tough job," snickers Van Brecht. "What is that, like, four phones? I'd take that job."

8:58 am

"Any tickets?" DeBolt squawks over a speakerphone, shattering the lighthearted mood. Van Brecht leans over to report that the morning has been quiet. DeBolt, operating on only a few hours sleep, orders his troops to start checking whether a database of client information is accessible. The ribaldry ceases and the tappity-tap of keyboards commences. The networks demand constant attention, even if the proverbial ICBM never comes streaking across the digital sky.