The building at 4500 Southgate is indistinguishable
from the cookie-cutter offices that dot the outskirts of Dulles
Airport in Virginia. But beyond a phalanx of security cameras,
behind doors controlled by retina scanners and handprint readers,
sits a room resembling the Norad command center depicted in
the movie WarGames. This is a Secure Operations Center run by
Counterpane Internet Security, one of a growing number of companies
that monitor clients' computer networks -- from e-commerce sites
to internal servers -- in search of malicious intruders.
With cybercrime paranoia soaring in the wake of several high-profile
incidents -- from last spring's "I Love You" fiasco to late September's
Disney World intrusion -- wired companies increasingly rely on
third-party experts to keep a digital eye peeled for miscreants.
According to the Gartner Group (IT) , $7.1 billion will be spent
on security services this year, and that figure will grow by
40 percent annually for the near future.
It all sounds quite scary. Ninety percent of the respondents
to a recent Computer Security Institute survey reported "computer
security breaches" last year. But that figure includes such
banal transgressions as employees downloading porn, exchanging
bawdy e-mail jokes and pirating software. The real headline-grabbers
-- stolen credit card numbers, pilfered trade secrets -- are frightening
yet rare.
Hence, Secure Operations Centers like Counterpane's are the
new-economy equivalent of the Alaskan radar stations that once
scanned the skies for incoming Soviet ICBMs. Still, clients
pay outfits like Counterpane, RIPTech and Pilot Network Services
(PILT) as much as $12,000 per month for the peace of mind that
comes with knowing their systems are being constantly monitored
for unauthorized uses.
Curious to witness the action on the front lines of network
security, The Standard spent 24 hours monitoring the monitors
at Counterpane. Founded in 1999 by cryptographer Bruce Schneier,
inventor of the still-unbroken Blowfish algorithm (and a contributor
to The Standard), Counterpane was among the first companies
to offer around-the-clock human surveillance.
On their watch, the monitors sift through a constant tidal
wave of information looking for the minuscule anomalies -- a
failed log-in attempt, a malfunctioning router -- that can indicate
that a nascent attack is under way. A cyberattack is nearly
impossible to detect with an untrained eye; a massive denial-of-service
onslaught can appear to be nothing more than a few jargony command
lines. Thus the culture of surveillance is one of patience.
A typical 24-hour stretch includes a fair share of alerts, but
also long stretches of thumb-twiddling. Sealed in a sterile,
windowless room, SOC employees play digital-age voyeurs, scanning
the horizons for the next calamity.
9: 00 am
Arrayed behind desks in a quasi-lecture hall arrangement, the
three-person morning shift sits in near darkness facing a wall
adorned with several massive screens. One displays a never-ending
loop of shots from the 13 security cameras strewn throughout
the facility. Another shows a continuous stream of data culled
from the "sentries," Counterpane-speak for the PC installed
behind a client's firewall to monitor network activity. The
scrolling data resembles the hypnotic thicket of characters
that Keanu Reeves gawked at in The Matrix.
Each sentry emits a regular "heartbeat," a signal that indicates
it's up and running. A dormant heartbeat is cause for alarm;
it indicates that Counterpane's surveillance abilities are temporarily
crippled, and therefore a customer's network is ripe for exploitation
-- whether by a disgruntled insider, a precocious preteen armed
with hacking scripts or an evil mastermind in search of digitized
loot. The latter is the most dangerous adversary, the one SOC
analysts live to combat. But for every mastermind, there seems
to be a million kiddies. And, even worse, a billion false alarms.
9:48 am
Eyes slightly straining in the SOC's dim, blue glow, Rob Jamison
pores over a chronicle of the past week's activity for a client.
The Web-hosting firm that Jamison is vetting experienced 1,642
so-called tickets in the previous seven days. "Ticket" is shorthand
for an incident, and they are given one of four classifications.
The lowest grade is "interesting," which refers to elementary
glitches such as "printer out of toner" messages or brief traffic
spikes. The next level is "security relevant," which can be
something as minor as a mistyped password. Above that is "suspicious,"
which includes activities that can be preludes to attacks, such
as scans that can detect weak firewalls or pliable backdoors.
Finally, there is "critical," an attack in progress, something
that requires immediate attention.
Only two tickets merit the suspicious label. Both are related
to malfunctioning sentries that lost their heartbeats for over
10 minutes. But alas, it's nothing to get the heart racing;
Jamison simply chalks up the downtime to hardware problems on
the customer's end.
10:02 am
Don DeBolt may be a technogeek, but he has the solid build
and close-cropped hair of a newly minted Marine. He terms himself
an "ethical hacker" and did his fair share of penetration testing
while at Ernst & Young. Now, as a senior SOC engineer, he decides
when to "escalate" a ticket -- that is, to notify a client of
a security issue and provide counsel on how to react. DeBolt
calls his cramped office the War Room.
11:41 am
DeBolt emerges from the War Room, brow furrowed. One sentry
has been down for 14 hours, reportedly because of a hardware
problem. But the continued lack of a heartbeat is worrying him.
He studies the ticket on a console and makes a fateful judgment
call: "Let's go on and escalate this one."
Jamison calls the client to assess the situation. But the tech
person he reaches there isn't exactly in the loop. "The guy
doesn't have root access and he doesn't have physical access,
so there's not a lot they can do," groans Jamison.
Clients may be willing to let Counterpane log and analyze their
most sensitive network traffic, but they stop short of giving
them the power to hit the kill switch. DeBolt calls this look-but-don't-touch
access "limited keys to the kingdom."
1:59 pm
The boredom is beginning to take its toll as the morning watch
draws to a close. MSNBC has been playing over the PA system
for hours, and the umpteenth story about VP Al Gore is starting
to grate on people's nerves.
2:17 pm
Differing musical tastes cause a mini ruckus. John Glasscock,
the newest member of the SOC team, says "I like country, I like
ska, I like all sorts of stuff. Except for what he likes." He
points at a grinning Jason Van Brecht, the team's resident Unix
snob and a rabid industrial music fan.
The sterility of the SOC contributes to the stir-craziness.
There are none of the typical accoutrements of "wacky" new-economy
culture. The only sign of color is a lone mousepad shaped like
a pizza.
3:24 pm
Second-shifter Kathy Wang plows through a bowl of microwaved
ravioli and mulls over her career anxiety. "I feel like I'm
running out of time," sighs the 27-year-old. Eager for a slice
of computer-world glory, she's developing an intrusion-detection
system for ISPs but has found it difficult to get research done
on the job, especially since laptops are verboten at the SOC
to ensure that analysts won't walk away with sensitive data.
11:27 pm
Techie to the core, graveyard-shifter Rodney Mitchell pontificates
on a future in which computer chips will be embedded in everything
from rocking chairs to neckties. "Soon you'll be able to walk
up to a Coke machine and put your watch up to it, or your cell
phone, and you'll get a Coke and it'll all be paid for," he
predicts. "But of course, that increases the vulnerabilities.
Somebody can go up and hack a Coke machine." That insecurity
partly explains why Mitchell joined Counterpane; as a man who
takes pride in trendspotting, he foresees a rosy future for
companies aimed at keeping technothugs from swiping sodas.
2:27 am
Dozing behind his console, Mitchell is startled awake by a
series of four bings. He scrambles to make sense of the incoming
tickets, which indicate that a system error has occurred on
a client's network. "They've been doing maintenance since Friday,
but this looks different from a maintenance event," says DeBolt,
emerging slightly bleary-eyed from the War Room.
Mitchell gets on the horn to the sister SOC in Mountain View,
Calif. "You guys looking at these tickets? ... They're having
some adjacency problem? ... Things quiet on that end? ... OK,
take care."
4:59 am
Jamison arrives a minute early for his shift to find Mitchell
studying a Cisco manual. Van Brecht arrives shortly and immediately
opens a cable cabinet to retrieve his stealthily concealed wool
blanket. "It gets so cold in here in the morning," he says,
bundling up as he flicks on his console's monitors. "The air
conditioning gets a little crazy."
These are the slowest hours at the SOC, when even the hardiest
cybercriminals are probably fast asleep. Bings are rare, and
the crew occupies themselves with debating the relative merits
of sourdough vs. whole-wheat toast. ZDTV's programming has turned
from late-night infomercials to a show on international business,
featuring a segment on the Bahamian communications minister.
"Oh, that must be a tough job," snickers Van Brecht. "What is
that, like, four phones? I'd take that job."
8:58 am
"Any tickets?" DeBolt squawks over a speakerphone, shattering
the lighthearted mood. Van Brecht leans over to report that
the morning has been quiet. DeBolt, operating on only a few
hours sleep, orders his troops to start checking whether a database
of client information is accessible. The ribaldry ceases and
the tappity-tap of keyboards commences. The networks demand
constant attention, even if the proverbial ICBM never comes
streaking across the digital sky.
Copyright 2000, The Industry Standard
Join the Conversation
Please log in below through Disqus, Twitter or Facebook to participate in the conversation. Your email address, which is required for a Disqus account, will not be publicly displayed. If you sign in with Twitter or Facebook, you have the option of publishing your comments in those streams as well.
Your tax-deductible gift will help bring promising new voices and ideas into our nation's discourse, and help shape the future of vital public policies.
Join the Conversation
Please log in below through Disqus, Twitter or Facebook to participate in the conversation. Your email address, which is required for a Disqus account, will not be publicly displayed. If you sign in with Twitter or Facebook, you have the option of publishing your comments in those streams as well.