Professional security researchers and practitioners all have their horror stories. In this article, I share a few that are sure to make you wince.
I have seen each of these problems in my travels this year. They were ugly. They were recoverable, save one.
The tone I set is designed to make you mad. That’s intentional; I want you to act on your anger by either smugly saying “It’s done!” or by admitting that you’re going to work on it. Now.
1. Your Good Old DNS Server Is Attacking an Unknown Domain
Your DNS server may easily become an attacker or an attack amplifier if its configuration is incorrect. You need to know and understands its settings to prevent this. BIND is different from Microsoft’s implementation, the two most popular DNS servers, so the settings are slightly different; but they have to be dead-on or you’ll end up allowing your DNS server to burn lots of traffic at DoS and DDoS targets.
DNS works to resolve a canonical name into an IP address. It has a place where it has a start of authority—your internal stuff—and where it looks to upstream to other DNS servers for further information. Requests for information to your DNS servers (yes, you have one: Find it and check it) should only come from inside your organization. Monitoring your DNS servers for outgoing traffic to external destinations to find actual traffic, is a dead-giveaway that your DNS server’s being inadvertently used to send DoS or amplified DoS traffic.
If you allow what are called recursive queries, your DNS server can be suddenly abused in a scheme known as a recursion DDoS or amplified recursion attack on another organization’s DNS server. It’s a DoS attack that often is a member of a Distributed (DDoS) attack. There are public DNS server settings that can prevent the manipulation of a DNS server into an attack barrage of queries against other servers.
You need to ensure that your DNS server isn’t an open resolver, unless you know exactly what you’re doing. You do no service to anyone by being an open resolver, unless you’re someone like Google, who with Level3, supports the open resolver known as 8.8.8.8 in the IPv4 world. That one is bolted down correctly.
2. Your Backup Media Cartage Logs Are Missing
If someone is taking your backup media offsite for safe storage, you’re likely complying with the best practices that say: Keep them off premises in a logged archive for safe keeping. Perhaps your motivations to do so are various industry regulations or legal requirements like e-discovery. But you can’t just throw them in the back seat, drive them someplace, and that’s magically the end of it—as though these were the contents of a recycling can.
Procedurally, your organization’s data is worth a lot. Your CEO would be happy to lecture on its value prior to your exit interview. This is why a chain of authorities and responsibilities for all contractors and personnel that touch that data must be established, along with a set of handling procedures and receipts. Paper is great but an online SaaS application is even better.
Tapes left in taxis, missing in a truck, a box, a cabinet, or someplace where they’re not supposed to be is a no-no of Biblical proportion. Rapid restoral, e-discovery, and the mandated audit chain requirements mandate that you should know exactly where all backup media is currently located, its state, its contents, and the next several steps in where it will be next.
Barring any piece of that information, or doubts of its integrity, is an unemployment line looking for a spot marked X. You’ve checked all of the sign-offs recently, right?
3. Patch/Fix Verification Audit Failures
It seems that most break-ins involve finding servers that aren’t patched. This seems like a no-brainer, but many organizations lack a central repository for verification of patch/fix state of both public and private facing servers. Once an infected machine connects via either side of a network boundary (public or private), searches begin to find targets for other cracks. Intrusion detection systems often find infected machine request packets, but zero-day and especially very old attacks may still work.
The devil is in the details. Perhaps you’ve got great patch/fix verification software that sends klaxons screaming and text messages to your phone at the restaurant when a single server falls out of patch levels. Or maybe you have to do the tedious work of first learning which levels are appropriate for what operating systems and apps, then two, going through the logs or using other query methods to verify and audit that 100% of your domain is patched. But you’re busy.
Basic server configuration for patches and fixes varies by operating system, which begs the practice of limiting the number that your organization supports. In turn, supporting fewer operating systems has the effect of preventing entropic efforts towards keeping patch levels tight and audited.
4. It’s the Password, Stupid
Here’s a record from one of my secure logs, pasted from just a few minutes ago:
Oct 17 04:53:20 nframex sshd[48674]: Invalid user server from 222.33.61.196
Oct 17 04:53:22 nframex sshd[48678]: Invalid user linux from 222.33.61.196
Oct 17 04:53:24 nframex sshd[48682]: Invalid user info from 222.33.61.196
Oct 17 04:53:26 nframex sshd[48686]: Invalid user mail from 222.33.61.196
Oct 17 04:53:28 nframex sshd[48690]: Invalid user operator from 222.33.61.196
Oct 17 04:53:31 nframex sshd[48694]: Invalid user guest from 222.33.61.196
Oct 17 04:53:33 nframex sshd[48698]: Invalid user ftp from 222.33.61.196
Oct 17 04:53:35 nframex sshd[48702]: Invalid user oracle from 222.33.61.196
Oct 17 04:53:37 nframex sshd[48706]: Invalid user oracle from 222.33.61.196
Oct 17 04:53:39 nframex sshd[48710]: Invalid user test from 222.33.61.196
Oct 17 04:53:42 nframex sshd[48714]: Invalid user soporte from 222.33.61.196
Oct 17 04:53:44 nframex sshd[48731]: Invalid user postgres from 222.33.61.196
Oct 17 04:53:46 nframex sshd[48739]: Invalid user henry from 222.33.61.196
Oct 17 04:53:48 nframex sshd[48744]: Invalid user test from 222.33.61.196
Oct 17 04:53:50 nframex sshd[48748]: Invalid user testing from 222.33.61.196
Oct 17 04:53:53 nframex sshd[48752]: Invalid user webadmin from 222.33.61.196
Oct 17 04:53:55 nframex sshd[48756]: Invalid user trixbox1 from 222.33.61.196
Oct 17 04:53:57 nframex sshd[48760]: Invalid user testing from 222.33.61.196
Oct 17 04:53:59 nframex sshd[48764]: Invalid user test from 222.33.61.196
Oct 17 04:54:01 nframex sshd[48768]: Invalid user xbox from 222.33.61.196
Oct 17 04:54:15 nframex sshd[48782]: Invalid user oracle from 222.33.61.196
Oct 17 04:54:17 nframex sshd[48786]: Invalid user phpl from 222.33.61.196
Oct 17 04:54:19 nframex sshd[48790]: Invalid user user from 222.33.61.196
Oct 17 04:54:21 nframex sshd[48794]: Invalid user maggie from 222.33.61.196
I don’t know who this joker named 222.33.61.196 is, but I’ll complain shortly. The main ExtremeLabs.Com servers get attacked like this a dozen times daily, and we’re comparatively unknown. Note that the attack is on ssh, and someone’s trying a dictionary attack using commonly used logon names. Once this app gets an invalid key error, this automated attacker knows a valid user name and will start to attack the ssh key.
There should never never never be a logon name for any service, secure or not, that can be attacked with cheap dictionary attacks. Password strength is commonly argued among security professionals, as there are password attack applications that can pound 300K+ names per second if the application lets them. That’s why strong passwords combined with application time outs after failed logons are important.
In our case, I dropped the above offending IP address into an exclude table on the firewall, complained to the ISP, and went on my merry way.
5. With Physical Access, Anything Is Possible
Every single operating system maker will tell you the same thing: If you can touch a machine physically, all bets are off. Anyone who can hit the big red reset button or the power switch can do damage to your servers, SANs, routers, patch panels, even your PBX and phone equipment.
ISPs and MSPs often subscribe to audits undertaken to get SAS70-II and the newer replacement SSAE 16 ratings. These ratings ought to specify which employees under what circumstances get access to what equipment when, and how logs are kept for personnel that have direct access to your infrastructure.
There are countless ways to insert equipment into NOCs, or otherwise to compromise assets your organization owns across the planet, unless accessibility is defined, systematized, enforced, and audited. One flash drive can take your organization down, or cause endless amounts of litigation or regulatory issues.
Perhaps your organization already employs physical access policies, has nice access control infrastructure, and rules and regulations. When’s the last time you audited these? Have you considered hiring penetration testers to see if your organization is so cozy that it can be socially engineered into a breech? How about crawling through plenums?
The last breach we saw was a guy in a service uniform walking in behind employees who were returning from a lunch outing. He walked up to the Halon switch and took a picture of himself holding the switch, then MMS’d the picture to the CIO. His check was in the mail the next day.
6. And Just How Controlled Is Your Records Storage Facility?
Old records are like the trash; we hope we never see them again. I watch Iron Mountain trucks outside of larger organizations, merrily digesting carton after carton of paper. Such methods of destruction are often really good, but what of the records that have to be kept?
Most organizations have a record retention and e-discovery-fueled records policy. Along with the policy are a set of contractors who do the heavy lifting of organizational records, sometimes tapes or media, sometimes paper receipts and docs. Have you seen exactly where they go?
Some storage vendors subscribe to high standards. But these morgues of data are rarely visited, except by the records destruction people and occasionally the legal staff. Rarely does anyone actually go to the facility and audit the vendors.
Early this spring I traveled with an exec to their facility, located not far from his facility. The drill was to find a specific tape, in a specific box. We went to the storage facility’s counter. It would be a few minutes. We played Angry Birds and did texts. A half hour later, the tape couldn’t be found. We traveled back to where the storage container was supposed to be. There were mouse droppings everywhere, and numerous boxes had been chewed. By the looks of the dust on the floor, several mouse traps had been removed, but dried blood was still nearby.
Eventually the box turned up and so did the tape, and the tape worked. The humidity controls were working, and the tape hadn’t oxidized. The exec knows that others have oxidized, and are probably useless now—but are old enough not to matter. He has to keep them for ten years.
7. Your Leaky Hole of Social Media
Facebook, Google +, Twitter, and a few dozen other sites are really popular, and they’re a great place to exchange information and gain insight. And for regulated organizations (which includes most of us, today), they’re great leaky pipes.
People talk. And when they talk, they can give vast amounts of what should be proprietary information, unwittingly, to brag, or sooth egos, and so forth. The motivation isn’t the problem. Leaking information, however, is a problem.
Going to get a great bonus? Oh—that means sales are up! Hmmm, let’s go buy a few of those shares. Changes in the boardroom? Uh oh. New product coming out? Vendor trouble? The subpoena power in litigation today means that virtually all public network conversations are an easy grab when an injured party decides to sue. New e-Discovery tenets give large sway to admitting things like Twitter posts and conversations, and just about any kind of public online media.
You might be able to stop social media use on “company time.” But you need to establish, with the help of your legal department, a policy regarding the dissemination of your organization’s information, personnel, and employee involvement with your organization as it regards social media, social media e-mail, even what a Linked-In profile can say.
Loose lips sink ships. Social media seems to have become an ideal way to kvetch, complain, express joy and happiness, as well as share interesting information. Sometimes the information can be free expressions of ideas and opinions; other times, constraint should be shown. It’s a tricky area, and you need to understand how to implement social media information dissemination policies sooner, rather than later. Otherwise that new building you might be moving into at work might have bars.
- Subscribe to RSS Feed
- Mark as New
- Mark as Read
- Bookmark
- Subscribe
- Email to a Friend
- Printer Friendly Page
- Report Inappropriate Content
