Saturday 30 March 2013

Scams in security testing

Dedicated to people who submit Web scanner results to their software vendors.

A while ago I stumbled upon a book on software testing. Not security, mind you, just plain normal software testing. By my favourite "techie" author Gerald Weinberg - Perfect software and other illusions about software testing. It's a great read for app security folks, as long as you are capable of making basic domain substitutions.

My favourite chapter in the book is "Testing scams", where the author follows up his earlier discussion of fallacies in testing with a list of outright scams by vendors promising to sell a magic testing tools. He says
"Here's the secret about tools: Good tools amplify effectiveness. If your testing effectiveness is negative, adding tools will only amplify the negativity. Any other claim a tool vendor makes is, most likely, some kind of scam."
I made a short summary of this chapter, with examples from security testing domain (mostly web, "dynamic" and source code, "static" scanners). Text in quote marks is from the book, apart from the obvious phrases.

1. "Tool demonstration is a scam" - where you are shown a perfect demo, of a scanner running on WebGoat. And when you try it on your own code, the scanner explodes with junk. Of course the demo was carefully designed and tuned to produce the best impression.

Subtype 1a: You are not allowed do your own PoC without vendor's helpful supervision. At the least, they will give you a spreadsheet with criteria to compare their product against others.

Note: If you are not capable of conducting your own PoC without asking vendors for help, you should not be buying any of those tools in the first place.

2. "With all these testimonials, it must be good" - where there isn't even a demo, but you get a pile of whitepapers with pseudo-test results (comparisons, certifications, endorsements). These docs usually "appear to contain information, but ... only identify charlatans who took a fee or some payment [in kind] for the used of their names".

As a test, try requesting personal testimonies from the customers involved, ideally see how they use the tool. If the vendor cannot produce a single customer who is excited about their product so much that they want to show you how wonderful it is, it's a crap tool.

3. "We scam you with our pricing" - where the vendor creates cognitive dissonance among previously scammed people. As a result, despite the expensive purchase being a failure on all levels, from purchasers to end users, they keep this fact to themselves.

Subtype 3a: "[is] discrediting competitive tools with disingeniousness, suggesting, 'With a price so low, how could those tools be any good?'"

4. "Our tool can read minds" - where the tool is presented as a complete replacement of a security specialist - testing even better than a human, and not requiring any human post-processing. In personal experience, this is one of the most common scams in security testing market, with scam #1 being the next most popular, and #3 and #4 reserved for very pricey tools (you know who you are).

A belief that a magic app security silver bullet exists is so deep, that when the tool (or the "cloud" service) quickly fails to deliver to its promises, their customer concludes that this was his/her own mistake and there is another magic mind-reading service elsewhere. Rinse and repeat.

Note: There is no silver bullet. Artificial Intelligence is hard. Turing was right.

5. "We promise that you don't have to do a thing" - where a "cloud" service promises that all the customer has to do is to point it to the web app, and the service will spit out actionable results with no false positives and no false negatives. Less experienced security teams or managers fall for this one quite often, since many of the services come with a promise of manual postprocessing of results by the most experienced analysts in the world (or close to that). Where this fails is lack of context for the testing. The vendor does not know your code, they do not know your context in terms of exploitability, mitigating controls, or impacts. They do not know what technologies and methodologies your developers use. What usually comes out of such services is slightly processed results of a scanner with generic settings.

Subtype 5a: - when the service vendor does the old "bait and switch" between its personnel involved in the sales process (gurus) and who you get once you pay (little to no experience button pushers in a cheap outsourced location).

Still with me? Here's the summary:

If someone promises you something for nothing, it is a scam (or they are Mother Theresa, that is, not a business person). Even if you are promised a magic tool in exchange for a lot of money (scam 3 above), this is still a promise of something for nothing. 

It is impossible to do good security testing other than by employing (in one way or another) people who know the context of your environment and code or who are willing to learn it.

Thursday 14 March 2013

Medievalism in infosec

Dedicated to the last pope.

In my quest to understand the elusive American puritanist psyche I've been reading up on origins and history of Christianity recently.

As a side note - original biblical languages are so much fun. Not only nobody is quite sure which tense in Biblical Hebrew is past and which - future, but even when the meaning is obvious, translations do so much moralising and sweeping all the blood sex and genocide in the Old Testament under the carpet.

Example: did you notice how many times a woman fiddles (uncovers, kisses, touches etc) man's feet in the OT? But never the other way around or, God forbid, a man to a man? It turns out, "feet" is an euphemism :)

Anyhow, this post is about striking parallels between some old religious metaphors and the modern "cybersecurity" ones.

Infosec thinking as Judaism of 1st century BCE

A quote from a very respected Biblical scholar:

"Apocalyptic eschatology" is ... centering in the belief that  
(1) the present world order, regarded as both evil and oppressive, is under the temporary control of Satan and his human accomplices, and
(2) that this present evil world order will shortly be destroyed by God and replaced by a new and perfect order corresponding to Eden before the fall.  
During the present evil age, the people of God are an oppressed minority who fervently expect God, or his specially chosen agent the Messiah, to rescue them. The transition between the old and the new ages will be introduced with a final series of battles fought by the people of God against the human allies of Satan. The outcome is never in question, however, for the enemies of God are predestined for defeat and destruction. The inauguration of the new age will begin with the arrival of God or his accredited agent to judge the wicked and reward the righteous, and will be concluded by the re-creation or transformation of the earth and the heavens. This theological narrative characterized segments of early Judaism from ea. 200 BCE to ea. 200 CE"
Let's change some words:

  • 'World order' => information systems
  • 'Satan and his human accomplices' => evil hackers, APT!!1!
  • 'God' => well, I guess, it stays?
  • 'The people of God' => various infosec consultants, from Mandiant to the forthcoming 13 tribes of the US cyberdefense.

You get my drift. There is also an awesome document called The War Scroll, describing the final fight of Sons of Light and Sons of Darkness... then there are Gnostics... There is a PhD in this somewhere.

Infosec industry as pre-reformation Catholic church

No needs for extensive quotes here, Catholicism is a popular topic this week. Just one example: exorcisms.

A typical exorcism: a person was apparently possessed by an invisible evil spirit who caused all kind of trouble (Exorcist is a fine movie, watch it), then a licensed exorcist priest was called, who did strange things and expelled the immaterial possessor out of the victim's body, collected fee and told the victim to install a fountain with holy water, pray and not sin any more.

People sinned, confessed, sinned again...

The priests themselves were idealistic young men or hypocritical old farts who did not practice what they preached.

Substitution table:

  • 'Satan', as before => 'bad hackers'
  • 'possessed human' => 'infiltrated company'
  • 'exorcists' => security consultants, especially DFIR type
  • 'sins' => "bad" security practices
  • 'confession' => audit or pentest, perhaps.

I could go on but it's time to wrap up, since we ought to celebrate: according to Iranian sources, Habemus Papam has been elected the new pope