Ajit Gaddam spoke on "Securing Your Big Data Environment" at Black Hat.

He is a CISSP (Certified Information Systems Security Professional), VISA's Chief Security Architect, and a coauthor of Hadoop in Action 2. This talk largely focused on the world of Hadoop and how to deal with it, what some of the surprises may be, etc. When you engage with a vendor to get a Hadoop rollout, expect a vendor to add their own way of doing things into the mix with the Hadoop deployment to such a degree that another vendor won't be able to augment their work. Hadoop in and of itself is open source but what you will end up buying will likely not be truly open source. This is a variation of this making-people-pay-for-free-stuff trickery I suppose. Hadoop will, if worthy of the indulgence, contain sensitive data and needs to be approached carefully. Beyond your own ability to sleep easy at night you may be subject to regulatory compliance as if your ETL is moving cardholder data it may be in scope for a PCI audit. Plan your defenses around a potential category of attacker based upon who would likely attack you. Threat model your environment such that sensitive data can be broken off into one cluster (set of connected servers), for example, independent of non-sensitive data in another cluster, for example. Circling back to cardholder data, credit card numbers are useless by themselves if they fall into the wrong hands. Supporting data has to accompany them in order for the numbers to be worthwhile to a hacker so perhaps the numbers and the other data points should be kept in very different places. Understand how your end-to-end data flows, especially the ingress and egress methods from your big data cluster. Bake that into your threat model. Why is our data compressed with gzip? Storage is cheap. Just buy storage in lieu of compressing data. In cryptography, Format-Preserving Encryption (algorithmically tokenizing one credit card number as another credit card number) is still evolving and there are no real standards yet. NIST (The National Institute of Standards and Technology) has FFX (Format-preserving, Feistel-based encryption, with the X reflecting the multiple instantiations based upon the number of parameters handed in) and BPS (Eric Brier, Thomas Peyrin, Jacques Stern) as its finalists.