Jagged Thoughts | Dr. John Linwood Griffin

November 7, 2015

Every Eighteen Months, part 2: Career coaching

Filed under: Opinions,Work — JLG @ 5:42 PM

It can be an uphill battle to get personalized advice about your job performance and your career progress.  Earlier this week a friend and I texted about the challenge of extracting good feedback from management in our modern litigation-averse corporate America.

He noted that in his job he hasn’t been getting the feedback he’s been asking for:

“I like the idea of knowing how I’m actually doing.  I credit honest feedback at every stage of my life where I claim to have grown.”

I replied:

“Hard to get that from an employer.  Good to try to cultivate a mentor wherever you go—someone senior but not in your org chart—but even then I’ve never been able to get the kind of constructive criticism that I’d like from anyone other than same-level colleagues.”

So the trick is to find those mentors.  I appreciate constructive criticism from anybody willing to lob it my way, but I especially appreciate getting it from successful entrepreneurial types (S.E.T.) who are living their dream.

Along those lines I recently had a great discussion with another S.E.T. acquaintance about my career goals, plans, and aspirations.  I described how my long-term goal has been to found and lead my own company (preferably successfully, that second time around).  I explained that as a step in that direction, my current dream job would have me being an entrepreneurial engineer.  I noted that I’ve been seeking out jobs where I am able to come up with ideas, pursue the good ones, and (if they fail) come up with more ideas and pursue them—jobs where I can:

  • have a product vision unique in the industry,
  • execute independently on my vision by applying my (and my colleagues’) unique and most prominent skills, and my interesting background, toward realizing the vision, and
  • increase my employer’s profits by the boatload (not all ideas are big wins, but all big wins come from ideas).

Unfortunately, such jobs are few and far between.  While there are some great aspects of my current job (it’s challenging, fast-paced, exposing me to new-to-me parts of the business, and immersing me in new technologies), I’m not in a role in which I am expected to—or in which I or my peers have largely been able to—have a broad degree of product, business, and technical influence.  So I asked my S.E.T. friend:  How do I sell myself and my capability, to advance into more senior roles—what do I need to convey about myself, and to whom?

The answer was to talk with a career coach.

Great idea!  One I’d never considered before.  And via a friend I came across a good coach willing to squeeze me in for one session.  (Protip: If your prospective coach suggests that you meet over a slice of pizza and a glass of beer, you know you’ve found a good one.)

Here were my takeaways from my chat with a career coach:

Do I need more experience in startups to do another startup?

He suggests not.  If I have an idea and can grab people and start pursuing it, the sink-or-swim-myself model is apparently just as good as watching someone else sink or swim.

He was definitely big on the idea of me going to startup(s) though.  He noted that if I feel that my career has stalled that’s because it has stalled, and at mid-sized companies filled with relatively young people there just aren’t going to be opportunities that open up for me for vertical movement.

He was also very cautionary to do my due diligence before going to a startup.  Know the founders, know whether they’re in good shape technically—are they hiring me to grow or to survive?

How do I market myself to land the opportunities I crave?

Change my resume from a laundry list of things I’ve done to an expression of who I want to be.  De-technicalize it, replacing the jargon with evidence of leadership—in general, present myself in my capacity to lead.  Clarify through the top-of-page-1 material that I’m looking to be considered for executive leadership, and that (separately) I have the credentials of coming from a solid technical background.

As an example, if I’m being evaluated for a CTO role then a CEO is going to look at whether I led a team through something instead of just the individual things I’ve done.  Don’t make the reader search for it, put it clearly and up front.  If you want to be an executive you need to tell the reader that you’re an executive.

Simultaneously update LinkedIn profile to be a resume supplement, focusing on brief powerful statements in the “summary” section at the top as an attention-grabber.  Also create an AngelList profile.

What might I get out of business school?  (The coach works frequently with current and former business school students, so it seemed apropos to ask.)

If I want to inject speed into an executive career path—moving through a career at a faster pace than I could expect by simply working through promotions—business school can provide that.  But don’t look at school as all you need or as the last piece of the puzzle; you’re being taught in the classroom a lot of what you’ll have learned in the workplace.

There are too many B-school graduates already and they’re mostly in their mid-20s.  It’s less of a unique credential/ticket than it used to be.

He espouses a go-big-or-go-home philosophy: If you’re gonna go to business school, and if you can afford it, go full time so that you get an intense, immersive, fast-paced experience.

What are my next steps?

Make an effort to land that next job that gets you on the right track to find the position you want.  Look for director or VP roles.  Don’t move horizontally.

Always be job hunting; never settle down.  Submit your resume early and often.  Write your CV and cover letter in such a way that you can recycle them in 5 minutes for any new opportunity you see.

(When asked about whether I should try to find a longer-term coach to work with.)  Don’t worry about finding a regular coach for now.  He doesn’t know anyone who does coaching on the side (except coaching for C-level executives) but in any event he feels that being in the game (circulating, getting interviews under my belt) would be more useful for me than being on the sideline (talking to a coach).

Overall a great conversation.  The main takeaway matches something I’ve heard before (though it’s hard to do):  Market yourself by presenting your past in the context of the job you want, not necessarily by conveying the minutiae of the job you had.  Describe your previous work by highlighting the things you did that you did well, that you got something out of, that you enjoyed, and that are most relevant to the position you seek.

April 19, 2015

An open letter to the Women’s Flat Track Derby Association

Filed under: Opinions — JLG @ 9:08 PM

In one of my all-time favorite newspaper articles, the author asserts:

 “Everyone should listen to what I have to say and heed my advice because I am correct … One of the things I like best about myself is that not only do I know what is best for everyone, I always make sure to come forward with this information.” (Edwin Wiersbicki, “I Know What Is Best For Everyone“, The Onion 34(12): 21 Oct 1998)

Following in Mr. Wiersbicki’s bold example, I myself have a few suggestions to offer to the Women’s Flat Track Derby Association (WFTDA)—the international governing body of women’s flat track roller derby—even though

  1. I am not a woman,
  2. I have never participated in a roller derby game,
  3. I haven’t even been on quad roller skates in probably 20 years, and
  4. I wasn’t really clear on the concepts and principles behind flat track roller derby until I watched a game last night.

(For example, it didn’t register to me flat track roller derby is played on a flat track, unlike the delightful banked-track sport a young John found himself glued to on television on Saturdays in 1989.)

I’d been wanting to attend a Boston Derby Dames match for the past two years, ever since I heard that Boston had an active roller derby team. And last night I did! It was a doubleheader, featuring a B-league matchup (Boston B-Party vs. Charm City Female Trouble) and the main event (Boston Massacre vs. Charm City All-Stars).

Charm City Female Trouble at Boston B-Party, April 18, 2015

Charm City Female Trouble at Boston B-Party, April 18, 2015

The skater names these athletes choose for themselves are hilarious. A few that I texted to Evelyn: “Boston Creamer”, “M.C. Slammer”, “Allie B. Back”, “I. M. Pain”. Even the officials were in on the fun: “Buxom Melons”, “TestosteRon Jeremy”.

As a novice spectator it was pretty hard to follow along with the action until two former skaters sitting near me (one named “Jodie Faster”) patiently explained what was going on. The key principles:

  • A 60-minute game is made up of a series of 2-minute “jams”.
  • A jam is 5-on-5 skating in a counterclockwise direction.
  • In a jam, each team fields one “jammer” (similar to a football quarterback) and four blockers.
  • Jammers score points by lapping members of the other team.
  • Blockers try to (a) stop the other jammer from lapping them, and (b) disrupt the other blockers so that their jammer can make progress. Skaters are not allowed to grasp with their hands, so hip checks are common.
  • It’s not (much) violence! Rather, strategy and agility are important. Brute force only gets you so far.

That last bullet was my favorite part of watching the games: As the game wore on, and the blockers became more familiar with the jammers’ jukes, and the skaters collectively grew more tired, and more skaters got sent to the penalty box (yielding 5-on-4 or 5-on-3 skating), it was very interesting to watch the teams shift their strategies—switching up personnel to get more favorable matchups; lining up differently at the start of a jam; inducing the pack of blockers to move more quickly or more slowly around the rink; trapping a skater on one side of the track and forcing her out of bounds; making use of a quirky rule to switch which skater is declared the jammer in the middle of a jam.

Protip:  In order to notice these things it was very helpful to be sitting next to Jodie and Brittany; I highly recommend that anyone new to roller derby make it a point to sit next to people who look like former or current skaters.

An interesting aspect of the roller derby leagues [according to my guides] is that it takes up about 20 hours a week for each of the team members (on top of their full time jobs needed to pay for skating equipment)—not just practice time, but also business activities such as reserving venues or merchandising or soliciting sponsors. As an away game, the teams from Baltimore likely drove themselves up I-95 and stayed at skaters’ houses to save money. Somehow knowing that each skater had an integral role in the functioning of the team and the league (instead of, say, a small powerful board taking care of such details) made the skating more fun to watch.

The games were held at the nifty Shriners Auditorium in Wilmington. $16 for a general admission ticket plus two $4 cheeseburgers and a $5 Shock Top beer (all served by genuine Shriners!) worked out to about $7.25 per hour of entertainment. Pretty good value, especially since the A-teams turned out to be well matched.  Both games featured several lead changes, and the major game stayed nail-bitingly close into the final minute—generating thunderous excitement from the audience of about 700. (The audience seemed to consist mostly of current skaters, former skaters, friends of skaters, or partners of skaters.)

I had a good time, though I doubt I’d go back to see another game. There are two perplexing rules whose enforcement took away a big part of my enjoyment of the games. One rule imposes a grossly unfair penalty for infractions by a certain class of skater (a violation of the Equal Protection Clause). The other rule flushes down the toilet almost about 75% of the opportunities for skaters to demonstrate strategy and agility (“my favorite part of watching the games”, as described above). These two rules vexed me so much that they inspired me to write this blog post to suggest that the rules be changed.

"Think Logically" by Randall Munroe. http://xkcd.com/1112/

“Think Logically” by Randall Munroe. http://xkcd.com/1112/

So, to the WFTDA, here are my three unsolicited recommendations:

  1. Eliminate penalties that send jammers to the penalty box (6.1.1). Each time you send off a jammer it’s a blank check for the other team to score 15 to 20 unanswered points. Baltimore lost by 14 points last night, in a game where three seemingly bogus penalties on Baltimore jammers netted about 45 total points for Boston.

The disparity in impact between a blocker getting a penalty (perhaps 3 points) and a jammer getting a penalty (15+ points)—for the same infraction—is absurd. This rule should be changed. Perhaps a jammer’s penalty should be immediately served by a blocker, as is done with penalties against goalies in hockey? (I note that other writers have proposed other rule changes for jammer penalty enforcement.)

Certainly other sports have asymmetric positional rule impact. For example, in this article Gregg Easterbrook complains about asymmetric football rules such as holding, where offensive holding is a 10-yard penalty and replay the down, whereas defensive holding is a 10-yard penalty and an automatic first down:

Under current rules, the defense is penalized more than the offense for the same foul. Let’s make defensive and offensive holding equivalent.

But back to roller derby, the rule to send off jammers for penalties is one of the worst rules I’ve ever seen in sports.

  1. Limit (or eliminate) the concept of Lead Jammer calling off the jam (2.4.7). It is truly dull from an audience perspective to watch the jammer make one loop of the track, pass the blockers, and immediately call off the jam for a ho-hum 4-0 score. You’re wasting 90 seconds of potentially audience-stimulating action!

Yes, the current rule makes it valuable to be the Lead Jammer, but there are many ways that the rules could be improved to preserve the value of being the first jammer out of the gate:

  • Teams could be limited in the number of times per half that a jammer could call off the jam, making it a strategic (instead of tactical) decision for when to call off the jam.
  •  The Lead Jammer (or perhaps just the jammer currently in the lead) could receive a bonus point every time she laps the pack, or bonus points just for being the Lead Jammer (cf. NASCAR points for leading the most laps in a race).
  • The Lead Jammer could accumulate more than 1.0 point for each blocker passed, or could receive another scoring modifier.
  • Jammers could only be eligible to call off the jam if in the lead under certain conditions (for example in every other jam [cf. vulnerability in Bridge], or only when trailing in score, or only in the first half of the period).

I felt this rule was just plain dumb. For the first 20 minutes of action I couldn’t figure out why the jams were ending with time still left on the jam clock. Once it was explained to me, I couldn’t fathom why the rules authority would go out of its way to make the game boring to its audience.

  1. Eliminate some of the 30-second delays between jams (1.5.3). Instead of having a new set of personnel out to line up start ever discrete jam, consider having a continuous “extended jam” concept where winded skaters are allowed to be replaced with fresh skaters while the action continues around them (as in relay speed skating, or hockey, or tag-team wrestling).

The personnel changes could happen continuously (for example by a skater tagging out with another skater in the clearance area outside the track); or they could happen on a referee’s whistle at 2-minute intervals. Scoring could continue with whichever skater is currently wearing the star helmet. Perhaps the last 10 minutes of each period could be skated under the “extended jam” concept?

(This rule isn’t something that vexed me, but I feel such a change would be interesting to try and could make the games more interesting for players and spectators alike.)

Anyone from the WFTDA Rules Committee is welcome to contact me (see instructions on this page); I would be happy to elaborate on any of the above three points.

April 5, 2015

Every Eighteen Months, part 1: First steps

Filed under: Opinions — JLG @ 7:05 PM

Imagine, if you will, a much younger John.

Ten years ago, almost exactly to this day, I stood on the rain-slick precipice of darkness, scrying into the future to decide what’s next? for my life.

I had spent the past six years as a student in the best storage systems research laboratory in the world, itself part of the #1-ranked computer engineering program in the world. I’d learned how to think in graduate school—how to explore the question why?—how to look at trees and see the forest. I’d gained confidence: confidence to formulate and pursue a technical hypothesis, confidence to take the risk that a line of investigation might be wrong, confidence to lead and advise others in the pursuit of their personal goals.

I had spent most of those six years assuming that I would follow in the footsteps of my thesis advisor—become a tenure-track faculty member, set a visionary research agenda and rally a large team to sponsor and drive the work, change the world through my research results, and nurture my own set of wide-eyed/wet-behind-the-ears students into the change-the-world visionaries of the future.

I had a wonderful girlfriend ready to help me succeed in my career, supportive colleagues and contacts in academia and industry, a solid record of publication and public speaking, a reasonably compelling vision for my future research agenda, and an eagerness to move somewhere new and experience/travel/enjoy being there.

I had my choice of off-the-hook amazing job offers in amazing places. In academia, both the University of Edinburgh (Scotland) and the University of Waterloo (Canada) offered me the academic positions I’d dreamed about. I’d also interviewed at world-class industrial and business firms—including Google, Hitachi, Intel, McKinsey & Co., and Microsoft Research—and had received offers to work directly with two of the most impressive people I’d ever met (John Colgrove at Veritas Software near San Francisco, or Dr. Leendert van Doorn at IBM Research near New York City).

I was the king of the world!

But then I asked my advisor a question he couldn’t answer.


I asked for my advisor’s advice on how to choose which job to take—what to consider in terms of internal and external impact, work-life balance, and personal satisfaction; what I could expect over the first few years of my career; what were my personal strengths and weaknesses that would be unique (or be problematic) in each position; how each choice would tie in with my long-term career and life plans.

He answered that he couldn’t advise me. He couldn’t because he had only ever worked in academia; he didn’t have the experience to help me understand what I could expect; to suggest which choices would open or close which doors; to guide me toward the right decision for me.

That answer stunned me. It opened my eyes that I would have the same limitation if I went straight into an academic position.

It was the mentorship aspect of a job in academia—the “confidence to lead and advise others in the pursuit of their personal goals” and the “nurture my own set of wide-eyed/wet-behind-the-ears students into the change-the-world visionaries of the future”—that was the key draw to my pursuing such a position. From this conversation with my advisor I realized that I needed more personal work experience in order to pay forward the gifts of encouragement, nurturing, guidance, and freedom that I’d received from my teachers and mentors over the years.

Moreover, if I was going to change the world I realized that I needed a better understanding how the real world works. At Carnegie Mellon I saw firsthand how many successful faculty members pursued technology transition: Some, like my advisor, built relationships with key technology companies in his field and freely shared results (and freely shared graduate students for internships) to ensure a steady flow of ideas and technology both into and out of his group. Some, like Dr. Phil Koopman, applied his years of industry experience toward choosing projects that were both academically rigorous and immediately practical for solving the complex problems that had vexed him but that industry couldn’t (or wouldn’t) solve for itself. And some, like Dr. Garth Gibson, took the flat-out entrepreneurial approach and founded a company to productize the game-changing technology concept he’d conceived, defined, explored, and proselytized over the past decade.

So I decided not to pursue a professorship, at least not for a while, while I instead went out to discover how the real world works.

(Saying “no” to the position at Edinburgh is the hardest phone call I’ve ever made.)

I accepted the position at IBM, and the next decade turned out to be far more interesting (and far more interestingly chaotic) than I imagined they would be.

[Author’s note: I wrote this text on May 10, 2014.]

November 30, 2014

How to Peer Review

Filed under: Opinions,Reviews — JLG @ 8:25 AM

I tried to title this blog post “7 things to keep in mind during peer review (#5 will shock you)” but just couldn’t bring myself to do it.

It’s paper review season; I’ve been working on reviewing papers for a conference for which a former IBM colleague invited me to be on the program committee.  Then this morning a long-time reader of this blog submitted a question:

First time on an academic CFP review team. Have the papers. Any tips for what to do?

Do I ever.  Here’s my advice:

  1. Don’t wait until the last minute to do your reviews.  I try to do at least 1/day so that I make progress and am able to give each paper enough time to do a good job.
  2. The words of the day are ‘constructive criticism’. Especially for papers you grade as ‘reject’, give the authors suggestions that, if adopted, would have moved the paper towards the ‘accept’ column.
  3. Don’t be biased by poor English skills. Judge the merit of the ideas. If needed, the program committee chair can assign a ‘shepherd’ to work with the authors to improve the phrasing and presentation. (There is usually a way for you to provide reviewer comments that are not given to the authors — you can flatly say “if we accept this paper then such-and-such must be fixed before publication.”
  4. Be fair, in that it’s easy to say “well that’s obvious” and assign a paper a low score. Was it obvious *before* you read the paper? Every manuscript doesn’t have to be groundbreaking. Rather, every paper should advance the community’s understanding of important issues/concepts in a way that can be externally validated and built upon in future work.
  5. The Golden Rule applies: I try to give the kind of feedback (or: do the kind of thoughtful evaluation) that I wished other reviewers did on the papers I submitted.  That doesn’t mean I accept everything, of course; historically speaking I think I recommend ‘accept’ for only about 20% of papers.
  6. If you are working with junior academic types it can be a good idea to farm out a paper or two to them, both to give them experience/exposure and to give you sometimes a better evaluation of the paper than you might have done yourself. Be sure to convey the importance of confidentiality and ethics (not stealing unpublished work). Also be sure to read the paper yourself, and file your own review if there are important points not stated by your ‘external reviewer’.
  7. Peer review is part of the fuel that makes our scientific engines work. I’ve always thought of it as an honor to be asked to review a paper (and a *big* honor the few times I’ve been asked to be on a Program Committee). So I try to deliver reviews that ‘feed the scientific engines’, so to speak.

August 8, 2013

KB1YBA/AE, or How I Spent My Weekend In Vegas

Filed under: Opinions,Reviews — JLG @ 8:15 PM

Last week I attended Black Hat USA 2013, BSidesLV 2013, and DEF CON 21 in fabulous Las Vegas, Nevada, thanks to the generosity of my employer offering to underwrite the trip.  Here were the top six topics of discussion from the weekend:

1. Not surprisingly, PRISM was the main topic of conversation all weekend.  Depending on your perspective, PRISM is either

“an internal government computer system used to facilitate the government’s statutorily authorized collection of foreign intelligence information from electronic communication service providers under court supervision” (Director of National Intelligence statement, June 8, 2013)

or

“a surveillance program under which the National Security Agency [NSA] vacuums up information about every phone call placed within, from, or to the United States [and where] the program violates the First Amendment rights of free speech and association as well as the right of privacy protected by the Fourth Amendment” (American Civil Liberties Union statement, June 11, 2013)

The NSA director, Gen. Keith Alexander, used the opening keynote at Black Hat to explain his agency’s approach to executing the authorities granted by Section 215 of the USA PATRIOT act and Section 702 of the Foreign Intelligence Surveillance Act.  His key points were:

  • The Foreign Intelligence Surveillance Court (FISC) does not rubber-stamp decisions, but rather is staffed with deeply-experienced federal judges who take their responsibilities seriously and who execute their oversight thoroughly.  Along similar lines, Gen. Alexander stated that he himself has read the Constitution and relevant federal law, that he has given testimony both at FISC hearings and at Congressional oversight hearings, and that he is completely satisfied that the NSA is acting within the spirit and the letter of the law.
  • Members of the U.S. Senate, as well as other executive branch agencies, have audited (and will continue to audit) the NSA’s use of data collected under Section 215 and Section 702.  These audits have not found any misuse of the collected data.  He offered that point as a rebuttal to the argument that the Government can abuse its collection capability—i.e., the audits show that the Government is not abusing the capability.
  • Records collected under Section 215 and Section 702 are clearly marked to indicate the statutory authority under which it is collected; this indication is shown on screen (a “source” field for the record) whenever the records are displayed.  Only specially trained and tested operators at the NSA are allowed to see the records, and that only a small number of NSA employees are in this category.  The collected data are not shared wholesale with other Government agencies but rather are shared on a case-by-case basis.
  • The NSA has been charged with (a) preventing terrorism and (b) protecting U.S. civil liberties.  If anyone can think of a better way of pursuing these goals, they are encouraged to share their suggestions at ideas@nsa.gov.

In the end I was not convinced by Gen. Alexander’s arguments (nor, anecdotally speaking, was any attendee I met at either Black Hat or DEF CON).  I walked away from the keynote feeling that the NSA’s collection of data is an indiscriminate Government surveillance program, executed under a dangerous and unnecessary veil of secrecy, with dubious controls in place to prevent abuse of the collected data that, if abused, would lead to violations of civil rights of U.S. citizens.  In particular, if this program had existed on September 11, 2001, I harbor no doubt that the statutory limits (use or visibility of collected data) would have been exceeded in the legislative and executive overreaction to the attacks.  This forbidden fruit is just too ripe and juicy.  As such I believe the Section 215 and Section 702 statutory limits will inexorably be exceeded if these programs—i.e., the regular exercise of the federal Government’s technical capability to indiscriminately collect corporate business records about citizen activities—continue to exist.

I do appreciate how the NSA is soliciting input from the community on how the NSA could better accomplish its antiterrorism directive.  Unfortunately, Pandora’s Box is already open; I can’t help but feel disappointed that my Government chose secretly to “vacuum up information” as its first-stab approach to satisfying the antiterrorism directive.  As I wrote in a comment on the Transportation Security Administration (TSA)’s proposed rule to allow the use of millimeter wave scanning in passenger screening:

I fly monthly.  Every time I fly, I opt out of the [millimeter wave] scanning, and thus I have no choice but to be patted down.  I shouldn’t have to submit to either.  In my opinion and in my experience, the TSA’s intrusive searching of my person without probable cause is unconstitutional, period.

I appreciate that the TSA feels that they’re between a rock and a hard place in responding to their Congressional directive, but eroding civil liberties for U.S. citizens is not the answer to the TSA’s conundrum.  Do better.

Eroding civil liberties for U.S. citizens is not the answer to the NSA’s conundrum.  Do better.

2. Distributed Denial of Service (DDoS) attacks.  There were at least five Black Hat talks principally about DDoS, including one from Matthew Prince on how his company handled a three hundred gigabit per second attack against a customer.  The story at that link is well worth reading.  Prince’s talk was frightening in that he forecast how next year we will be discussing 3 terabit/sec, or perhaps 30 terabit/sec, attacks that the Internet will struggle to counter. The DDoS attacks his company encountered required both a misconfigured DNS server (an open DNS resolver that is able to contribute to a DNS amplification attack; there are over 28 million such servers on the Internet) and a misconfigured network (one that does not prevent source address spoofing; Prince reports there are many such networks on the Internet).

Another interesting DDoS talk was Million Browser Botnet by Jeremiah Grossman and Matt Johansen.  The essence is that you write your botnet code in Javascript, then buy Javascript ads on websites…resulting in each reader of those websites becoming a node in your personal botnet (as long as they’re displaying the page).  Wow.

3. CreepyDOL.  To quote the Ars Technica article about this work: “You may not know it, but the smartphone in your pocket is spilling some of your deepest secrets to anyone who takes the time to listen. It knows what time you left the bar last night, the number of times per day you take a cappuccino break, and even the dating website you use. And because the information is leaked in dribs and drabs, no one seems to notice. Until now.”  The researcher, Brendan O’Connor, stalked himself electronically to determine how much information an attacker could glean by stalking him electronically.  In his presentations he advanced a twofold point:

(a) it’s easy to track people when they enable wifi or Bluetooth on their phones (since it sprays out its MAC address in trying to connect over those protocols), and

(b) many services (dating websites, weather apps, apple imessage registration, etc.) leak information in clear text that can be correlated with the MAC address to figure out who’s using a particular device.

O’Connor’s aha! moment was that he created a $50 device that can do this tracking.  You could put 100 of these around the city and do a pretty good job of figuring out where a person of interest is and/or where that person goes, for relatively cheap ($5,000) and with no need to submit official auditable requests through official channels.

The researcher also brought up an excellent point about how the chilling effect of Draconian laws like the Computer Fraud and Abuse Act (CFAA) make it impossible for legitimate computer security researchers to perform their beneficial-to-society function.  If the CFAA had existed in the physiomechanical domain in the 1960s then Ralph Nader could have faced time in federal prison for the research and exposition he presented in Unsafe At Any Speed: The Designed-In Dangers of The American Automobile—and consumers might never have benefitted from the decades of safety improvements illustrated in this video.  Should consumer risks in computer and network security systems should be treated any differently than consumer risks in automotive systems?  I’m especially curious to hear arguments from anybody who thinks “yes.”

4. Home automation (in)security.  In a forthcoming blog post I will describe the astonishingly expensive surprise expense we incurred this summer to replace the air conditioner at our house.  (“What’s this puddle under the furnace?” asked Evelyn.  “Oh, it’ll undoubtedly be a cheap and easy fix,” replied John.)

As part of the work we had a new “smart” thermostat installed to control the A/C and furnace.  The thing is a Linux-based touchscreen, and is amazing—I feel as though I will launch a space shuttle if I press the wrong button—and of course it is Wi-Fi enabled and comes with a service where I can control the temperature from a smartphone or from a web application.

And, of course, with great accessibility comes great security concerns.  Once the thermostat was up and running on the home network I did the usual security scans to see what services seemed to be available (short answer: TCP ports 22 and 9999).  Gawking at this new shuttle control panel got me interested in where the flaws might be in all these automation devices, and sure enough at Black Hat there were a variety of presentations on the vulnerabilities that can be introduced by consumer environmental automation systems:

Clearly, home automation and/or camera-enabled insecurity was a hot topic this year.  I was glad to see that the installation manual for our new thermostat (not camera-enabled, I think) emphasizes that it should be installed behind a home router’s firewall; it may even have checked during installation that it received an RFC 1918 private address to ensure that it wasn’t directly routable from the greater Internet.

5. Femtocells and responsible disclosureTwo years ago I wrote about research that demonstrated vulnerabilities in femtocells (a.k.a. microcells), the little cellular base stations you can plug into your Internet router to improve your cellular reception in dead zones.  This year, Doug DePerry and Tom Ritter continued the femtocell hacking tradition with a talk on how they got root access on the same device I use at home.  The researchers discovered an HDMI port on the bottom of the device, hidden under a sticker, and sussed out that it was actually an obfuscated USB port that provided console access.  Via this console they were able to modify the Linux kernel running on the device and capture unencrypted voice and SMS traffic.  The researchers demonstrated both capabilities live on stage, causing every attendee to nervously pull out and turn off their phones.  They closed by raising the interesting question of why any traffic exists unencrypted at the femtocell—why doesn’t the cellular device simply create an encrypted tunnel to a piece of trusted back-end infrastructure?  They also asked why deploy femtocells at all, instead of simply piggybacking an encrypted tunnel over ubiquitous Wi-Fi.

Regarding responsible disclosure, the researchers notified Verizon in December 2012 about the vulnerability.  Verizon immediately created a patch and pushed it out to all deployed femtocells, then gave the researchers a green light to give the talk as thanks for their responsible disclosure.  Several other presenters reported good experiences with having responsibly disclosed other vulnerabilities to other vendors, enough so that I felt it was a theme of this year’s conference.

6. Presentation of the Year:Adventures in Automotive Networks and Control Units” by Charlie Miller and Chris Valasek.  It turns out it’s possible to inject traffic through the OBD-II diagnostic port that can disable a vehicle’s brakes, stick the throttle wide open, and cause the steering wheel to swerve without any driver input.  Miller and Valasek showed videos of all these happening on a Ford Escape and a Toyota Prius that they bought, took apart, and reverse engineered.  It’s only August but their work gets my vote for Security Result of 2013.  Read their 101-page technical report here.

Wow.

I mean, wow.  The Slashdot discussion of their work detailed crafty ways that this attack could literally be used to kill people.  The exposed risks are viscerally serious; Miller showed a picture from testing the brake-disabling command wherein he crashed uncontrolled through his garage, crushing a lawnmower and causing thousands of dollars of damage to the rear wall.  (In a Black Hat talk the day before, Out of Control: Demonstrating SCADA Device Exploitation, researchers Eric Forner and Brian Meixell provided an equally visceral demonstration of the risks of Internet-exposed and non-firewalled SCADA controllers by overflowing a real fluid tank using a real SCADA controller right on stage.  I for one look forward to this new age of security-of-physical-systems research where researchers viscerally demonstrate the insecurity of physical systems.)

Regardless, other than those gems I was unimpressed by this year’s Black Hat (overpriced and undergood) and felt “meh” about DEF CON (overcrowded and undernovel).  Earlier in the year I was on the fence about whether to attend the Vegas conferences, having been underwhelmed last year, which prompted my good friend and research partner Brendan to observe that if I had a specific complaint about these conferences then I should stop whining about it and instead do something to help fix the problem.  In that spirit I volunteered to be on the DEF CON “CFP review team,” in hopes that I could help shape the program and shepherd some of the talks.  Unfortunately I was not selected to participate (not at all surprising, since I work indirectly for The Man).

In my offer to volunteer I offered these specific suggestions toward improving DEF CON, many of which are equally relevant to improving Black Hat:

I’d like to see the DEFCON review committee take on more of a “shepherding” role, as is done with some academic security conferences — i.e., providing detailed constructive feedback to the authors, and potentially working with them one-on-one in suggesting edits to presentations or associated whitepapers.

I think there are things the hacker community can learn from the academic community, such as:

* You have to answer the core question of why the audience should care about your work and its implications.

* It’s one thing to present a cool demo; it’s another to organize and convey enough information that others can build upon your work.

* It only strengthens your work if you describe others’ related work and explain the similarities and differences in your approach and results.

Of course there are plenty of things the academic community can learn from the hacker community!  I’m not proposing to swoop in with a big broom and try to change the process that’s been working fine for DEFCON for decades.  In fact I’m curious to experience how the hacker community selects its talks, so I can more effectively share that information with the academic community.  (For example I spoke at last year’s USENIX Association board meeting on the differences between the USENIX annual technical conference and events like DEFCON, Shmoocon, and HOPE, and I commented on lessons USENIX could take away from hacker cons.)

But each year I’ve been disappointed at how little of a “lasting impression” I’ve taken away from almost all of the DEFCON talks.  A “good presentation” makes me think about how *I* should change the way I approach my projects, computer systems, or external advocacy. I wish more DEFCON talks (and, frankly, Black Hat talks) were “good presentations.”  I’m willing to contribute my effort to your committee to help the community get there.

One academic idea you might be able to leverage is that whenever you’re published, you’re expected to serve on program committees (or as a reviewer) in future years.  (The list of PC members is usually released at the same time as the RFP, and the full list of PC members and reviewers are included in the conference program, so it’s both a service activity and resume fodder for those who participate.)  So perhaps you could start promulgating the idea that published authors at BH and DC are expected to do service activities (in the form of CFP review team membership) for future conferences.

Finally, KB1YBA/AE in the title of this post refers to the culmination of a goal I set for myself eighteen years ago.  There are three levels of achievement (three license classes) that you can attain as a U.S. ham radio enthusiast:

  • Technician class.  Imagine, if you will, a much younger John.  In 1995, at the urging of my old friend K4LLA, I earned technician (technically “technician plus”) class radiotelephone privileges by passing both a written exam and a 5-words-per-minute Morse code transcription exam.  [Morse code exams are no longer required to participate in amateur radio at any level in the United States.]  At that time I set a goal for myself that someday I would pass the highest-level (extra class) exam.
  • General class.  In 2011, at the urging of my young friend K3QB, I passed the written exam to earn general class privileges.  With each class upgrade you are allowed to transmit on a broader range of frequencies.  With this upgrade I was assigned the new call sign KB1YBA by the Federal Communications Commission.
  • Amateur Extra class.  At DEF CON this year, with the encouragement of my former colleague NK1B, I passed the final written exam and earned extra class privileges.  It will take a few weeks before the FCC assigns me a new call sign, but in the meantime I am allowed to transmit on extra-class frequencies by appending an /AE suffix onto my general-class call sign when I identify myself on-air.  For example: “CQ, CQ, this is KB1YBA temporary AE calling CQ on [frequency].”

I don’t mean to toot my own horn, but it feels pretty dang good to fulfill a goal I’ve held for half my life.  Only 17% [about 120,000] of U.S. ham radio operators have earned Amateur Extra class privileges.

April 8, 2013

A survey of published attacks against M2M devices

Filed under: Opinions,Work — JLG @ 12:00 AM

Last year I became interested in working with M2M (machine to machine) systems.  M2M is the simple idea of two computers communicating directly with each other without a human in the loop.

As an example of M2M, consider a so-called smart utility meter that is able both to transmit load information in real time to a server at the power company, and to receive command and control instructions in return from the server.  (The actual communications could take place over a cellular network, a powerline network, or perhaps even over the Internet using a telephony or broadband connection.)  An excerpt from that Wikipedia article demonstrates the types of new functionality that are enabled through real-time bidirectional communications with utility meters:

The system [an Italian smart meter deployment] provides a wide range of advanced features, including the ability to remotely turn power on or off to a customer, read usage information from a meter, detect a service outage, change the maximum amount of electricity that a customer may demand at any time, detect unauthorized use of electricity and remotely shut it off, and remotely change the meter’s billing plan from credit to prepay, as well as from flat-rate to multi-tariff.

Of course, with great power comes great opportunities for circumventing the security measures engineered into M2M components.  In an environment where devices are deployed for years, where device firmware can be difficult to update, and where devices are often unattended and not physically well secured—meaning potential attackers may have complete physical access to your hardware—it can be very challenging to implement low-impact, cost-effective protections.

Responding to this challenge, several researchers have given presentations or released papers that describe fascinating attacks against the security components of M2M systems.  In one well-known example, Barnaby Jack explained the technical details behind several attacks he created that reprogram Automated Teller Machines (and demonstrated live attacks against two real ATMs on stage) in a presentation at the Black Hat USA 2010 security conference.  In another, Jerome Radcliffe described at Black Hat USA 2011 how he reverse engineered the communication protocols that are used to configure an insulin pump and to report glucose measurements to the pump.

In reviewing these published attacks, I’ve developed a threefold taxonomy to help M2M engineers consider and mitigate related risks to their security architectures they develop.  In each category I list three examples of published attacks:

1. Attacks against M2M devices

A. Use a programming or debugging interface to read or reprogram a device.
B. Extract information from the device by examining buses or individual components.
C. Replace or bypass hardware or software pieces on the device in order to circumvent policy. 

2. Attacks against M2M services

A. Inject false traffic into the M2M network in order to induce a desired action.
B. Analyze traffic from the M2M network to violate confidentiality or user protection.
C. Modify component operation to fraudulently receive legitimate M2M services. 

3. Attacks against M2M infrastructure

A. Extract subscriber information from M2M infrastructure control systems.
B. Identify and map M2M network components and services.
C. Execute denial-of-service (DoS) attacks against infrastructure or routing components.

I’ve written a whitepaper that explores the technical details of three published attacks in the first category:  A survey of published attacks against machine-to-machine devices, services, and infrastructure—Part 1: Devices.  (TCS intends to publish parts 2 and 3 later this year, covering attacks against M2M services and infrastructure.)

My goal with the whitepapers is to illustrate the hacker methodology—the clever, creative, and patient techniques an adversary may use to attack, bypass, or circumvent your M2M security infrastructure.  (As a side note, I am grateful to the M2M security researchers and hackers who have been willing to share their methodology and results publicly.)

The key takeaway is to think like an attacker! by preparing in advance for when and how security systems fail.  A Maginot Line strategy for M2M may not be effective in the long term.  I often recommend such planning to include (a) a good security posture before you’re attacked, (b) good logging, auditing, and detection for when you’re attacked, and (c) a good forensics and remediation capability for after you’re attacked.

January 25, 2013

The 5 P’s of cybersecurity

Filed under: Opinions,Work — JLG @ 12:00 AM

Earlier this month I had the privilege of speaking at George Mason University’s cybersecurity innovation forum.  The venue was a “series of ten-minute presentations by cybersecurity experts and technology innovators from throughout the region. Presentations will be followed by a panel discussion with plenty of opportunity for discussion and discovery. The focus of the evening will be on cybersecurity innovations that address current and evolving challenges and have had a real, measurable impact.”

(How does one prepare for a 10-minute talk?  The Woodrow Wilson quote came to mind: “If I am to speak ten minutes, I need a week for preparation; if fifteen minutes, three days; if half an hour, two days; if an hour, I am ready now.”)

Given my experience with network security job training here at TCS, I decided to talk about the approach we take to prepare students for military cybersecurity missions.  It turned out to be a good choice:  The topic was well received by the audience and provided a nice complement to the other speakers’ subjects (botnet research, security governance, and security economics).

My talk had the tongue-in-cheek title The 5 P’s of cybersecurity: Preparing students for careers as cybersecurity practitioners.  I first learned of the 5 P’s from my college roommate who captained the Auburn University rowing team.  He used the 5 P’s (a reduction of the 7 P’s of the military) to motivate his team:

Poor Preparation = Piss Poor Performance

In the talk I asserted that this equation holds equally true for network security jobs as it does for rowing clubs.  A cybersecurity practitioner who is not well prepared—in particular who does not understand the “why” of things happening on their network—will perform neither effectively nor efficiently at their job.  And as with rowing, network security is often a team sport:  One ill-prepared team member will often drag down the rest of the team.

I mentioned how my colleagues at TCS (and many of our competitors and partners in the broad field of “advanced network security job training”) also believe in the equation, perhaps even moreso given that many of them are former or current practitioners themselves.  I have enjoyed working alongside instructors who are passionate about the importance of doing the best job they can.  Many subscribe to an axiom that my father originally used to describe his work as a high-school teacher:

“If my student has failed to learn, then I have failed to teach.”

After presenting this axiom I discussed several principles TCS has adopted to guide our advanced technical instruction, including:

  1. Create mission-derived course material with up-to-date exercises and tools.  We hire former military computer network operators to develop our course content, in part to ensure that what we teach in the classroom matches what’s currently being used in the field.  When new tools are published, or new attacks are put in the news, our content-creators immediately start modifying our course content—not simply to replace the old content with the new, but rather to highlight trends in the attack space & to involve students in speculating on what they will encounter in the future.
  2. Engage students with hands-on cyber exercises. Death by PowerPoint is useless for teaching technical skills.  Even worse for technical skills (in my opinion, not necessarily shared by TCS) is computer-based training (CBT).  Our Art of Exploitation training is effective because we mix brief instructor-led discussions with guided but open-ended hands-on exercises using real attacks and real defensive methodologies on real systems.  The only way to become a master programmer is to author a large and diverse series of software; the only way to become a master cybersecurity practitioner is to encounter scenarios, work through them, and be debriefed on your performance and what you overlooked.
  3. Training makes a practitioner better, and practitioners make training better.  A critical aspect of our training program is that our instructors aren’t simply instructors who teach fixed topics.  Our staff regularly rotate between jobs where they perform the cybersecurity mission—for example, by participating in our penetration test and our malicious software analysis teams—and jobs where they train the mission using the skills they maintain on the first job.  Between our mission-relevant instructors and our training environment set up to emulate on-the-job activities, our students experience in the classroom builds to what they will experience months later on the job.

The audience turned out to be mostly non-technical but I still threw in an example of the “why”-oriented questions that I’ve encouraged our instructors to ask:

The first half of an IPv6 address is like a ZIP code.  The address simply tells other Inetrnet computers where to deliver IPv6 messages.  So the IPv6 address/ZIP code for George Mason might be 12345.

Your IPv6 address is typically based on your Internet service provider (ISP)’s address.  In this example, George Mason’s ISP’s IPv6 address is 1234.  (Continuing the example, another business in Fairfax, Virginia, served by the same ISP might have address 12341; another might have 12342; et cetera.)

However, there is a special kind of address—a provider-independent address—that is not based on the ISP.  George Mason could request the provider-independent address 99999.  Under this scheme GMU would still use the same ISP (1234), they would just use an odd-duck address (99999 instead of 12345).

Question A:  Why is provider-independent addressing good for George Mason?

Question B:  Why is provider-independent addressing hard for the Internet to support?

Overall I had a great evening in Virginia and I am thankful to the staff at George Mason for having extended an invitation to speak.

December 14, 2012

I can’t keep on renaming my dog

Filed under: Opinions,Work — JLG @ 12:00 AM

A clever meme hit the Internet this week:

“Stop asking me to change my password. I can’t keep on renaming my dog.”

If you (or the employees you support) aren’t using a password manager, clear off your calendar for the rest of the day and use the time to set one up.  It’s easy security.  Password managers make it simple to create good passwords, to change your passwords when required, to use different passwords on every site, and to avoid reusing old passwords.

The upside to using a password manager:

  • You only need to remember two strong passwords.  (One to log into the computer running the password manager, and one for the password manager itself.)

The downside to using a password manager:

  • All your eggs are in one basket.  (Therefore you need to pay close attention to choosing a good master password, protecting that password, and backing up your stored passwords.)

Generally speaking a password manager works as follows:

  1. You provide the password manager with a master passphrase.
  2. The password manager uses your master passphrase to create (or read) an encrypted file that contains your passwords and other secrets.

(For deeper details, see KeePass’s FAQ for a brief technical explanation or Dashlane’s whitepaper for a detailed technical explanation.  For example, in the KeePass FAQ the authors describe how the KeePass product derives 256-bit Advanced Encryption Standard [AES] keys from a user’s master passphrase, how salt is used to protect against dictionary attacks, and how initialization vectors are used to protect multiple encrypted files against known-plaintext attacks.  Other products likely use a similar approach to deriving and protecting keys.)

Password managers often also perform useful convenience functions for you—inserting stored passwords into your web browser automatically; generating strong passwords of any desired length; checking your usernames against hacker-released lists of pwned websites; evaluating the strength of your existing passwords; leaping tall buildings in a single bound; etc.

The root of security with password managers is in protecting your master password.  There are three main considerations to this protection:

(A) Choose a good passphrase. 

I’m intentionally using the word “passphrase” instead of “password” to highlight the need to use strong, complex, high-entropy text as your passphrase.  (Read my guidance about strong passphrases in TCS’s Better Passwords, Usable Security whitepaper.  Or if you don’t read that whitepaper, at least read this webcomic.)

Your master passphrase should be stronger than any password you’re currently using—stronger than what your bank requires, stronger than what your employer requires.  (However, it shouldn’t be onerously long—you need to memorize it, you will need to type it every day, and you will likely need to type it on mobile devices with cramped keyboards.)  I recommend a minimum of 16 characters for your master passphrase.

(Side note:  For similar reasons, another place where you should use stronger-than-elsewhere passphrases is with full-disk encryption products, such as TrueCrypt or FileVault, where you enter in a password at boot time that unlocks the disk’s encryption key.  As Microsoft’s #7 immutable law of security states, encrypted data is only as secure as its decryption key.)

(B) Don’t use your passphrase in unhygienic environments.

An interesting concept in computer security is the $5 wrench.  Attackers, like electricity, follow the path of least resistance.  If they’ve chosen you as their target, and if they aren’t able to use cryptographic hacking tools to obtain your passwords, then they’ll try other approaches—perhaps masquerading as an IT administrator and simply asking you for your password, or sending you a malicious email attachment to install a keylogger onto your computer, or hiding a pinhole spy camera in the light fixture above your desk.  So even with strong encryption you are still at risk to social engineering attacks targeting your passwords and password manager.

One way to reduce the risk of revealing your passphrase is to avoid typing it into computer systems over which you have neither control nor trust, such as systems in Internet cafes, or at airport kiosks, or at your Grandma Edna’s house.  To paraphrase public-service messages from the 1980s, when you give your passphrase to an untrusted computer you could be giving that passphrase to anyone who used that computer before you.

For situations where you simply must use a computer of dubious provenance—say, you’re on vacation, you take a wrong turn at Albuquerque, your wallet and laptop get stolen, and you have to use your password manager at an Internet cafe to get your credit card numbers and bank contact information—some password managers provide features like one time passwordsscreen keyboardsmultifactor authentication, and web-based access to help make lemonade out of life’s little lemons.

(C) Make regular backups of your encrypted file.

If you have a strong passphrase [(A)] and you keep your passphrase secret [(B)] then it doesn’t matter where copies of your encrypted file are stored.  The strong encryption means that your file won’t be susceptible to a brute-force or password-guessing attack even if an attacker obtains a copy of your file.  (Password management company LastPass had a possible security breach of their networks in 2011.  Even so, users with strong passphrases had “no reason to worry.”)  As such you are safely able to make backup copies of your encrypted file and to store those backups in a convenient manner.

Some password managers are designed to store your encrypted file on your local computer.  Other managers (notably LastPass) store your encrypted file on cloud servers managed by the same company, making it easier to synchronize the password file across all devices you use.  Still other managers integrate easily with third-party cloud storage providers (notably Dropbox) for synchronization across multiple devices, or support direct synchronization between two devices over a Wi-Fi network.  (In all remote-storage cases I’ve found, the file is always encrypted locally before any portion of the file is uploaded into the cloud.)

Whichever type of manager you use, be aware that that one file holds your only copy of all of your passwords—it is critical that you not lose access to the contents of the file.  Computers have crashed.  Password management companies have disappeared (Logaway launched on May 4, 2010, and ceased operations on February 2, 2012).  Cloud services havelost data and have experienced multi-day disruptions.  Protect yourself by regularly backing up your encrypted file, for example by copying it onto a USB dongle (whenever you change or add a password) or by printing a hard copy every month to stash in a safe deposit box.

If you maintain a strict separation between your home accounts and your work accounts—for example to keep your employer from snooping and obtaining your Facebook password—simply set up two password managers (one on your home laptop, the other on your work PC) using two unique passphrases as master keys.

Password manager software is easy to set up and use.  The biggest problem you’ll face is choosing from among the cornucopia of password managers.  A partial list I just compiled, in alphabetical order, includes: 1PasswordAnyPasswordAurora Password ManagerClipperzDataVaultDashlaneHandy PasswordKaspersky Password ManagerKeePassKeeper,LastPassNorton Identity SafeParanotic Password ManagerPassword AgentPassword SafePassword WalletPINsRoboFormSecret ServerSplashIDSticky PasswordTK8 Safe, and Universal Password Manager.  There is even a hardware-based password manager available.

Your top-level considerations in choosing a password manager are:

  1. Does it run on your particular OS or mobile device?  (Note that some password managers sometimes charge, or charge extra, to support synchronization with mobile devices.)
  2. Do you already use Dropbox on all your devices?  If not, consider a manager that provides its own cloud storage (LastPass, RoboForm, etc.)  If so, and only if you would prefer to manage your own encrypted file, choose a service that supports Dropbox (1Password, KeePass, etc.)

I don’t recommend or endorse any particular password manager.  I’ve started using one of the “premium” (paid) password managers and am astonished at how much better any of the managers are over what I’d been using before (an unencrypted manual text-file-based system that I’d hacked together last millennium).

November 20, 2012

Gabbing to the GAB

Filed under: Opinions,Work — JLG @ 12:00 AM

Earlier this month the (ISC)² U.S. Government Advisory Board (GAB) invited me to present my views and opinions to the board.  What a neat opportunity!

The GAB is a group of mostly federal agency Chief Information Security Officers (CISOs) or similar executives.  Officially it comprises “10-20 senior-level information security professionals in their respective region who advise (ISC)² on industry initiatives, policies, views, standards and concerns” and whose goals include offer deeper insights into the needs of the information security community and discuss matters of policy or initiatives that drive professional development.

In terms of content, in addition to discussing my previous work on storage systems with autonomous security functionality, I advanced three of my personal opinions:

  1. Before industry can develop the “cybersecurity workforce of the future” it needs to figure out how to calculate the return on investment (ROI) for IT/security administration.  I suggested a small initial effort to create an anonymized central database for security attacks and the real costs of those attacks.  If such a database was widely available at nominal cost (or free) then an IT department could report on the value of their actions over the past year: “we deployed such-and-such a protection tool, which blocks against this known attack that caused over $10M in losses to a similar organization.”  Notably, my suggested approach is constructive (“here’s what we prevented”) rather thannegative (“fear, uncertainty, and doubt / FUD”).  My point is that coming at the ROI problem from a positive perspective might be what makes it work.
  2. No technical staff member should be “just an instructor” or “just a developer.”  Staff hired primarily as technical instructors should (for example) be part of an operational rotation program to keep their skills and classroom examples fresh.  Likewise, developers/programmers/etc. should spend part of their time interacting with students, or developing new courseware, or working with the sales or marketing team, etc.  I brought up the 3M (15%) / Hewlett-Packard Labs (10%) / Google (20%) time model and noted that there’s no reason that a practical part-time project can’t also be revenue-generating; it just should be different (in terms of scope, experience, takeaways) from what the staff member does the rest of their time.  My point is that treating someone as “only” an engineer (developer, instructor, etc.) does a disservice not just to that person, but also to their colleagues and to their organization as a whole.
  3. How will industry provide the advanced “tip-of-the-spear” training of the future?  One curiosity of mine is how to provide on-the-job advanced training.  Why should your staff be expected to learn only when they’re in the classroom?  Imagine if you could provide your financial team with regular security conundrums — “who should be on the access control list (ACL) for this document?” — that you are able to generate, monitor, and control.  Immediately after they take an action (setting the ACL) then your security system provides them with positive reinforcement or constructive criticism as appropriate.  My point is that if your non-security-expert employees regularly deal with security-relevant problems on the job, then security will no longer be exceptional to your employees.

I had a blast speaking.  The GAB is a group of great folks and they kept me on my toes for most of an hour asking questions and debating points.  It’s not every day that you get to engage high-level decision makers with your own talking points, so my hope is that I gave them some interesting viewpoints to think about — and perhaps some new ideas on which to take action inside their own agencies and/or to advise the government.

October 15, 2012

Public-key cryptography & certificate chaining

Filed under: Opinions,Work — JLG @ 12:00 AM

Of the many marvelous Calvin and Hobbes cartoons by Bill Watterson, one of the most marvelous (and memorable) is The Horrendous Space Kablooie.  Quoth Calvin, “That’s the whole problem with science.  You’ve got a bunch of empiricists trying to describe things of unimaginable wonder.”

I feel the same way about X.509, the name of the international standard defining public key certificates.  X.509?  It’s sort of hard to take that seriously — “X.509” feels better suited as the name of an errant asteroid or perhaps a chemical formula for hair restoration.

But I digress.  X.509 digital certificates are exchanged when you create a “secure” connection on the Internet, for example when you read your webmail using HTTPS.  The exchange happens something like this:

  • Your computer:  Hi, I’m a client.
  • Webmail server:  Howdy, I’m a server.  Here’s my X.509 certificate, including the public key you’ll use in the next step.
  • Your computer:  Fabulous.  I’ve calculated new cryptographic information that we’ll use for this session, and I’ve encrypted it using your public key; here it is.
  • (Further traffic is encrypted using the session cryptographic information.)

Several things happen behind the scenes to provide you with security:

  1. Your computer authenticates the X.509 certificate(s) provided by the server.  It checks that the server uses the expected web address.  It also verifies that a trusted third party vouches for the certificate (by checking the digital signature included in the certificate).
  1. Your computer verifies that there is no “man in the middle” attack in progress.  It does this by ensuring that the server has the private key associated with its certificate.  It does this by encrypting the session cryptographic information with the server’s public key.  If the server didn’t have the private key then it wouldn’t be able to encrypt and decrypt any further traffic.

Unfortunately the system isn’t perfect.  The folks who programmed your web browser included a set of trusted root certificates with the browser.  Those root certificates were issued by well-known certificate authorities [CAs] such as Verisign and RSA.  If an attacker breaches security at either a root CA or an intermediate CA, as happened with the 2011 Comodo and DigiNotar attacks, then an attacker could silently insert himself into your “secure” connection.  Yikes!  Efforts like HTTPS Everywhere and Convergence are trying to address this problem.

Public-key cryptography is pretty neat.  When you use public-key cryptography you generate two keys, a public key (okay to give out to everyone) and a private key (not okay).  You can use the keys in two separate ways:

  • When someone wants to send you a private message, they can encrypt it using your public key.  The encrypted message can only be decrypted using your private key.
  • When you want to publish a message, you can encrypt (sign) it using your private key.  Anyone who has your public key can decrypt (validate) your message.

In a public key infrastructure, a root CA (say, Verisign) uses its private key to sign the public-key certificates of intermediate certificate authorities (say, Thawte).  The intermediate CAs then use their private key to sign the public-key certificates of their customers (say, www.google.com).  When you visit Google’s site using HTTPS, Google provides you both their certificate and Thawte’s certificate.  (The chained relationship Verisign-Thawte-Google is sometimes called the “chain of trust”.)  Your browser uses the certificates provided by Google, plus the Verisign root certificate (bundled with the browser), to verify that the chain of trust is unbroken.

[I use Google as the example here, since you can visit https://www.google.com and configure your browser to show the certificates that Google provides.  However, I have no knowledge of Google’s contractual relationship with Thawte.  My assertions below about Google are speculative, but the overall example is valid.]

Recently I was asked “We have been trying to understand Certificate Chaining and Self Signing.  Would a company [like Google] be allowed to purchase one certificate from a Certificate issuer like Verisign and then issue its own signed additional certificates for additional use?”

Great question!  (Where “great question” is defined as “um, I don’t know, let me check into that.”)  It turns out the answer is no, a company’s certificate(s) cannot be used to sign other certificates.

Using Google as an example, the principal reason is that neither Verisign nor Thawte let Google act as an “intermediate certificate authority.”  It’s (1) likely against the license agreement under which Thawte signed Google’s certificate, and (2) prohibited by metadata fields inside both Thawte’s certificate and Google’s certificate:

  • Google’s certificate is prohibited from signing other ones because of a flag inside the certificate metadata.  (Specifically, their Version 3 certificate has an Extension called Certificate Basic Constraints that has a flag Is not a Certificate Authority.)  And Google can’t modify their certificate to change this flag, because then signature validation would fail (your browser would detect that Google’s modified certificate doesn’t match the original certificate that Thawte signed).
  • Certificates signed by Thawte’s certificate cannot be used as Certificate Authorities (CAs) because of a flag inside Thawte’s certificate.  (Specifically, their Version 3 certificate has an Extension called Certificate Basic Constraints that has an field Maximum number of intermediate CAs that’s set to zero, meaning that no verification program should accept any certificates that we signed using their key.)

If your company needs to issue its own signed certificates, for example to protect your internal servers, it’s relatively easy to do.  All you have to do is run a program that generates a root certificate.  You would then be like Verisign in that you could issue and sign as many other certificates as you wanted.  (The down side of your “private PKI” is that none of your users’ browsers would initially recognize your root certificate as a valid certificate.  For example, anyone surfing to a web page protected by certificates you signed would get a big warning page every time, at least until they imported your root certificate’s signature to their trusted-certificates list.)

The article I found most helpful in digging up this answer is here:
http://unitstep.net/blog/2009/03/16/using-the-basic-constraints-extension-in-x509-v3-certificates-for-intermediate-cas/

(The full name of the X.509 standard is the far worse ITU-T Recommendation X.509: Information technology – Open systems interconnection – The Directory: Public-key and attribute certificate frameworks.  One name with four hyphens, two colons, and the hyphenated equivalent of comma splicing?  Clearly rigorous scientific work.)

Older Posts »