Don't promote your most talented technologists into people management. Promote them into technical leadership roles instead. You'll keep happy, engaged leaders and avoid the perils of the Peter Principle.

It's a common policy in hierarchical organizations to promote top performers into people management. Your best programmer comes up with great ideas, the team listens to her, she can fix anything. It's review time and what happens? She's offered a job in management. What are her new responsibilities? Paperwork, reviews, operational alignment summits, budgets. How much code is she writing now? Not much and she's probably miserable.

"I don't want to stop coding." — Almost every good programmer, ever.

A giant cup of I'm the fucking boss.
This was a Christmas present from a member of my team in 2014.

There's another alternative. Technical leadership. Here are some common tech leadership roles:

Lead Engineer
You're coding everyday and helping others to code better, too. You have deep experience with the tools and technologies, as well as the problem domain, and you use that to inform internal system design. You keep the code healthy.
Software Architect
You're designing the story arc of how this software fits into its problem domain. You have visibility into the long-term plans for the software, and strong technical skills to work with engineering to build something that keeps the software focused. You keep the software healthy.
CTO
The most senior architect. You know what every product line is aiming for. You know how it fits into the long-term story of the product and the industry. You share this insight with your architects and leads to give them vital context for making great choices. You keep the company healthy.

This is the intersection of technology and people.

People are at the heart of tech leadership roles. It's a socio-technical role, a mix of people and technology. Hacking on one and not the other will cause the performance of both to suffer. This is the critical glue missing from most engineering organizations where engineers are promoted to managers. This is where technical leadership plays its key role. You're not managing people: you're leading them.

Why is there a gap? Because newly-promoted managers now have too many jobs. Promoting a tech lead into a manager forces them into the problem of having too many roles. Their boss is expecting them to be a full-time people manager, but their team depends on their technical leadership. They'll either have to choose one or do both poorly, for lack of time, which will eventually force their hand into choosing people management. Again, because their boss is expecting them to do that role. Now there's a void in technical leadership.

In a healthy organization the tech lead (lets say a Principal Architect) and the people manager are a powerful pair. When they're working together the people on their team are taken care of at every level. The team is comfortable because they know what to build and why, and that they're building on the right path. They're also getting the individual attention of their manager.

Tech leadership is a force multiplier.

As a technical leader you aren't always writing code. If you're a recognized leader chances are you already know that. You spend a lot of time helping others. Teaching, mentoring, reviewing code, standing at the whiteboard and explaining the permissions model for the hundredth time. Or whatever your pet thing is…

You may not always be pressing many keys and causing an implementation to occur, but for the people who are you're a valuable asset. You help them do it better. You help them avoid the pitfalls and mistakes you've already made five times in your career.

This work is vital to the health of an organization. It has a direct impact on productivity and happiness. You are helping each team member avoid pain, and you're helping the organization avoid pain. How is this a force multiplier? The evidence is proven over time.

Applying the skill of technical leadership is a force multiplier in several ways. If any of the following apply to you, you might be a tech lead:

  • Your experience helps engineers at all skill levels create solutions that last longer, are easier to maintain, and respond to change better.
  • You've thought through integration problems so there are fewer surprises.
  • You understand the long-term goals for the software so your short-term solutions don't paint everyone into an architectural corner.
  • Your team is constantly learning from you so retention rates are high because everyone is growing.
  • When a customer is upset you can code up a solution which keeps the team focused on their planned work.
  • You've been planning ahead for the next big development effort — have five prototypes ready for comment by the time the team is ready — so nobody feels lost in the face of a new challenge.

This is how a technical leader is a force multiplier: enhancing the attributes of your organization to make it more effective than others. It's like skill boosts in role-playing games (be honest, if you're reading this you know exactly what I mean).

Lead by example.

Technical leadership is the role best suited for my skill set. I was fortunate enough to have been supported by a fantastic people manager at my current company. With support I used everything in my technical leadership toolbox to build teams of outstanding engineers, create a culture for them to thrive in, write the story of our next great architectural revolution, and lead the team down that path.

I love this work. I'm passionate about it. As a programmer who always wants to code, I don't get to do it as much as I'd like, but when when I do it's usually pairing with someone on my team. As a leader I get to work directly on the problems getting in the way of my team, very real people-problems, and resolve them. I do that with the support of people management.

One of the core values in my team culture is transparency. I'll exercise that by sharing a piece of my self evaluation from last fall. It's the part where I'm supposed to say what I ought to work on for the next year. This is a high-level view of my approach to technical leadership:

What should the focus be for the next review period?

I'd will continue to focus on the areas I feel are the greatest value to WhiteHat. This has been my focus for the last year and will remain so. Specifically:

Empower Others — I will continue to create space and opportunity for the members of my team, my organization, and anyone in the company to contribute at their best level.

Challenge Assumptions — I will continue to challenge the things we do, how we do them, and why we do them in a relentless effort to eliminate waste in our organization.

Eliminate Pain Points — I will continue to learn about others' work in order to find resolution for things that are difficult and painful in their daily work. I will empower my team to find ways to automate most of those pain points away, or challenge assumptions in our process and workflow which may have created them.

Influence Culture — I will continue to be the change I want to see at WhiteHat, and be myself with no bad politics or hidden agendas, in order to help create the culture of trust, learning, productivity, fun, and reward I want to work in; that I want to recruit my friends to work in.

Lead Technology and Architecture — I will continue to be a leader, mentor, and teacher to my team members and organization members. I will take an active role in leading the development and design of our software and operational architectures to ensure we make aggressive strides toward modern, sustainable solutions that meet our customers' needs.

By this time next year I will have been successful if the Engineering organization is delivering high quality software that delights our customers.

I expect my team to have delivered an infrastructure and web application framework that puts us in the position to aggressively compete in our market, to have delivered several key features which empower WhiteHat to make impressive advancements within our industry and in comparison to our key competitors.

We need a tech leadership track.

I'm sure you've heard this before: "we're going to put you on the management track." The management track is a career advancement concept designed to mold high performers into managers. Within engineering we need a new, parallel track. We need a technical leadership track.

Top performers who want to remain technical and advance their career should have the option. Organizations are in desperate need of highly qualified, deeply experienced leaders making strategic decisions for the health of their software. A technical leadership track is the best way to source that talent from within.

If you are a mid to senior level engineer right now, and you don't want to advance to management, I want to encourage you to advance to technical leadership instead. Insist on a promotion to Lead Engineer, then Architect, then Principal Architect.

If you run an engineering department and want to offer career advancement that keeps your top performers engaged and happy, don't force them into management. Create an advancement track designed to build a strong technical leadership pipeline. Your software will be better for it.

Find your best technologist and promote them to Director of Architecture. Your new Director should have people managers in her peer group. This creates space for the tech leaders in your organization to grow. Build your technical leadership team under her in the hierarchy, but let them keep working with the teams their leading in day-to-day work. Let them continue to do what they're great at. This is how you make technical leadership just as important as people management.

It's good to be back in the shire.
I am not a manager but I love this mug.

Update: We are no longer looking for new team members in Pittsburgh.

I work for WhiteHat Security and I lead a team called The Shire. Our team primarily builds web applications in Ruby on Rails and we're based in downtown Pittsburgh.

We're hiring a couple more developers to join our outstanding team. I'm fortunate enough to work with some of the best people in Pittsburgh. Bright, curious, hilarious, and they're good hackers1, too! We're interested in talking to programmers of all levels. Don't shy away if you are interested but have limited experience. We want to hire great people who put in the effort, and you are hard enough to find as it is.

It's difficult to write a "please come work with me" essay so here's my plan: I'm going to tell you about our team culture, then about what we're looking for in a candidate. The method I'm using is to post our internal team culture document verbatim, and our internal job position document. They're both on our team wiki, and editable by anyone on the team.

If you read all this and you'd like to work with us please email me to talk about it. You can reach me at work with the following obfuscated email address:

1
2
3
4
5
6
7
8
9
10
11
name    = %w(Casey West)
company = %q(WhiteHat Security)
email   = [
  name.join('.'),
  [
    company.tr(' ','')[0..10], # We go to 11. \o/
    'com'
  ].join('.')
].join('@').downcase

puts email

You can also find links to me on social media on the left if you'd like to chat casually.

Shire Culture

The Shire operates under two guiding principles: trust and transparency. As a member of the team you are trusted to use the best ideas to do the most important work. As a team we agree to be transparent in our work, our ideas, and our struggles.

This means we are all accountable for the success of our work as a team. It doesn't matter what functional role you have, which skills you bring to the team when you start, or where you are located. If there's work that needs to be done we take responsibility for it, complete it, and ship it as a team.

If we succeed in our goals it's because we did it together. If we fail it's because we all failed together. Thankfully, failure is not terminal. We learn from it, adapt, and do better going forward.

This brings us to The Shire Motto: If something is broken we fix it.

We don't put a ticket on another team's queue and wait around or complain. We do all the work we can to fix whatever is broken. We accept external roadblocks only when we are physically or administratively blocked, and we try to avoid those blockers in the future, too.

The Shire has adopted this attitude. Doing so has meant we've accumulated a wealth of knowledge about much of the WhiteHat Architecture. Everything from the hardware we run on to the way we document our APIs.

We are not afraid of solving hard problems correctly, and we're not afraid to spend the time getting things right. We’re not afraid to make good change anywhere in the company with whatever the right tool for the job is: technology, knowledge, experience, culture, or even a good hearty laugh.

With this understanding of our Shire culture it's clear to see what your role as a team member is: Get the job done. Do whatever it takes, and do it well.

We work at WhiteHat to meet customer needs with positive business outcomes.

Here are some things we don't work at WhiteHat to do:

  • Write beautiful code.
  • Test everything.
  • Make it pixel perfect.
  • Handle every edge case.
  • Document every process.

When we are doing well we get some of these for free, which is a wonderful thing, but they are not our primary objective.

We don't work beyond our physical limits. Each team member should work to maintain their health and happiness, and the team should hold each other accountable for that. If someone is burning out we should encourage them to rest. Take care of yourself and each other.

What does this mean for you, for each team member? Lets consider it by functional team, since we already understand we're all in this together, anyway.

Hackers1

You are the expert on the software we create. You know where it is, how it's organized, how it's tested, and how it's deployed. If a problem can be solved by writing some code you know how to do that and you will. If you don't know how—no problem—you will soon. You aren't afraid of a challenge. You're courageous!

You know how to test what the team has created. You collaborate with other team members regularly to share knowledge and ideas, and to work together to find the best way to do the work because, after all, the team is all responsible for it.

When you're concerned there's a problem you raise the issue openly, trust the team to respond, and get to work fixing it with the team.

You help others always, especially lending support in areas of your expertise. You take care to teach others what you know so we all learn and grow.

You know what to do next because you follow the team's progress. If ever you don't know you will soon because you'll ask. Nobody has to give you a detailed task list, though, because you're self motivated and will dig into the next unsolved problem.

Developers: See Hackers

Testers: See Hackers

Product Management and Design: See Hackers

Scrum Master: See Hackers

Interns: See Hackers

These are the expectations the team has of every member. Regardless of your title or tenure this is your charge.

Everyone on The Shire is a hacker, so happy hacking!

Shire Hacker Job Description

About Us

WhiteHat Security helps prevent website attacks by providing the most complete Web security solution. We keep our customers safe in some of the most heavily regulated and attacked industries out there.

At WhiteHat we code and have fun. We're hacker culture1. We're focused on delivering quality code every day and we do that by taking risks, staying flexible, and moving fast. Our work is challenging and our team is passionate about it. We're creative and dedicated.

Position Summary

Do you want to be challenged by your job and overcome those challenges with awesome people? If you do we want you on the team!

We want full stack, modern web application developers: polyglots who use the right tool for the job. As a team member you will work on products that run our business. We need you to passionately champion great ideas, collaborate on a distributed team, and exercise unencumbered intellectual curiosity so we can ship fantastic software together.

Desired Skills and Experience

A qualified candidate will:

  • Have experience writing applications that are secure and have to scale.
  • Have experience with responsible development practices: writing unit and functional tests as part of development; writing maintainable, well structured code thoughtfully.
  • Be comfortable working collaboratively with distributed teams.
  • Have some exposure to the full stack: *nix, devops, automated systems, application development, javascript, CSS, and HTML5.
  • Be fearless: you know how to find the answer and you're not afraid to take a risk and try things that might not work.
  • Be a scientist: collect data and make informed decisions whenever possible.

An ideal candidate will also:

  • Love beautiful code: elegant code is often the result of elegant solutions.
  • Have extensive hands-on experience with web application frameworks such as Ruby on Rails, Django, or Express.
  • Contribute to open source projects and participate in technical communities.
  • Understand distributed architectures such as data sharding, service oriented architecture (SOA), and load balancing, and know when to use them.

This position will likely be filled at our headquarters in Santa Clara, or our downtown Pittsburgh office. Remote work may be an option for an exceptional candidate with remote work experience outside the Bay Area or Pittsburgh. Please apply if you are an exceptional candidate!

Thank you for considering WhiteHat Security. When you apply please include your GitHub user ID and tell us which open source project really excites you right now, and why.

WhiteHat Security is an equal employment opportunity company.

  1. "The act of engaging in activities (such as programming or other media) in a spirit of playfulness and exploration is termed hacking. However the defining characteristic of a hacker is not the activities performed themselves (e.g. programming), but the manner in which it is done: Hacking entails some form of excellence, for example exploring the limits of what is possible, thereby doing something exciting and meaningful." — Hacker (programming subculture), Wikipedia 2 3

I love developing applications with the peek gem. It's designed for development or staging environments; there are a few tricks if you don't want it on production or test environments.

This gem puts a little bar on your pages which gives you all sorts of helpful information about the request you just made.

peek screenshot

As you can see there is information about the branch you're on, performance metrics, database queries, caches, shared memory, and job queues in there. Read the documentation for peek on GitHub and find the plugins that work best for you. They're excellent.

The trouble is I definitely don't want to install Peek on production, or in my test environments, and the peek usage instructions would, indeed, require me to do so for a few key reasons: initializers, routes, assets, and views. Using peek requires a few steps of setup in your application. Each of the areas I enumerated requires peek to be a loaded gem in the environment you're running on.

My first attempt to work around this involved wrapping calls with a defined? test. For example, in config/routes.rb:

1
mount Peek::Railtie => '/peek' if defined?(Peek)

To style and script peek bar you need some CSS and JavaScript assets as well, which are loaded through the asset pipeline. Problem is, if the gem isn't in your bundle those loads fail, and so does pre-compilation if you attempt it.

Finally, your views should be checking the peek_enabled? method, which is defined by peek in your ApplicationController class. So, for example, when rendering the peek bar itself you should be doing this:

1
2
body
  = render 'peek/bar' if peek_enabled?

Again, if peek isn't in your Bundle, such as in the production environment, this render will fail.

I'm not particularly enthusiastic about this situation, so here is my guide to using peek responsibly. There are enough moving parts I decided to use the Table of Contents.

Gemfile

Peek must be loaded into our environment. I'm loading it into my :development environment only, along with a list of Peek plugins I want to use.

1
2
3
4
5
6
7
8
group :development do
  # Peek Bar
  gem 'peek'
  gem 'peek-git'
  gem 'peek-gc'
  gem 'peek-performance_bar'
  gem 'peek-pg'
end

PeekBar Class

This class is meant to answer two questions:

  1. Is Peek available for use? This question is asked when we start and configure the application.
  2. Should peek be enabled for this request? This question is asked during requests. Specifically during view rendering for any view that incorporates peek, which is most likely a layout.

Definition

I put this class in lib/peek_bar.rb.

1
2
3
4
5
6
7
8
9
10
11
12
class PeekBar
  def self.available?
    !defined?(Peek).nil?
  end

  def self.enabled?(current_user)
    return false unless available?

    # The default test.
    %w(development staging).include? Rails.env
  end
end

Here you can see we've encapsulated the defined? test into PeekBar.available?. We use it in PeekBar.enabled? as the first test.

If peek is available, and we're in development or the current_user likes nerdy toys we enable peek.

Test

I use rspec for testing, so my spec is in spec/lib/peek_bar_spec.rb.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
require 'rails_helper'

RSpec.describe PeekBar do
  subject { described_class }

  context 'without Peek' do
    before(:each) do
      Object.send(:remove_const, :Peek) if defined?(Peek)
    end

    it "isn't available" do
      expect(subject.available?).to be(false)
    end

    it "isn't enabled" do
      expect(subject.enabled?(anything)).to be(false)
    end
  end

  context 'with Peek' do
    before(:each) { Peek = Class.new }
    after(:each) { Object.send(:remove_const, :Peek) }

    it 'is available' do
      expect(subject.available?).to be(true)
    end

    it 'is enabled' do
      allow(Rails.env).to receive(:development?).and_return(true)
      expect(subject.enabled?(anything)).to be(true)
    end
  end
end

In order to ensure this test is isolated I take extra precautions to ensure the Peek constant isn't available to my tests unless I explicitly allow it.

Initialization

By default Rails doesn't load ruby code from the lib/ directory, so lets be sure to tell it to. In config/application.rb add this:

1
2
3
config.autoload_paths += %W(
  #{config.root}/lib
)

PeekBarHelper Module

As I said earlier peek implements a peek_enabled? method and suggests if you want to customize it define the method in your ApplicationController class. I recommend doing that in a helper class instead.

We need to know if peek should be enabled and to verify that we also need to know if it's available. With the PeekBar class this is simple.

Definition

In app/helpers/peek_bar_helper.rb.

1
2
3
4
5
6
module PeekBarHelper
  # Typical Peek Integration
  def peek_enabled?
    PeekBar.enabled?(current_user)
  end
end

Test

Since we're programming responsibly here's a test for the helper. In spec/helpers/peek_bar_helper_spec.rb.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
require 'rails_helper'

RSpec.describe PeekBarHelper, type: :helper do
  before(:each) do
    allow(helper).to receive(:current_user).and_return(anything)
  end

  it 'is not enabled' do
    allow(PeekBar).to receive(:enabled?).and_return(false)
    expect(helper.peek_enabled?).to be(false)
  end

  it 'is enabled' do
    allow(PeekBar).to receive(:enabled?).and_return(true)
    expect(helper.peek_enabled?).to be(true)
  end
end

Peek Initialization

The peek documentation explains how to set up config/initializers/peek.rb to enable the plugins you've selected. We need to first test for the availability of peek like this:

1
2
3
4
5
6
if PeekBar.available?
  Peek.into Peek::Views::Git
  Peek.into Peek::Views::GC
  Peek.into Peek::Views::PerformanceBar
  Peek.into Peek::Views::PG
end

Assets

We don't want to load or pre-compile the peek assets unless we intend to use them. My recommended approach is to wrap their inclusion in a new set of CSS and JavaScript files. I use the asset pipeline, so that's how I'll be including the gem's assets.

CSS and JavaScript Definition

CSS is in app/assets/stylesheets/peek_bar.scss.

1
2
3
4
5
6
7
8
9
10
11
//= require peek
//= require peek/views/performance_bar

// Peek bar on bottom of page.
#peek {
  position: fixed;
  bottom:   0;
  left:     0;
  right:    0;
  z-index:  999;
}

I prefer CoffeeScript so I loaded the JavaScript in app/assets/javascripts/peek_bar.js.coffee.

1
2
#= require peek
#= require peek/views/performance_bar

Initialization

The asset pipeline complains if you don't tell it about additional assets you intend to include separately (see the next section on View Configuration). Thing is, we don't always intend to include them or pre-compile them in this case. Just like when we initialized peek itself we must check for its availability.

Add this to config/initializers/assets.rb:

1
2
3
if PeekBar.available?
  Rails.application.config.assets.precompile += %w(peek_bar.css peek_bar.js)
end

Default Assets Gotcha

By default application.css and application.js include all the CSS and JavaScript in their respective directory trees. That's achieved through the use of require_tree . in each of those files.

Now that we've created a couple assets to conditionally load we need to remove require_tree. If you don't remove it your default assets will try to include peek_bar.* which will lead to the aforementioned failures on environments where peek is not installed.

If you were relying on it, which I don't recommend for reasons beyond the scope of this essay, learn how to deal with it by reading the Asset Pipeline Rails Guide.

View Configuration

All this hard work for what? Now it's time to tie it all together in our layout view. I like slim so this is an extremely condensed version of my default layout, app/views/layouts/application.html.slim.

1
2
3
4
5
6
7
8
9
10
11
12
13
doctype html
html
  head
    = stylesheet_link_tag 'application', media: 'all', 'data-turbolinks-track' => true
    - if peek_enabled?
      = stylesheet_link_tag 'peek_bar', media: 'all', 'data-turbolinks-track' => true
    = csrf_meta_tags
body
  = render 'peek/bar' if peek_enabled?
  = yield
  = javascript_include_tag 'application', 'data-turbolinks-track' => true
  - if peek_enabled?
    = javascript_include_tag 'peek_bar', 'data-turbolinks-track' => true

Finally

Voila!

At this point peek is incorporated into my application in a robust, responsible way. Environments we didn't install it into operate perfectly without it, and the ones we do have an encapsulation to use for managing its inclusion in our interface.

It was a little more work but it paid off. We have an excellent developer tool installed without hurting production.

If we want to use peek in production with only staff accounts, for example, all we have to do is change the Gemfile to install peek in the :production group and adjust the PeekBar.enabled? method:

1
2
3
4
5
def self.enabled?(current_user)
  return false unless available?

  Rails.env.development? || current_user.staff?
end

That PeekBar class really tied the room together, did it not?

If you read my post on running bundle-audit from rake you'll also know I run RuboCop as part of continuous integration in Travis CI. Today I switched a project to run on Travis' new docker infrastructure and something hilarious happened.

Running Travis CI on Docker

This part is super easy. Amazingly easy. Add this to your .travis.yml file:

1
2
3
# Docker Infrastructure
sudo: false
cache: bundler

The sudo key is the magic word to build your project in a Docker container. The cache key will enable bundler caching for your project.

These two lines can speed up your build times by nearly eliminating startup costs. I went from 2.5 minutes to 24 seconds on builds which used the cache and archived images. I'm running bundle-audit, brakeman, rubocop, and rspec in 24 seconds. That is awesome!

The funny thing is…

The first run I had a little hiccup. RuboCop was doing its thing and, on this project, it only has 30 files to inspect. I know that because I just ran it locally and, yep, 30 files. In the Travis environment, however, it was scanning 5320 files. That's probably wrong.

Turns out, by enabling the cache in Travis I caused RuboCop to lose its mind. How? Because of where the cache was stored. Travis put the cache in vendor/bundle which is in my Rails app directory and within sight of RuboCop. That's legit, but I don't actually want RuboCop to check any vendor code.

Why was RuboCop doing that? Because I wanted to Exclude a few things in my Rails app—mostly generated code—so I had a .rubocop.yml file that looked a little like this:

1
2
3
4
5
6
7
8
9
10
require: rubocop-rspec
inherit_from: .rubocop_todo.yml
AllCops:
  Exclude:
    - 'db/schema.rb'
    - 'bin/**/*'
    - 'config/initializers/devise.rb'
    - 'config/initializers/simple_form*'
    - 'db/migrate/*'
  RunRailsCops: true

The default RuboCop configuration does exclude vendor. I need that, too, but my Exclude above overrides the default, so it was gone.

The simple fix

Lets just add vendor back into the exclusion list:

1
2
3
4
5
6
7
8
9
10
11
require: rubocop-rspec
inherit_from: .rubocop_todo.yml
AllCops:
  Exclude:
    - 'db/schema.rb'
    - 'bin/**/*'
    - 'config/initializers/devise.rb'
    - 'config/initializers/simple_form*'
    - 'db/migrate/*'
    - 'vendor/**/*'
  RunRailsCops: true

Voila!

Be careful when modifying RuboCop's configuration that you copy what you need to keep from the defaults, because defining an entry in your project will discard the them, not merge them.

Originally published by me on December 18, 2006 and reprinted here as-is.

I recently read Frederick P. Brooks, Jr.'s essay No Silver Bullet: Essence and Accidents of Software Engineering. Brooks wrote this paper 20 years ago and it still rings true today. I like the whole essay and encourage you to read it if you haven't already.

Brooks is an oft quoted writer. I'm going to indulge in the practice by pointing out a few things from this essay that I think will always be true.

Likewise, a scaling-up of a software entity is not merely a repetition of the same elements in larger sizes, it is necessarily an increase in the number of different elements. In most cases, the elements interact with each other in some nonlinear fashion, and the complexity of the whole increases much more than linearly.

The complexity of software is an essential property, not an accidental one. Hence, descriptions of a software entity that abstract away its complexity often abstract away its essence.

I really like this, because today there are a lot of arguably excellent attempts to abstract complexity in modern application development. Many software developers look to frameworks and application environments to manage the complexity of software development for them. These developers expect incredible gains in productivity by using a tool that makes building applications "really easy." Don't be fooled. Tools help you build things, yes, but don't expect to get a pony when you install Ruby on Rails (although, as it happens, you do get a pony when you install Jifty). Frameworks are not 100% tools, they're 80% tools. If they were 100% tools then you wouldn't be making applications you'd be writing configuration files. What kind of programmer writes configuration files for a fun?

In many cases, the software must conform because it is the most recent arrival on the scene. In others, it must conform because it is perceived as the most conformable. But in all cases, much complexity comes from conformation to other interfaces; this complexity cannot be simplified out by any redesign of the software alone.

This most obvious form of complexity through conformity in web applications is browser compatibility. I don't think that is the scenario that takes the cake, however. Anything relating to or interfacing with email clients increases complexity in an extremely non-linear fashion. There are other issues, too, such as representing information for RSS and Atom feeds. A lot of cruft exists around the edges of software because it must conform to its environment. You cannot simplify conformity within your software alone.

In spite of progress in restricting and simplifying the structures of software, they remain inherently unvisualizable, and thus do not permit the mind to use some of its most powerful conceptual tools. This lack not only impedes the process of design within one mind, it severely hinders communication among minds.

If you've ever worked with someone else when writing software you know this to be true. The inability to visualize software is a serious friction point between team members.

When asked "What do you think makes some programmers 10 or 100 times more productive than othes?" Peter Norvig answers, "The ability to fit the whole problem into their heads at one time." That's often a critical ability and it speaks to Brooks's admission that software design is hard enough within a single mind. David Heinemeier Hansson has a complimentary answer to the same question, "The ability to restate hard problems as easy ones." Having the ability to clearly communicate software design is important, yes. I would add to these answers, "The ability to explicitly articulate the design of complex software." Having these attributes brings you closer to making the invisible visible.

Imagine how important these skills must be if you're building a geographically distributed team like 37signals or Socialtext? The lowest common denominator for being hired on these teams is likely expressed in the previous paragraph.

I do not believe we will find productivity magic here. Program verification is a very powerful concept, and it will be very important for such things as secure operating-system kernels. The technology does not promise, however, to save labor. Verifications are so much work that only a few substantial programs have ever been verified.

20 years after writing this Brooks is still right. In context, Brooks is talking about "test first" methodologies here, asserting that testing during design and specification will not save you labor. He goes on:

More seriously, even perfect program verification can only establish that a program meets its specification. The hardest part of the software task is arriving at a complete and consistent specification, and much of the essence of building a program is in fact the debugging of the specification.

In my opinion this statement is a key gem. I interpret this statement to be a nod to iterative development, suggesting that the imperfect art of testing our assumptions over time is a large part of the essence of software development. One could bastardize this as "release early, release often." I feel that's an oversimplification that leaves large room for error.

Taking time to design software is critical. I agree with Joel when he says "Programmers and software engineers who dive into code without writing a spec tend to think they're cool gunslingers, shooting from the hip. They're not. They are terribly unproductive. They write bad code and produce shoddy software, and they threaten their projects by taking giant risks which are completely uncalled for."

Let's skip to the end, to the real message within the message of the essence of software development:

Hence, although I strongly support the technology-transfer and curriculumdevelopment efforts now under way, I think the most important single effort we can mount is to develop ways to grow great designers.

No software organization can ignore this challenge. Good managers, scarce though they be, are no scarcer than good designers. Great designers and great managers are both very rare. Most organizations spend considerable effort in finding and cultivating the management prospects; I know of none that spends equal effort in finding and developing the great designers upon whom the technical excellence of the products will ultimately depend.

This, I believe, is the most important advancement since Brooks wrote his paper. Organizations have recognized the need to grow great software designers. As a result the pace of development and product creation has increased dramatically. The state of the world is much better now than 1986 but we still have a long way to go.