Moving beyond ad-hoc automation to take advantage of patterns that deliver predictable capabilities.
A piece I wrote on what it means to be cloud-native.
Moving beyond ad-hoc automation to take advantage of patterns that deliver predictable capabilities.
A piece I wrote on what it means to be cloud-native.
I was honored to speak at YAPC::Asia 2015 in August. It was a well-organized conference, and the attendees and speakers were all friendly and fun.
Most of the talks I went to were English-language based. I saw excellent talks on kubernetes, docker, refactoring, golang, and lattice. Fellow Pittsburgh engineer Marylou Lenhart also spoke at the conference about posture for engineers, and I'll be trying to do my cat/cow stretches regularly.
My talk was about working remotely and working on distributed teams. It was based on my writing on durable communication. I got a lot of excellent questions after my talk and at the official dinner, and the feedback was quite positive. My slides are online and the organizing team has already posted the video:
Since I was speaking in English the conference provided live translation into Japense. It was an incredible experience. I felt like a diplomat, especially during Q&A when questions in Japanese were translated into English so I could understand them. The translators were wonderful and I'm very thankful they were available.
Finally, thank you to Daisuke Maki for inviting me to speak this year!
The mission of a DevOps team is to eliminate itself.
DevOps is a practice, not a role. It's like agility. Does your company have an agile team? From Dave Thomas's agility redux:
- You aren’t an agile programmer—you’re a programmer who programs with agility.
- You don’t work on an agile team—your team exhibits agility.
- You don’t use agile tools—you use tools that enhance your agility.
The same applies to the practice of DevOps. I suggest:
If your company has created a dedicated "DevOps team" it signals one of two likely scenarios:
I have built DevOps teams (responding to scenario #1, everything is on fire) and the goal of that team is always to pay the up-front investment in modernizing the software delivery pipeline. Once the investment is paid thier role is to empower engineers to run the software they create themselves, and to enable operators to avoid fires in production and get a good night's sleep. Once that equilibrium is reached it's time to move on to work which genuinely creates business value.
If your organization has a long-lived "DevOps team" and the idea is they'll always be around, how is that different from "throwing it over the wall?" In this case it's just a second wall to an intermediary team between IT and engineering, and it isn't really a good long-term strategy. It can, however, get you over the initial changes required to embrace the DevOps ideal of "you built it, you run it."
It isn't a negative that DevOps teams are created. The rationale for their existence is often quite positive, just that the implementation isn't quite as polished and inline with the intent of the DevOps movement. It should be seen as a short-term tactic and you need to have a long-term strategy. Your DevOps team will never generate business value directly, so your sustained investment isn't wise.
The reduction of cycle time is, without question, a strategic advantage. Your dedicated DevOps team can get you from waterfall to agility much faster, and reduce cycle time drastically, which is a significant improvement in the short term. As the health of your architecture improves, and isn't materially affected by modification (deployment), the value gained by sustained investment in a DevOps team goes down. This is good.
A recent report by Nancy Gohring of FierceDevOps suggests a new approach to solidifying the gap between engineering and IT: adding business people to its DevOps teams. (This reporting is not confirmed by Netflix, the company cited as its example.)
If you have a DevOps team their focus should be the reduction of operational costs over time, to the point where the [software deployment pipeline] is automated and the engineering and operations teams are well educated. A well automated pipeline enables engineering teams to bring their business value to customers: shipping. Educating both engineering and operations on how to automate and manage your architecture empowers them to do it themselves, which distributes knowledge and reduces your bus factor risk.
At this point you've achieved the success of enabling and empowering everyone in your product delivery organization to practice DevOps. Like the practice of agility, DevOps should be a consistent undertone as engineering delivers business value with low operational overhead.
Don't focus on operating your business. Focus on building it.
Trust is absolutely essential for human survival. Until we feel we can rely on the person the the left of us, and the person to the right of us, we can't really achieve anything great.
The reason trust is important is because when we’re surrounded by people who believe what we believe, we’re more confident to take risks. We’re more confident to experiment, which requires failure by the way, we’re more confident to go off and explore knowing that there is someone from within our community, someone who believes what we believe, someone we trust and who trusts us who will watch our back, help us when we fall over and watch our stuff and look after our children while we’re gone.
When we feel safe trust will emerge. This is what the foundations of leadership really are. The reason we call someone leader is because they choose to go first. They choose to extend trust first, even before there are any signs they should. The willingness to express empathy before anyone else.
It's not inherently harder to be a tech lead while remote or on a distributed team – it's more deliberate.
I tweeted this sentiment recently along with the concept of durable communication. That phrase is inspired by data storage which is typically grouped into two categories: durable and ephemeral. I often hear people say it's obviously harder to work remotely, and very hard to be in a leadership position remotely. I think that's a cop out. I don't buy it for a second.
Before we continue: This post is about being a remote worker or working on a distributed team. I am convinced this is the path software organizations are rightly headed, and I'm determined to help you get there. If your organization has no remote workers, by choice or otherwise, or if you think remote work or remote leadership is out of the question this post may not be for you. That's cool. On the other hand, maybe it'll give you the tools, vocabulary, and confidence to give it a shot. Either way I look forward to hearing about it.
Collaboration using deliberately inclusive, transparent, and reliable tools and techniques. It's the cornerstone of successful remote work.
I currently lead three teams in a distributed engineering organization. One is entirely based in California, another entirely in Pennsylvania, and the third is split between those locations. If I'm going to be successful as a leader it's critical that we that we get communication right. I admit to being a change agent and this is one of the ways I've changed the engineering organization. This essay describes how my teams communicate.
Communication is more deliberate and intentional when you're remote and communicating effectively. In a distributed team where pockets of people are co-located it's easy to unintentionally make decisions and share facts only with team members who are in the same physical location. It happens in the kitchen or at lunch. When communicating with someone far away from you, physically, you have to choose to share. It's not an accident.
If you want to succeed at remote work it requires deliberate changes to the way your team communicates. It's a conscious choice to alter behaviors and culture to better suit a distributed group of people.
This is a skill which can be acquired. It takes practice. Here's the thing: it's no more effort than accidental communication and, in fact, it saves everyone time and energy in the long run when information is freely flowing in channels designed to share broadly.
Over time it becomes more comfortable to communicate in a shared space and you'll soon find yourself choosing that by default. You'll say, "I just learned something weird about our data schema. I'll tell you in our chat room." As you describe the oddity that is your data model, in the chat room, you're sharing with everyone regardless of physical location.
One of the most striking changes a team experiences when practicing durable communication is the substantial increase in their transparency. Most communication is in the open. It's archived and indexed. Anyone can read it.
It's like giving your team the super power of time travel. In three years someone is going to be asking, "Why the hell did we define an enum for this obviously boolean attribute and by the way querying it is timing out on large data sets?!" They're going to search your chat archives, read commit history, find tickets, and learn that the attribute was supposed to have five values and was simplified three days before launch because otherwise we weren't going to finish1.
Thank your lucky stars! All the engineers who worked on that feature have moved on to new opportunities and there's nobody left to ask in the hallway. Good thing your communication was durable. Always err on the side of transparency. It'll save your future ass.
Responsibility for durable communication is shared among everyone in a distributed organization; everyone is accountable for their part in making collaboration maximally effective.
If your organization is expanding from co-located to distributed it will come with growing pains. Once durable communication has been fully embraced it will feel effortless. Transition won't feel that way, and that's okay. Change requires additional effort and that is hard, but it's effort spent going in the right direction so it's well spent.
What follows are concrete methods for reducing the pain of distributed work. It's tailored for teams building software.
I'm going to list a handful of options ranging from completely free to paid. These are things I know about. They aren't endorsements unless explicitly stated. In other words, you should do the cost/benefit analysis for your organization and choose wisely.
This is the primary device for durable communication. In my experience it's the most critical tool in your collaboration toolbox. There are some specific features which make it invaluable, and any deployment should have them one way or another:
ChatOps is the use of text chat tools for the transparent automation of business practices. It's so awesome it has a heading. Typically ChatOps is achieved by deploying a bot, which connects to your chat system, and can be programmed to do tasks for you. For example, it could build a new release, do a full deployment, pick a place for lunch, or find the perfect animated GIF for the conversation. It's fun and useful!
ChatOps also helps introduce context into your conversations. It can be hard to know what's going on when you're on a distributed team and we can bridge that gap by doing some of that work right in a chat room where everyone can see it. No more wondering when a release is happening. You just saw someone start a release, and you just saw your bot report the release happened successfully.
Everyone should be able to have a phone call, or Internet equivalent, when they're at work. Voice is fine, usually, and video is higher bandwidth. Non-verbal communication is a huge part of human communication and it should absolutely be a part of your remote work environment. (Please don't make the joke about not wearing pants. It's not funny anymore.)
This tool requires some hardware. It's necessary equipment to effectively do your job and your company should provide it or reimburse you for reasonable expenses.
There are three important features of any effective audio/video communication tool: the ability to have one-on-one calls, the ability to conduct group conference calls, and the ability to do either in 60 seconds or less. This collection of features isn't always available in a single tool, and that's okay. Use a few if you have to. It'll pay off.
This is your fast-track to remote pair programming, troubleshooting, hallway testing, and collaborating with other disciplines. The important features are the same as for audio/video calls: one-on-one sharing, group sharing, and the ability to do either in 60 seconds or less.
"The only difference between science and screwing around is writing it down." – Adam Savage, Mythbusters
Commits are like emails to future developers. Please write good commit messages for our future selves. And not just ourselves, but any future developers working on this code.
Think of it as time travel. When Marty wrote Doc that letter, it wasn’t a cryptic, short message he would only understand in-context. It wasn't simply a P.O. Box address for Doc to go find the details at. It was a clear description of the problem and solution. It was helpful as soon as it was read.
Commits are always in sync with the code. This is awesome. It means we can write documentation about our code without the worry of it getting stale. That's better than comments, even. A commit message for a piece of code lasts exactly as long as the code it's talking about lasts.
More on commit messages:
Written, asynchronous code review is a central part of any solid development cycle. Don't take my word for it. From the excellent book, Code Complete:
…software testing alone has limited effectiveness – the average defect detection rate is only 25 percent for unit testing, 35 percent for function testing, and 45 percent for integration testing. In contrast, the average effectiveness of design and code inspections are 55 and 60 percent. Case studies of review results have been impressive:
- In a software-maintenance organization, 55 percent of one-line maintenance changes were in error before code reviews were introduced. After reviews were introduced, only 2 percent of the changes were in error. When all changes were considered, 95 percent were correct the first time after reviews were introduced. Before reviews were introduced, under 20 percent were correct the first time.
- In a group of 11 programs developed by the same group of people, the first 5 were developed without reviews. The remaining 6 were developed with reviews. After all the programs were released to production, the first 5 had an average of 4.5 errors per 100 lines of code. The 6 that had been inspected had an average of only 0.82 errors per 100. Reviews cut the errors by over 80 percent.
- The Aetna Insurance Company found 82 percent of the errors in a program by using inspections and was able to decrease its development resources by 20 percent.
- IBM's 500,000 line Orbit project used 11 levels of inspections. It was delivered early and had only about 1 percent of the errors that would normally be expected.
- A study of an organization at AT&T with more than 200 people reported a 14 percent increase in productivity and a 90 percent decrease in defects after the organization introduced reviews.
- Jet Propulsion Laboratories estimates that it saves about $25,000 per inspection by finding and fixing defects at an early stage.
Copied from Coding Horror's copy.
Yes, you still need to write tests. Automated testing is not enough to find defects early. Combining them with code review is a winning strategy. Code review reduces defects, increases learning and knowledge sharing, and serves as a written artifact of technical decisions made over time.
More on code review: Derek Prior, "Implementing a Strong Code-Review Culture" (video).
Sticky notes don't scale. I'll caution you here: this is not an opportunity to over-architect your development process. You need a few queues to understand the workload of each step in your development pipeline. This gives you insight into capacity and constraints, and provides a high-level picture of work in progress.
Being able to take notes together in meetings is very powerful. A real-time document sharing tool brings people together in meetings and serves as a very powerful alternative to a whiteboard. Someone can decide to take live notes while others cleanup structure and edit behind them. Everyone can see the conversational history and it can be preserved for posterity.
Preservation should take place elsewhere. These tools usually aren't good choices for long-term, curated document stores. Instead it may make sense to copy your artifacts from here to a story, or a
README, or a specific document under a
docs/ directory in a repository.
Sharing large files is hard, still. Especially at work. It might be PSDs, PDFs, or OVAs. Making this easier will ease a common pain point of distributed teams. The solution you choose will depend on your particular risk profile.
Availability must be constantly communicated. Picking a time to get people together across locations and timezones can be a challenge. Shared calendars solve that problem. Frankly, I wish there were more options here. If you know of any solid alternatives I'd love to hear about them.
No, not really. Email is often one of the slowest, least effective means of collaboration and I don't recommend relying on it for that purpose. It does have a place, and I feel that place is in non-critical communication with no time pressure. For example, when you want to chat with someone who isn't "in the office" at that particular moment and it's not important enough to call them.
It's also an effective way to receive reports and notifications which, often, will lead you to the other collaboration tools mentioned in this document.
If you need a quick answer from a co-worker or team and you're starting to write an email I have this advice: stop. You're about to waste time and introduce needless delay. I bet you'll get a faster answer on text chat.
At some point you should send every team member home, or to their local coffee shop, or into separated conference rooms to simulate being remote. Empathy is a great motivator, and if the folks at headquarters are having trouble adjusting to your remote workforce this is a fast-track way to help.
I love the idea of having everyone work from home (or wherever) every Friday. A policy like this tends to make people happy. They like it. However, it's not a big enough burden to change minds most of the time. It'll become your "heads down day" and the point of the exercise will be lost.
I recommend at least a full week. Everyone is remote. Meetings continue as scheduled. So do pair programming and planning sessions. Standup? Yes. Meetings with other departments? Yes, plan to dial in or video conference.
There's a higher-level business benefit to doing this regularly, too. It can serve as a good, real test of your operational business continuity plans. It's a sustained use of the tools you've put in place for when your office burns to the ground. You are thinking about how to operate the company when your office is on fire, right?
There are team members for whom home is not a suitable work environment. Speaking as a dad who had toddlers I can tell you sometimes it just isn't going to be viable. Don't require everyone to be at home. For folks who can't they should be offered the option of renting a desk at a co-working space (to be expensed), or commandeering a conference room for the week. If they stay in the office, in a conference room, set a rule of no talking in person. Stay true to the simulation.
Get people together. Relationships are best created in person and can be sustained remotely for a while. Minimum twice a year. Being together is too valuable to waste on business as usual. Close the laptops and get to know each other for a while.
It is inevitable, and still good and healthy, that you'll strike up an ad hoc conversation at your desk. It'll probably be about work, and chances are someone in another location is interested in that topic. This is a moment to invite them in. How do you do that when they're in another timezone? A few ways.
Invite them on video chat. This is the best. It's like you're right there in the room because your head is! Unplug your headphones so they can hear everyone and everyone can hear them. You can also invite them to audio, or you can all hop on chat and have the conversation in text. Video is still the highest bandwidth option and for that reason I recommend it.
If you work in an open office you're probably saying something I hear all the time, "but we'll need to find a conference room because it's rude to be on a call in the pit." No, it's not. It's no more rude than the conversation you're having right now. Will it surprise people? Yes, at first. They'll get over it. This is a stigma that should be eliminated. It's also what headphones were invented for.
Your working relationships will benefit from the old adage "always over-communicate", just like any relationship.
A benefit to being co-located with your peers is they get to know you. They learn about your kids, your hobbies, and your affinity for mango flavored tea. Even if you never said anything about it out loud they'd learn about you just from being nearby.
Remote co-workers don't have the luxury of osmosis. Our relationships grow over time through familiarity and comfort. The more we know someone the closer we feel. That's why it's important to share your personality. Share it a lot. How? Off topic.
Every organization needs an Off Topic chat room. This is where the stream of nonsense we all spew every day goes to live. This is where you tell someone you're getting coffee, where you post pictures from last weekend's mountain biking adventure, or where you drop a link to the latest badass thing Neil deGrasse Tyson just said. Off Topic is a safe place to chat about whatever you want with your co-workers.
You may end up with more off topic channels, for specific interests, than work-specific channels. That's okay! I see channels for music, cooking, exercise, and DIY often. Let this thrive and grow organically.
For those of you who lead remote or distributed organizations: (with the exception of offensive content) don't monitor or question the traffic in this room. There will probably be a lot of it, and if you've forgotten what it's like to build a thing you'll probably also wonder how these people get anything done all day. Don't worry about it. This is work. Instead of hearing it in the halls you're seeing it on the screen. No big deal.
When you're remote it's your responsibility to exhibit a visible pulse. You may think nobody cares about every time you make a pot of coffee or push a branch, and you're probably right about that. What they do care about is knowing you're alive, there, and available when needed. Sending those seemingly useless messages is sending them a more important signal: you are reliable and okay.
Your company probably has a home office. It can be helpful to agree that its timezone is your Standard Operating Time (SOT). I've employed this technique before and it works.
When is standup every day? 10am SOT
I'll be leaving the office a little early today so reach be before 1PM SOT if you need anything.
When discussing time you should always be explicit about timezones. The Standard Operating Time technique provides an generalized solution. Each individual can translate SOT to their native timezone. After a little practice it'll be second nature.
Durable communication exhibits the same characteristics as accidental, convenient communication in a co-located space. The powerful difference is how inclusive, transparent, and reliable it is. Durable communication tools and techniques not only scale well with your organization, they'll empower your organization scale well, too.
Conway's law states that "organizations which design systems … are constrained to produce designs which are copies of the communication structures of these organizations".
The health and quality of your product will be a direct reflection of the health and quality of your organization. Take great care of your organization and it will pay off in your product.