I just got back from Devops Days Austin. It was a really good conference. I think I enjoyed the speakers and Open Spaces at this event more than I did at Devops Days Mountain View last year. Huge props to the organizers and speakers for putting together such a great set of topics.

One thing irked me a bit though, I heard a number of comments about how there was too much talk about culture.

I am confused - but I think I understand.

I attended one Open Space session called “Culture Hacks” which ended up not being so much about hacks as a plea for help. A number of folks expressed concerns about being in a difficult situation where they associate their problems with organizational cultural problems and they were looking for ideas on how to make things better. The suggestions that folks raised, myself included, sounded a lot like typical lean/agile tools - retrospectives, stand-ups, putting developers on-call, communicating metrics, providing incentives, hackathons. I think these are all good suggestions, the problem is that they only go so far. The unfortunate reality is that old saying - you can lead a horse to water, but you can’t make him drink. These are all tools but they don’t solve a problem in an organization that doesn’t want to change.

I think the topic of culture is pretty big and scary for many folks. It’s also very subjective, as John Willis raised in his talk - the culture in which a group of crooks thrive is very different than the culture in which a group of hippies would thrive (John didn’t use Hippies in his example). For each group though, there is a specific culture that allows those individuals to reach their goals in an optimal way. Is one culture right or wrong? Functional or dysfunctional? I think the answer is that if the culture is aligned with the team and makes them perform in an optimal way then it’s good - but it’s good for that group, not for everyone.

As such, if you haven’t defined an objective for your culture, if you’ve hired a mish-mash of people with different objectives and principles then you really aren’t going to find a culture that makes everyone achieve their goals. You can make it better for some, but it’ll likely not be better for others. Getting the folks off the bus who do not align with your desired culture is an important part of a change like this and is not something that an Ops person in one group can do. Here, I think, lies one of the main barriers to DevOps being about culture change - it’s driven by Ops people.

Ops working with Dev is great, we can all do that, but we are individuals who can only set an example and hope the organization follows suit. If they don’t - you can vote with your feet or suck it up, I’m not sure there are many other options. Setting an example often manifests as Ops folks trying get closer to Dev - the most natural way to do this is to write code, help with release engineering, enable developers to gain access to monitoring, logs, config management, etc. All the tools that we say aren’t actually what Devops is about… These seem to me to be a manifestation of ops folks doing what they can do get closer to Dev, that’s all.

Of course it isn’t culture, company culture is bigger than this, but it can change a small part of an organization & help set an example for the broader organization. Sometimes it can have a bigger impact - I love this one by John Allspaw as an example. Still, the change was isolated.

I recently read & really liked this TechCrunch article because I feel like it hits on an important point. If I work at a company where the CEO, CTO, COO, VP of Engineering or a variety of other high level positions push a culture that I don’t like - my chances of changing that are small. Further still, there is already a culture which is somewhat defined by the team you’ve hired. Unless you’ve worked very hard to hire for a specific person & team fit then no amount of effort is going to change the organizational culture.

So what is my point with all of this? While Dev & Ops collaboration is important to having a healthy development process - it is so easily undermined by problems with the larger organization. I don’t think that means you do nothing - I just think it helps clarify why talking about culture is hard. I also think having more actionable steps, more examples to replicate would be helpful. Right now the examples we have are of companies who have a good culture throughout - but what about companies who can’t do that? How do I fix my piece? How do I get my 10 person startup pointed in the right direction? How can I build a functional bubble inside a dysfunctional behemoth? What can I do to make my small corner of the world suck less?

There aren’t enough good examples for these questions - and this is why we need to talk about it. Talking about the little bits and pieces that work for you helps - so don’t avoid talking about it because you don’t have all the answers - share what you can. In that Open Space I attended, plenty of folks had little ideas about things that worked for them - they didn’t have a complete solution, but they had some ideas.

I shared my thoughts about how this may look when you start from the ground up and we’re implementing some of these ideas at my current company. Are they all the right things for this company? Probably not - the team will decide. Some of the things that worked at a past company wont work here - does that mean we are doomed? I don’t think it does.

Culture is very personal - but so are a lot of things that we have some established patterns for. We are homing in on some ideas that work - we need more ideas - and we need to talk about them more. I hope this comes up more at other Devops Days events in the future. It’s an important topic.

As part of my role at a new company I’ve been asked to provide feedback about structuring Dev & Ops as well as what sorts of things work and don’t. I certainly don’t claim to have all the answers, but I’ve seen some very functional and some very dysfunctional organizations. I’ve spent a fair amount of time thinking about what works & why.

Below is a cleaned up version of a message I sent to our CEO who asked for my thoughts on what does and doesn’t work. This was intended as scaffolding for further discussion so I didn’t go into deep details. If you want more details on any particular area just throw some comments out there.

I realize not all these issues are black & white to many folks - there are gray areas. My goal with this message was to drive conversation.

I figure this is probably review to many folks, but maybe it’ll help someone.

---

First, there are some very simple goals that all these bullets drive toward & they’re somewhat exclusive to SaaS companies:

• Customers should continuously receive value from Developers as code is incrementally pushed out
• Developers should get early feedback from customers on changes by enabling features for customers to test
• We can address problems for customers very quickly - often in a matter of hours
• We can inspect and understand customer behavior very deeply, gathering exceptional detail about how they use the service.
• We can swap out components & substantially change the underlying software without the customer knowing (if we do it right)
• We can measure how happy customers are with changes as we make them based on behavior & feedback

The lists below are what I feel make that possible (Good) and what inhibit it (Bad)

Culture & Communication

Good:

• Stand ups.
• Retrospectives
• Small, self-formed teams (Let folks work on their area of passion)
• Use Information Radiators whenever possible (Kanban boards, stats on big monitors, etc)
• Decisions by teams, Leaders facilitate consensus
• Discovering what doesn’t work is part of finding the right solution, not something to fear.
• Hackathons allow Developers to do things they are passionate about
• Hire for personality & team fit first, technical ability second
• Data driven decisions, strive to have facts to back up decisions.
• Make the right behavior the easiest thing to do - build a low resistance path to doing the right thing.

Bad:

• Top down decision making
• Strict role assignments & Silos
• Fear of not getting it right the first time
• Hiring for technical ability thinking team fit will come later
• Creating process out of fear that makes it difficult to do the right thing.

Eliminate Manual Processes

Good:

• Continuous Deployment / Delivery
• Fully automated testing
• Test Driven Development
• Fully automated system monitoring, configuration & provisioning
• Separate Deploy & Release (Feature toggles)
• Deploy from master, do not branch (Forces particular behaviors)

Bad:

• Manual testing by a QA Team - sometimes it’s necessary, but should be avoided
• Deploying off a branch, slows things down & allows for other bad behaviors
• Writing tests after writing code, code isn’t written with testing in mind
• Developers relying on other teams to perform tasks that could be automated.
• Processes that are the result of fear rather than necessary business process.

If it moves, measure it

Good:

• Collect high resolution metrics about everything you possibly can
• Developers can add new metrics by pushing new code, do not rely on additional configuration by other teams.
• Graphs & metrics can be seen by anyone - Developers should rely on these.
• There should be individuals or teams who are passionate about data visualization & analysis.
• Dev teams rely on these metrics to make decisions, help identify what metrics are important
• Developers watch metrics after pushing new code, watch for trend changes (Devs take responsibility for availability)

Bad:

• Operations has to configure new metrics after developers have added support for them (Manual)
• Operations monitors metrics & asks Dev teams when they think there’s a problem
• Developers don’t look at metrics unless something is brought to their attention
• Code doesn’t expose metrics until someone else asks for it

And here is the long version of all of that…

#1 Culture & Communication

Above all else I consider these most important. I think most problems in other areas of the business can be overcome if you do well in these areas. Rally has been, by far, the best example of a very successful model that I’ve seen in this area. They aren’t unique - there are other companies with similar models & similar successes.

Main points

• Stand ups. By far the most effective tool for keeping everyone in touch. As teams grow you have to break them apart, so you have a 2nd standup where teams can bring cross-team items to share.
• Projects are tackled by relatively small, typically self-formed teams. Get individuals who are interested in working in an area together & they feed on each others passion.
• Perform retrospectives. This gives individuals & small groups the ability to voice concerns in a way that fosters resolution. There’s an art to facilitating this but it works well when done right. It also allows recognition of things that are done well.
• Use open information radiators - it should be easy to see what’s going on by looking at status somewhere vs. having to ask for status, go to meetings, etc. Kanban boards are great for this.
• Leaders exist to facilitate and help drive consensus but decisions are largely made by teams, not leaders. This makes being a leader harder, but it makes the teams more empowered.
• Accept that things may not work & the team and company will adjust when things do not work. This makes it easy to try new things & easy for people to vocalize when they think it isn’t working. If it’s hard to change process then people are more resistant to try new things. This goes back to retrospectives for keeping things in check. Also important in this are “spikes” or time boxed efforts explicitly designed to explore possibilities.
• Give developers time to pursue their own projects for the company. Many awesome features have come out of Hackathons where developers spent their own time to build something they were passionate about.
• Hire for personality fit first. I have seen many awesome people find a special niche in a company because they grew into a role that you couldn’t hire for - but what made that possible was that they worked well with the team as an individual. Hiring for technical skill also means you lose that skill when that person leaves, I would prefer to have cross-functional teams.
• Data driven decisions. This helps keep emotion and “I think xyz” out of the discussion & focuses on the data we do and do not have. If we don’t have data we either get more or acknowledge we may not be making the right decision but we’re going to move forward.
• Make the right thing the easiest thing. I’ve seen too many companies put process out there that makes the “right thing” really difficult, so it gets bypassed. The right thing should be an express train to done - very little resistance and very easy to do. It’s when you start wanting to do things differently that it should become harder, more painful.

Also, everyone owns the quality of the service. This includes availability, performance, user experience, cost to deliver, etc. At my last company, there was exceptional collaboration between Operations, Engineering and Product (and across engineering teams) on all aspects of the service and there was a strong culture of shared ownership & very little finger pointing.

If you want more details on this specific to Rally I wrote a blog post with some more info: Blog Post

#2 Obsessively eliminate manual process - let computers do what they are good at.

This is so much easier to do up front. There should be as little manual process as possible standing between a developer adding value for customers (writing code) and that code getting into production. There may be business process that controls when that feature is enabled for customers - but the act of deploying & testing that code should not be blocked by manual process. I refer to this as separating “Deploy” from “Release” - those are two very different things.

Testing should only be manual to invalidate assumptions, validating assumptions should be automatic When we assume that if x is true then y will occur, there should be a test to validate that this is true. Testers should not manually validate these sorts of things unless there is just no way to automate them (rare). Testers are valuable to invalidate assumptions. Testers should be looking at the assumptions made by Developers and helping identify those assumptions that may not always be correct.

Too many organizations rely on manual testing because it’s “easier”, but it has some serious drawbacks:

• You can only change your system as fast as your team can manually test it - which is very slow.
• Your testing is done by humans who make mistakes and don’t behave predictably so you get inaccurate results.
• The # of tests will only grow over time, requiring either more humans or more time, or both. It doesn’t scale.

Over time the software quality gets lower, takes longer to test, and the test results become less reliable. This is a death spiral for many companies who eventually find it very hard to make changes due to fear & low confidence in testing.

Avoiding this requires developers spend more time up front writing automated tests. This means developers might spend 60-70% of their time developing tests vs. writing code - this is the cost of doing business if you want to produce high quality software.

That may seem excessive, but the tradeoffs are significant:

• Much higher code quality which stays high (those tests are always run, so re-introduced bugs (regressions) get caught)
• Faster developer on boarding, the tests describe how the code should behave and act as documentation.
• Refactoring code becomes easier because you know the tests describe what it should do.
• Each commit to the codebase is fully tested, allowing nearly immediate deployment to production if done right.
• Problems that make it into product feed back into more tests & continually improve code quality.

Much of the time developing tests is spent thinking about how to solve the problem, but you are also writing code with the intent of making it testable. Code is often written differently when the developer knows tests need to pass vs. someone manually testing it. It’s much harder to come along later and write tests for existing code.

You will hear me talk about Continuous Deployment & Continuous Integration - I feel these practices are extremely important to driving the above “good” behaviors. If you strive for Continuous Deployment then everything else falls into place without much disagreement because it has to be that way. This has a lot of benefits beyond what’s listed above:

• Value can be delivered to customers in days or hours instead of weeks or months
• Developers can get immediate feedback on their change in production
• New features can be tuned & tweaked while they are fresh in a developers mind
• You can focus on making it fast to resolve defects, no matter how predictable they are, rather than trying to predict all the ways things might go wrong.
• Most of the tools and behaviors that enable Continuous Deployment scale to very large teams & very frequent deployments. Amazon is a prime example of this, deploying something, somewhere, about every 11 seconds. Many companies that are in the 30-100 engineer size talk about deploying tens of times per day.
• This also impacts how you hire QA/Testers. This is a longer discussion, but you want to hire folks who can help during the test planning phase & can help Developers write better tests. Ideally your testers are also developers & work in a way that’s similar to Operations, helping your Developers to be better at their jobs.

#3 If it moves, measure it

I mentioned above, two big advantages a SaaS organization has are the amount it can learn about how customers use the product & the ability to change things rapidly. Both of these require obsessive measurement of everything that is going on so that you know if things are better or worse. Some of these metrics are about user behavior & experience to understand how the service is being used. Other metrics are about system performance & behavior.

The ability to expose some % of your customer base to a new feature & measure their feedback to that is huge. Plenty of companies have perfected the art of A/B testing but at the heart of it is the ability to measure behavior. Similar to testing, the software has to be built in a way which allows this behavior to be measured.

System performance similarly requires a lot of instrumentation to identify changes in trends, to identify problems areas in the application & to verify when changes actually improve the situation.

I’ve been at too many companies where they simply had no idea how the system was performing today compared to last week to understand if things were better or worse. At my last company I saw a much more mature approach to this measurement which worked pretty well, but it required investment. They had two people fully dedicated to performance analysis & customer behavior analysis.

I’ve been wondering for a few years now, why it’s so hard to get companies to prioritize the work that I feel is important. I mean, I’m telling you how to do it and you aren’t listening - don’t you want to build quality software?

Would you listen to that argument? I wouldn’t. Everybody has an opinion about how to do things, what makes one better than another?

I think you should listen to me, but that’s irrelevant

I’m on my 4th SaaS company at this point. I’m starting early this time and hoping to steer things in the right direction. I feel like I’ve observed some good and some bad and some really ugly at this point and I have a pretty good idea of what patterns and anti-patterns are important. The problem hasn’t changed though, just because I feel they are important doesn’t make them a priority for the company.

When I go to work for a new company a very important question to answer is whether the company is ready and willing to implement all the cultural and technical requirements for Continuous Deployment. I’ve at least figured out that, from my position, it’s exceptionally hard (so far impossible for me) to convince a company they want to do this - they have to already want to. I know how to implement, I know how to enact change, but I need support that has to already exist.

I focus on Continuous Deployment not because it’s a technical solution that you just plug in and go. I focus on it because it drives conversation around all the other areas where organizations should improve. For each improvement you make working toward Continuous Deployment, you make your development process better and your software better. These aren’t things that only provide benefit once you are doing Continuous Deployment - but when you’ve done them all it becomes a fairly easy decision to deploy continuously.

I’m early into my latest venture but already the attention is there, the interest in doing this right. I’m being asked for my thoughts on what we need to prioritize to move toward Continuous Deployment, what do we need to focus on early so that it’s easier later on. I’m also being asked to educate folks on what is important and why, what have I watched work and what have I watched fail. What mistakes can we avoid and what mistakes are we just going to have to make on our own?

Oh, you were hoping not to make mistakes? Good luck with that. The best I can hope for is to make my own mistakes.

Dude, I would never do that…

My first flight on an airplane, in my life, was a skydiving trip. When the instructors discovered this as we ascended toward altitude they said “Well this is perfect, next time you fly you can tell the person next to you that you’ve flown before, but you’ve never landed”. My risk tolerance may differ slightly from others. I like to rock climb as well, plenty of people won’t do that. The thing is, both of these sports involve a very risky activity offset by copious amounts of safety. Still, when you watch the girl up on the cliff hanging by a limb all you can think of is “what if she falls?”. The answer is, she’ll get a little bruised up maybe, but if she’s doing things right she’ll keep right on climbing.

When you’ve been climbing a bit, when you understand the safety mechanisms, you pay much more attention to the climb, to the technique, to each movement. You know that the climber is probably safe, because you know what keeps them safe. You can take more risk with each movement knowing that any single mistake will only set you back so far.

These activities, like Continuous Deployment, look more risky from the outside than they are. If you don’t take the time to understand all the safety mechanisms then you can’t accurately evaluate the risk. For a company who pushes software every few weeks to consider pushing every commit without substantial other changes would be insane. Just like I would never go rock climbing without the right equipment (I’m allergic to free soloing - sorry). The act of Continuous Deployment is a realization of a ton of other effort - and all that effort has to be prioritized before you can ever get on the rock face.

Reality check - you are deploying more often than you think

Lets say you deploy every week - I’m being generous here, but lets just pretend. So you deploy on Thursday during the day because you have an awesome deploy process and you know it’s better to spot problems when everyone is in the office. You spot a problem, what do you do? I’m guessing your deploy was from a branch, so you just fix that branch & deploy. Then you merge the fix into master.

Friday comes along, hey there’s another critical issue. Fix, branch, deploy. Lather, rinse repeat. Meanwhile, depending on how involved the fix is and what other stuff you have going on, you’ve got a bunch of merging to get right and the closer to your next branch you get, the more of a problem this becomes. How about a fix on Wednesday before the next deploy? I’m guessing you’ve already cut the next branch, so now you apply the fix to 3 branches (last week, this week, master).

All this deploying and merging and branching, it’s all work. The problem is - it’s not automated work, it’s work asking for mistakes to be made. It’s risk. Where are your safety mechanisms? Are they your manual testers? Your automated test suite? If those automated tests aren’t good enough to test each commit before it goes to production, why are they good enough to test each weeks deploy? Because you do manual testing?

This all sounds risky to me, but for some reason it sounds less risky than Continuous Deployment to some. I think this can only be because of a lack of understanding around the safety mechanisms, the pre-requisites. The proof is in the pudding though, and if you still produce shitty software when doing Continuous Deployment because you write bad tests and don’t do retrospectives and don’t prioritize the important work of making the system work right - then you’re sunk either way.

There are some companies that are probably better off deploying every 8 weeks.

Wrapping up - why so much focus on Continuous Deployment?

The practices that surround Continuous Deployment/Delivery substantially reduce risk - things like Feature Toggles, automated testing, automated deployments, deploying off master, retrospectives, monitoring, accountability, access, ownership, reduced MTTR, and the list goes on. These all add up to make a software development and deployment environment so safe, anyone can commit code - if it doesn’t work it will not make it to production.

But, things will still break. In my experience you have to break things in very subtle ways for code to get into production & as time goes on and you build better monitoring, even those issues should be detected fast & resolved fast.

It can take a while to reach the end goal, but you’ve got to start somewhere. However, even if you never actually practice Continuous Deployment, all of these practices will produce better software and probably happier developers.

Further Reading

Here are a few other good resources to learn about Continuous Deployment/Delivery

• Continuous Delivery - Great book & site for info about all of this.
• Moving Fast at Scale (Video) - This is a great introduction to these ideas, it’s a few years old now but still very relevant.
• Web Operations - Keeping the data on time - Excellent book about much of these ideas and lots of others

I hung out in an excellent discussion at DevopsDays driven by Spike Morelli around culture. The premise was that DevOps started as an idea around culture - around behavior patterns that lead to better software. Somewhere along the way our industry shifted this discussion into a tools discussion & now the amount of noise out there about “DevOps tools” is magnitudes higher than any discussion about the real reason DevOps exists - to shift culture.

I looked up the definition of “culture” - here are a few definitions:

1. the quality in a person or society that arises from a concern for what is regarded as excellent in arts, letters, manners,scholarly pursuits, etc.
2. that which is excellent in the arts, manners, etc.
3. the behaviors and beliefs characteristic of a particular social, ethnic, or age group: the youth culture; the drug culture.

Note that culture is the manifestation of intellectual achievement. It’s the evidence and result of achievement. I think the 3rd definition is most appropriate for DevOps - what are the behaviors that are characteristic of a well integrated Development & Operations organization?

The challenge, the discussion, was how we can re-balance the scales and get the word out that this is actually about culture and that tools happen as a result of culture, not the other way around. This post begins my contribution to that effort.

The question was asked - do we all agree that culture is the most important thing when it comes to creating a successful business? The short answer is “yes”. If you wanted to hear all the if/and/but/what-if/etc discussions, you should have come to Devopsdays. For the sake of this blog post - culture is the most important factor. If you want case studies and analysis that proves that culture matters - read Jim Collins Good to Great.

My present company has a really excellent culture of Developer / Ops cooperation and collaboration. I wasn’t there when it wasn’t that way (if ever) and so I can’t tell a story about how to change your organization. What I can tell you is what a healthy and thriving Dev/Ops practicing organization looks like and what I think some of the key factors in that success are. I see this as two components - there are fundamental core values that enable and support the culture, and then there are tactical things that are done to make the culture work for us. I’d like to talk about both. The culture is the result of these actions and ideas put into practice.

Background

I work for a company with a well defined set of core values. Those values set forth parameters under which the culture exists. Here’s what they are:

These values are public and they matter - they matter a lot.

These might sound hokey to you - but every single one of them is held high at the company & strongly defended. Defending a list of values like this is hard sometimes. When someone doesn’t show respect to others, how do you uphold that core value? When someone’s idea of “work life balance” is different than another person, how do you support both of them? When creating your own reality means you don’t want to work for Rally anymore - what do you do?

I’m proud to say that in Rally’s case - they are generally true to the core values. Putting “Create your own reality” on a list of core values doesn’t create culture - what creates culture is having repeated examples where individuals have followed their passion & the company has supported them. This support doesn’t just mean they have permission, it means the company uses whatever resources it can to help. Sometimes this means using your resources to help someone find another job. Sometimes this means helping them get an education they can use at another company. Usually though, it means getting them into a role where they can do their best work. Whatever the case - Rally’s culture is to always be true to that core value and do whatever they can to support an employee in creating their own reality.

This is repeated for all of the core values. By being explicit & public about these values they set the stage for what an employee can expect from Rally as a workplace. But there’s more to it - you have to make sure these core values are upheld and you have to make sure they thrive - and this is where some of the tactical parts come in.

What are the tactical things?

• Collaboration - at Rally collaboration is a requirement. Development is done almost exclusively in pairs, planning is done as groups, retrospectives are done regularly and the actions from those retrospectives are announced publicly and followed up on. Architecture decisions are reviewed by a group comprised of Developers, Operations and Product.
• Self Formed Teams - teams are largely formed of individuals who have an interest in that teams work. When we need a task force, an email will go out to the organization looking for people interested in participating & those teams self-form. This also gives anyone in the company the ability to participate in areas of the business they may never otherwise get exposure to.
• Servant Leadership - Leaders at Rally often do very similar work to everyone else - they just have the additional responsibility of enabling their teams. Decisions about how to do things don’t often come from managers, they come from the teams.
• Data Driven Decisions - Not strictly associated with a core value, I think this is one of the most important aspects of the Rally culture. There is an expectation that we establish evidence that a decision is correct. Sometimes this evidence is established before any dev work is done but sometimes this data comes from dark launching a new service or testing out some new piece of software. Either way, it’s understood that the job isn’t really done until you have data to support why a particular decision is right & have talked to the broader group about it.

There are plenty of other things here and there but you get the general idea. We talk a lot & tell each other what we’re doing, we enlist passionate individuals in areas they have interest, we embrace & seek out change and we empower individuals to drive change by working with others.

So what? What does that have to do with Devops?

Everything

2.5 years ago the company had some performance & stability problems. Technical debt had caught up with them and the only real way to fix the problem was to completely change the way the company did development & prioritized their work. The good news is that they did it, but it was made possible by the fact that individuals were empowered to drive that change. Almost overnight, two teams were formed to focus on architectural issues. A council was formed to prioritize architectural work. The things we all complain about never being able to prioritize became a priority and remain a priority to a degree I’ve never experienced before at other companies. Prioritizing this work is defended and advocated by the development teams - something only possible because of the collaborative environment in which we operate.

I have been personally involved in two services that literally started out as a skeleton of an app when they went into production. The goal was to lay the groundwork to allow fast production deployments, get monitoring in place & enable visibility while the system was being developed. This was all done because the developers understand the value of these things, but they don’t know exactly how to build it - they need Ops help. Having tight Ops and Dev collaboration on these projects has made them examples of what works in our organization. These projects become examples for other teams in the company and they push the envelope on new tech. These two projects have:

• Implemented a completely new monitoring framework that allows developers to add any metric they want to the system
• Implemented Continuous deployment
• Established an example of how and why to Dark Launch a service

I’m sure the list will continue to go on… it’s fantastic stuff.

The Rub - culture isn’t much of anything without people who embrace it.

Along with a responsibility for pushing change from the bottom up in Rally comes responsibility for defending culture - or changing it. This means that when you hire people, they have to align with your core values - they have to be willing to defend that culture or the company as a whole needs to shift culture. All those core values and tactical things will not maintain a culture that the team members do not support. Rally’s culture is what it is because everyone takes it seriously and that includes taking it seriously when there’s a problem that needs fixing.

This has happened. There are core values that used to be on that list above but they aren’t anymore. At one point or another things changed and those core values were eroding at other core values. This takes time to surface, it takes time to collect data to show it’s true, but when the teams start to observe this trend they have to take action. This isn’t the job of management alone - this is the job of every member of the company. When the voice begins to develop asking for change - you need a culture that allows that change to take place and for everyone to agree on the new shape things take.

That said, it also isn’t possible if management doesn’t support those same core values. Management has the same responsibility to take those core values seriously.

DevOps is our little corner of a much bigger idea

There’s a problem that we’re trying to fix - we’re trying to improve the happiness of people, the quality of software, and the general health of our industry. Our industry is totally healthy when you look at the bottom line, but we’re looking for something more. We want a happy and healthy development organization (including Ops, because Ops is part of the Development organization), but we also want our other teams to be part of that. As Ops folks and Developers, we can clean up our side of the street - we can do better. We seek to set an example for the rest of the organization.

For culture to really improve in companies it has to go beyond Dev and Ops into Executives, Product, Support, Marketing, Sales and everyone else. You ALL own quality by building a healthy substrate (culture) on top of which all else evolves.

But in the end it’s about culture. It’s really only about culture for now - because when you get culture right the other problems are easy to solve.

Congratulations to those of you who read this far - shoot me a note and let me know you read this far because you probably share the same passion about this that I do. Also - putting up blog posts from 32,000 feet is awesome - thanks Southwest.

I’ve been asked in the past to rate my skills when chatting with a recruiter about a position, it goes something like this:

“Can you rate your skills in the following areas on a scale of 1 to 10:
Linux
Perl
SAN Storage
Networking
etc”

My response is generally, “no and goodbye”. This isn’t out of arrogance, but there’s just no point in moving forward. When we’re talking about a job I’m interviewing your company every step of the way. One key question I’m always asking myself is “does this seem like a hiring process that would hire people I want to work with?”. I don’t get a huge chance to interview my co-workers when I’m coming into a new place so my only real guide as to what type of folks they are is… the hiring process.

If your hiring process starts by selecting people based on how they rate skills, think of who you are excluding. Also, think of who ends up at the top of that list - arrogant, self-important folks who think they are experts in everything. Perhaps the HR person is clever and is looking for folks who rate themselves poorly to suggest humility but there are better ways.

I want to work on a team where you hire people because they fit well into the team, because they have passion about their job, and because they can learn. I don’t want you to hire someone who already thinks they know everything because every conversation with that person is going to be an argument about why something new isn’t as good as what they’ve already done before. People do what has worked for them in the past unless they are tinkerers who like to try new things - in which case they probably don’t become experts, they become generalists. The best folks I’ve worked with didn’t know how to do most of their day to day tasks when they started the job. They learned, they asked questions, they were humble and eager and engaging. Those are the people I want to work with.

Yes, I’m restricting my potential job opportunities by doing this - which is exactly the point.