Operation Bootstrap

Web Operations, Culture, Security & Startups.

Finding Joy in the Journey

| Comments

I am constantly pulled toward this idea, an idea I’m beginning to think of as delusional, that you can pivot the cultural direction of a company without that company first hitting rock bottom – or their version of it. I think if you are a CEO this doesn’t apply – but I’m not. Companies who do not naturally follow a particular trajectory also don’t respond very favorably to changing direction toward that trajectory. The existing direction is there for a reason, because the majority of the company, for whatever reason, has made decisions large and small which pointed the ship that way. Changing that direction requires changing hearts and minds (or staff) toward some new horizon.

insert over-used and possibly inaccurate Einstein quote here

On the other side of the coin – I’ve worked for companies where stuff worked pretty OK. I’ve participated in what I thought was a reasonably good version of a high functioning team. I left that company because I got bored, or just didn’t feel like the company itself was giving me the opportunity to do what I wanted. What I wanted was to grow a team into that high functioning place.

So I found myself recently being frustrated that my attempts to change a team aren’t meeting my expectations. Unlike me from a number of years ago I’ve become a bit more introspective about things and so I started asking myself some questions:

  1. What is your basis for expecting people to agree with you and change in the way you think they should within some window of time? Is there evidence supporting this expectation?
  2. Would the job be interesting if they all turned on a dime and started doing the right thing naturally tomorrow? No more problems to solve?
  3. Assuming they too want to succeed and do their best, what pressures are preventing them from changing?
  4. Are you being an asshole about it?

Item #1 is that I have expectations that have been established on assumptions not in evidence. This is much the same as being frustrated that when you let go of a bowling ball it falls to the ground – but you expected it to go up. Have you ever observed an object fall up? Why would you expect it? Wanting it, thinking it’s so obvious that it should, does not change physics. Few folks get all bent out of shape when it rains, you can’t control it, you accept it. So too are other people – cannot be controlled, have to be understood and accepted, no real value in getting frustrated.

Teams do change, and there are many stories of teams changing. What is often missing from those stories is some perspective on the effort, time, and patience required to get to that end. This is usually measured in years unless there’s dramatic (and usually very disruptive) action. So, patience.

Item #2 is why I’m here in the first place, this is what I want. I could go work for some company that already has their shit together but what fun is that? Nope, I need to learn how to find joy in this journey and how to become successful at it because this is what drives me. I need only look at my selection of books in amazon for evidence of where my interests lean. Improving my ability to do this means being in an environment where things aren’t right & practicing. When you want to improve your code you find a problem to solve – same goes here.

Again, patience. Whether I like to believe it or not, I’m changing & learning as much as the rest of the team. By adjusting my behavior, others adjust theirs – we learn and adapt to each other, we build rapport. As this happens it unlocks new opportunities and makes possible things that weren’t before.

Item #3 is my fundamental belief that all people want to do good. With few exceptions, people are driven by the same desires I am – to do well & succeed at their goals. When we observe people who aren’t doing what we expect them to do, we have to ask why. Sometimes they don’t even know why, so we have to allow them to show us why. Sometimes what we expect isn’t correct – so we have to be willing to learn.

This is also a core tenant of how I believe this process can work. There’s another blog post brewing about building a “pit of success” – about making the right thing the easiest thing. Doing this requires setting aside your belief about what works & observing and learning from behavior of others. It requires lettings things go wrong & then asking how it can be improved.

Item #4 is my reality check. I can absolutely get passionate, maybe even dogmatic, about how I think things should happen. Given enough resistance I can turn into a righteous asshole. Do you listen to people like that? I don’t. I’m not doing anyone any favors if I’m not listening, learning, and asking how I can help people do well.

I have to sit down with people and ask how I’m doing. I have to check in and make sure I’m not destroying relationships. It shouldn’t be necessary to create adversaries in this process – if we are listening & learning we should be finding common ground. This takes effort, but it’s worth it.

All of this gave me some perspective to look back over the last 6-8 months and what progress has been made. Actually, there’s been a lot. There are definitely some problems, but there’s now evidence that the play dough actually does move – it’s not rigid. The process might not be as fast or as direct as I expect, and it might not even look like what I expect when we get there – but there is movement. I need to realize that it is this movement & the positive results of it that are my journey and when we reach our destination, so too does my journey end and I go looking for another tribe to walk with.

No sense in rushing to that point.

Docker-spoon 1.0 Released

| Comments

I’ve pushed version 1.0.0 of Docker spoon. I moved the version to 1.0 for a few reasons – the release is pretty significant:

  • This release pulls in a version of docker-api (0.17) which is not backward compatible with older docker versions. If you need to run with older versions of Docker, use 0.8.x.
  • I’ve added a ton of test coverage using cucumber and aruba. The tests require a local docker daemon but otherwise should be portable. This is the most thorough coverage spoon has ever had.
  • The original framework used to build spoon – methadone – has been completely removed. This was done because it was getting in the way of setting up a correct default / config file / command line argument parsing hierarchy.

So what’s new?

  • New commands including --kill and --restart to ease stopping spoon containers without destroying them or having to use the docker cli.
  • Support for config/cli defined port specifications both for docker-forwarded ports or ssh-forwarded ports. This supports both dynamic and 1:1 port mappings for docker ports.
  • Privileged mode support when creating new instances.
  • Config support for a variety of docker params including:
    • cpu shares
    • memory limits
    • dns servers
    • volume mappings
    • privileged mode

The release has been tested against Docker 1.4 and is known not to work with docker 1.2.

Pairing With Docker-spoon

| Comments

Whether you pair or not I think this post is worth reading. Do you ever have needs like this?

  • Ability to create on-demand dev environments with all your dependencies in place?
  • Ability to collaborate with local or remote folks in a low latency environment that doesn’t suck?
  • Want bare metal performance instead of using a VM?
  • Want to kick off that build, close your laptop and head home to check on it later?
  • Want to share your work on your dev environment w/ others easily, even with your laptop in your backpack?

There are a bunch more reasons you might like docker-spoon, those are just a few.

This tool grew out of a desire to streamline developer workstation setup but it went even further to the point where developer environments are disposable, can be provisioned in seconds & allow local and remote pairing at any time. Are there other ways to do this? Yep. I’ve seen this tool used successfully by entire dev teams at two companies now – it’s new to you but it’s been around a bit.

The idea behind docker-spoon is pretty basic, you create a docker image with all your developer needs built into it, then you run spoon <name> where name is the name you assign to a new spoon instance running in Docker. Spoon does all the heavy lifting – currently including:

  • Check to see if the container exists, if not create it
  • Forward any exposed ports
  • Copy any files you want copied on creation into the new instance & add your ssh public key for future connections
  • Run any commands you want run on container creation
  • establish an ssh connection into the container

So after installing spoon and creating a config, the time from running spoon to being at a command prompt in your new dev environment is about a few seconds. How easy is it? Demo time!

Video can be viewed directly here

There are some features not shown in this demo which already exist:

  • Forward ports over ssh which aren’t exposed via ssh
  • Run commands on container creation
  • Using spoon to execute docker builds

That demo used the example docker container included in the docker-spoon repository. If you want to try it out quickly just follow the directions in the docker-spoon repo.

So here’s the thing, there are some caveats – all this awesomeness doesn’t come without some conditions. Spoon takes advantage of the idea that working in a terminal with tmux is low latency & easily shared. There is some work being done to use VNC inside spoon to allow for the use of GUI apps but that’s not the optimal use case.

If you find docker-spoon useful let me know. If you want to see something different, submit a pull request or shoot me a note. The usage should be thoroughly documented in the README.

Empathy in OSS - It’s Important

| Comments

Spending nearly my entire working day (and many evenings) with Open Source projects I don’t maintain I have come to form an opinion or two on the use of OSS – specifically when it comes to contributing to projects.

We’ve come a long way as a community with tools like Github, making it easier and easier to get an OSS project out in the wild and accept contributions. As we’ve done this though, projects seem to support users with varying degrees of success. Some projects flourish, graciously accepting community contributions. Other projects do quite well accepting feedback but largely having maintainers drive the direction of the project. Still others fall to extremes of accepting every pull request that comes along with reckless abandon or apathetically allowing the project to rot and become another abandoned github relic.

Along with this more and more companies are building their businesses on OSS. Not just a piece here and there, but entire complex ecosystems of software maintained by all kinds of different folks. Each of these business have their own unique quirks, personalities, and constraints.

I’ve worked for a few of these businesses and I know that those constraints and those personalities, they aren’t easy for me to change. I have to work with what I have and try to find tools that do what I need. When I find a tool that’s 90% of what I need and I’m willing to put in the effort to push it across the line and make it work for me I get to come face to face with the maintainers of that project.

I’d like to say this always goes smoothly, that I’m always able to express my use case in a way that folks understand, and that my use case always falls inline with the intent of the project. I’d love if my skillset in a particular language was always up to par, and that my ability to write quality tests was always appropriate. I’d love to always understand the roadmap for the project & the maintainers expectations around how I should contribute. This just aint so.

As much as I may feel like I don’t understand a particular project well enough, sometimes it feels like the maintainers don’t understand me. What’s worse is when I interpret (accurately or not) an attitude from maintainers that I “just don’t get it” rather than making an attempt to help me understand.

This sounds really familiar to me. This sounds like that whole observation the world made that Dev & Ops need to work more closely. The observation that the people who build the software need to interact more closely with the people who use the software. That building empathy and a collaborative environment where everyone can communicate and be involved will help get us past our differences. For me, this problem isn’t just with the developers in my company. This problem extends to the developers who build the systems I use every day and most of those developers don’t work for my company.

Just as we have to invest in building relationships within a company, when you use OSS you have to invest in building a relationship with those you work with outside your company as well. This goes both ways, and the more effort maintainers put into understanding users, the better this is for everyone. This isn’t easy – so if you aren’t a maintainer and you’re reading this, understand that very often the interactions you have with maintainers is done outside their full time job, outside their ordinary deadlines and pressures, and it’s done because they enjoy working on a project. If folks are like me, when something stops being enjoyable for too long, I stop doing it. Keep them in mind when you have a complaint.

Further, for a popular project the ratio of users to maintainers is pretty imbalanced and not in the favor of the maintainers. They have a hard job and for those that do that job with grace and empathy for their users, my hat is off to you.

So when we talk about DevOps and Empathy and all these great ideas to make your company work better – don’t stop there. Think about all those projects that make your job exist, and all those users that make your project a success. Try to take a little time and understand each other and work together to make more awesome.


Question: How Did You Build an Automated Service Delivery Pipeline?

| Comments

Here’s the scenario – you are a development team tasked with making the life of other developers easier. Your organization is breaking a monolithic service into many micro services and the current process for spinning up a new service and all the associated pieces is laborious, error prone, and not elastic at all. Your organization wants a self-service system that allows a team to spin up the scaffolding for a new service in days without being blocked by other teams.

If you have thoughts on this – I have some questions:

  • Have you done this before?
  • How would you describe what you built?
  • More interesting to me, what were the increments?
  • Did you try to use existing tools?
  • What was the team that delivered those increments comprised of (Ops, developers, both)? I know many Ops are developers – but mindsets are probably different.
  • How long did it take?

Keep in mind, I’m not talking about just deploying code, I’m talking about creating an entire dev workflow pipeline from desktop to prod – automatically. The assumption is that if you aren’t doing Continuous Deployment, you are at least doing Continuous Delivery. This probably looks something like a private PaaS at the end of the day – but automation that extends beyond just spinning up machines, it extends to CI, monitoring, everything.

If this story has been told in a blog post you have, or if you just copied what someone else wrote in their blog post, point me that way, but I want to hear about YOUR experience implementing it.

See, I am fairly certain that the answer to this question changes depending on the makeup of the group that built it. Further, I suspect that without the necessary increments – any group will build it wrong (like any software). So for a group that’s been down that road, I’m more interested in the journey than the destination.

Reply via email, in the comments, add links, use the twitters, whatever. I just want to learn – this is not the start of a debate.