The DevOps Therapist

A DevOps Cultural Safe Zone

   

   Ok so I’m going to try an experiment. I need to hire another member for my team, and I want to do it totally different this time. You are interviewing me, as much as I am interviewing you. To that end, I’m going to give you, the applicant, as much information as I can before you even apply. Yes there is a job description for this role, it can be found here, but that is just a fact filled document. It doesn’t really tell you about what it is like to be a part of the team. So instead, I’m going to provide that information here, as well as a few contacts that you can reach out to who have worked with me in the past.

   First off, you will be getting your paycheck from the company, but you will be working with the team. That means that we will build the team culture together. I don’t use pronouns like “me” and “my” when talking about the team. We decide together, we experiment and implement together and win or fail together. This doesn’t mean that we make every design or decision by committee, but we do strive to make sure everyone has a voice and and opportunity.

   So what do we do exactly? Well it’s an interesting and unique role to be sure. Hearst Business Media has 11 business units that are approximately 200-250 people each. They run autonomously, with their own leadership, products/services and goals. My team works for HBM Corporate, meaning that we do not belong to any single BU, but are a part of the overall HBM family. We provide support to the BU’s in a variety of ways. This means that the job is not just dev, not just ops and not just mentorship, it’s all three.

   The projects that come to my team come directly from the BU’s themselves. We build a rough and high-level roadmap with the BU’s each year and remain flexible in how we deliver and support those needs. It’s similar to an internal consultancy, but we aren’t paid on commission. Our interactions with the BU’s are conducted in the following two formats:

  • Dedicated engagements with the BU that can last up to two weeks (1 sprint). This engagement is highly target based, and is normally a portion of a larger project. This can be something like deploying Jenkins server in an automated and recoverable fashion, while teaching both the importance and steps in automating the deployment of code.
  • The second format is developing products/services that the BU’s can consume. These products are open source and are designed to provide benefit to all BU’s and potentially organizations outside of HBM. Some examples are the SLAPI bot framework for Slack, which allows anyone to create plugins in the language of their choice. We also centrally host Chef for configuration automation at all of the BU’s. We are looking to build and manage more product/tools/services for the businesses based on their feedback and needs.

Q&A

  • Q: Why did you put this blog up? What on earth were you hoping to accomplish?
  • A: That’s a great question. Let me break the answer down a bit
    • 1. I love to try new things. I love to experiment with process, tools and approaches. I don’t ever want to stop learning and trying new experiences.
    • 2. I want you to feel like you are interviewing me. You are, after all. You have the right to say no to me as much as I to you.
    • 3. When you work with me, you and I work as a team. This means that I want to hear your feedback. I want you to tell me when you don’t like something, don’t agree with something and/or have a better idea/approach/dance move. I don’t want you to feel intimidated or that you must be a silent objector. There will be times when you we are going to take a direction that you don’t like or agree with. My only request is that you don’t have to agree, but you must support the decision with the rest of the team. This give and take will allow us to all grow together, and if it blows up, you can always just say it was Pauly’s fault.
  • Q:Will I code?
  • A: Yes, you will. While it will not be the primary job, there is definitely dev work involved. We work mostly in Ruby and Node, although we are a small and flexible team willing to use the right tool for the right job, without falling into the “shiny object syndrome”.
  • Q: Will I need to know a lot Ops role type stuff?
  • A: We support both Dev and Ops, so a strong understanding of how infrastructure and support/monitoring of that infrastructure works is needed. This doesn’t mean you need to know everything about everything, but you MUST have an open mind and a willingness to learn.
  • Q: What would you say your leadership style is?
  • A: I believe in guidance rather than guiding. I hire, intelligent and driven individuals that form great teams. I trust my team to do the right thing and I work with the team daily. For the most part, I just help set the strategy, enable the team and then stay out of the way. I do enjoy coding still, and I work with the BU’s directly on a daily basis, but I prefer to set a target and let the creativity of the engineer shine through. If you are someone that likes to have tasks assigned on daily basis, you will most likely not enjoy my style.
  • Q: I see you have “DevOps” in your title. I thought you didn’t like DevOps as a title? What’s up with that? Are you a hypocrite?
  • A: You are quite right and astute at pointing that out. To be completely clear, I do not like DevOps as a specific engineering role, team or title. Especially if you are simply rebranding an Ops Engineer as DevOps but not actually embracing the ideals and tenets of DevOps. To take this further, if you are creating a team thats sole purpose is to exist between Dev and Ops or somehow operates as the “devops approval committee”, I believe you will miss the real value of DevOps. My role is not any of these. I am operating as a support for businesses that are both wanting to start a DevOps type transformation as well as those businesses that are somewhere along the path.
  • Q: I’ve watched some of your talks and it seems like you think you’re funny or something. I don’t want to work with someone that doesn’t take any of this seriously.
  • A: I take what we do seriously, but I don’t take life too seriously. I like to joke, laugh and have fun. This is never more true that in the toughest and most stressful situations. If you are someone that doesn’t like to have fun at work, I could see me being a little tiresome. I recommend you reach out to some of my references (see below) and ask some very honest and thought provoking questions.
  • Q: Where do you see yourself in 5 years?
  • A: Ideally the Czar of some smallish coastal nation, where the water is blue and clear and the food and drink is plentiful. If that doesn’t pan out, I hope to still be leading strong and passionate engineers through the dragon filled world of software development.
  • Q: Ok last question, why would I want to hire you as my boss?
  • A: I only have one super power, and that is creating great teams. All of the success I have had since starting in management has been from the amazing and talented people that worked with me. They deserved all the credit then and they still do. As your “manager” I will make sure you have what you need to be successful, I will put your needs ahead of mine (but not the teams) and I will always give you all the credit. I love my job and I hope that it shows in those that I have the honor of leading. If all of that sounds too gushy, then look at it this way. I love to code and do Ops type stuff and more than that I love having fun with life. Want to do some cool and interesting things while having a great time doing it? I’m your pick then. Want to sit in a dull cubicle where you have to punch in and out on your phone when you need to go to the little engineers room, then you probably should keep looking.

   If you have more questions that you think would be beneficial to anyone looking to apply, please post it in the blog comments and I will add it to the document.

Pauly’s Resume

  • VP DevOps, Hearst Business Media 2014-Present
  • VP Operations, Chef (formerly Opscdoe) 2011-2014
  • Director Operations, Silverpop 2007-2011
  • Senior Engineer, Intelsat 2000-2007
  • Misc 1200BC-2000AD
    • Ran several of my own small businesses. All were a blast, especially my very first BBS back in the day. I used Excalibur just so I could play LORD with my friends (if you know what LORD stands for you are as old as me, and we likely went to the same prehistoric school together). Ask me about the time I used Dry Ice to keep the servers cool in our closet.
  • REFERENCES: Upon Request. Seriously, ask folks I’ve worked with before what they loved and hated about me.

I am an avid reader and always have been. I don’t feel comfortable if I’m not reading something. I typically read two books at the same time. Well not exactly at the same time. I alternate between a book of learning and one of pleasure. I also have a tendency to read the same books over and over. They are like old friends come to visit, often unexpectedly.

Current Book of Pleasure:

I have to thank my dear friend Soo Choi for suggesting this book series to me. After a long overdue catch up, I mentioned that I was looking for something new to read. She suggested The Name of the Wind by Patrick Rothfuss. It is the first in a series of books called The Kingkiller Chronicles. I must admit I had never heard of this book before but after reading the jacket, I was intrigued. There are many reviews of this book, and the subsequent books in the series, so I will not cover the details of the book itself. Plus I hate spoilers and there are few good twists and turns in this book. Instead I will give you the basics and then try to describe what I perceive as the style of the book.

The protagonist in the book is a man named Kvothe (pronounce like quoth but with an inconvenient v stuck at the beginning). We meet Kvothe as a man, but the majority of this book (and I assume future ones) covers Kvothes life growing up as an entertainer in a traveling troupe of players. He faces many trials and tribulations along his path to becoming an Arcanist. The method the author uses to explain the “magic” is particularly interesting to me. I like it because it doesn’t leave it totally up to faith in the part of the reader. It’s not magic the way the most people think of it. It is mastery of the Arcane forces within the world around you. Kvothe the adult is a notorious magician, accomplished thief, master musician and infamous assassin but the real treasure of the story is how he came to be these things.

If you are fan of lore such as J. R. R. Tolkien or Frank Herbert you will be disappointed in the lack of depth regarding the world around Kvothe. The book reads like an enjoyable yet shallow approach to story weaving. That being said I do recommend the book for when you want to escape into a little world similar to Harry Potter, where magic is real and not everyone likes that fact. There is of course opposition to Kvothe in the same vein as Harry Potter’s Malfoy, and this interplay is both entertaining and interesting. You will think multiple times (in my case, even say out loud) oh come on Kvothe, don’t do that you know it will end in disaster! Only to have Kvothe do it anyway (think Greg Focker in Meet the Parents). For being a brilliant arcanist, he makes some pretty boneheaded errors.

If you have read it, I would enjoy your thoughts on the book as well.

 

 

 

 

I was asked a few questions for my upcoming talk about DevOps Middle Managers. I would like to share them and my responses below. I would love to hear your answers!

 

In one sentence, how do you define DevOps?

  A culture of bringing teams and people closer together by encouraging the core behaviors that make us more human.

What challenges remain as part of implementing DevOps in your organization? How are you addressing them?

  Bringing the periphery to the center of the change and making it about everyone.

If DevOps was an Olympic event, which event would it be and why?

  Relay racing. Everyone must do their own part (accountability), however there must be a high degree of trust and faith in each other. The handoffs must be aligned, smooth and muscle memory.

Tree_Handshake

I’ve noticed a trend amongst all the companies I work with and have interviewed, over 30 in all. That trend is the breakdown of communication, collaboration and empathy. The intent is not normally centered around vitriol but rather it is a lack of understanding and willingness to understand. The two groups at the center of this trend both want to be great at their jobs and want to do what is right for the organization. They genuinely want to be seen as helpful and successful and always doing the right thing. The problem is that the “right thing” for each team in a vacuum is not always the right thing for the other. Aren’t we far enough along in DevOps to have seen most of this be resolved? Most of the time my answer would be yes. In this case, it is definitely yes because it isn’t Developers and Operations that I am referring to. It is the business and developers.

DevOps is (for me) at its core about culture. It is about building relationships that allow sharing of knowledge in a transparent and open fashion. Since we often judge ourselves on our intentions and others on their behaviors, having this relationship smooths out the transition from misunderstanding to alignment. Knowing a person before a conflict helps us to understand their intentions and have the freedom to ask when we don’t know. This is as true for DevOps as it is for any inter-team relationship.

Defining “The Business”

For the sake of this blog post, I’m splitting the business up into three parts.
1. Business Leadership (CEO, CIO, CTO…)
2. Finance
3. Product Management
There are certainly other representatives that make up the business, however these are the three that seem to come up time and time again with my conversations with organizations. We are focusing mostly on the parts of the organization that help feed feature requests into the pipeline and require specific forecasting from developers. They are also a major part of the overall Value Stream Map.

The Core Problem?

There are really a couple of issues that are contributing to the bottleneck between the business and the IT teams. I will address each one as I have experienced it.
1. The business doesn’t understand the IT teams and the IT teams don’t understand the business.
2. There is a distinct (and often overt) lack of empathy between the teams.
3. There is little to no communication between the teams, and what communication that does exist is done with minimal effort.

Lack of Understanding, Empathy and Communication

In my interviews with product managers, business leaders and developers I am often met with statements like the following:
Lead Developer: “Product Managers have it so easy. They only have to take customers out for drinks and chat and then promise them the world. We are the ones that then have to make those unrealistic dreams come true with no additional help, money or time. Must be nice to have such a cush job.”
Product Manager: “The developers are such whiners. They don’t have to deal with the hassle of managing customer expectations or travel and they get all the best stuff. They only have to write a few lines of code, and do so in the leisure of their homes half the time! They have no idea what a real job is.”
Business Leader: “We’ve got a huge customer wanting this product/feature. We have to make this deal or we may not make our numbers for the year. We have done something similar in the past so I know we can do this again. Sometimes we have to work weekends and sacrifice for the good of the company.

This lack of fundamental understanding of what other teams do, how they do it and how it contributes to the overall success of the organization is wide spread. Even when teams attempt to share this information it is lost in the fog of war, and ends up creating more mistrust and angst. A large contributor to the problem is the lack of empathy across teams. It can be very easy (and tempting) to arm-chair-quarterback. It is very important to work towards creating a culture where we respect the difficulty and contributions of other teams. One of the best ways to begin this journey is by working to get the lines of communication open between teams.

Ok, yes that’s obvious, but how do you actually DO it? Most of the blog posts I see are really about high level problems like these, and they stop at the point of telling you what you already know… your culture is broken and it doesn’t feel right. So let’s talk about specific ways I’ve seen work at tackling these problems.

Consistently Missing Due Dates

I’ve gotten a lot of feedback from leadership regarding their concerns for missed due dates for products and/or poor quality around products. The business leadership really doesn’t understand how this could be happening. In many cases it stems from “scope creep” during the sprint. We see the work being done during that sprint growing well beyond the original planning, with product managers pushing more work into the pipeline mid-stream. There is no room or authority for pushback on the product managers, so in order to accommodate the additional work, corners get cut. The incentives drive the behavior, and in this case it is being judged on how many features are getting to market. Without including a focus on quality and velocity, you end up with a similar situation to teachers being judged on grades achieved rather than knowledge learned.

The business must focus on proper prioritization of features, projects and products first. Accept the tradeoff between work that can be accomplished and set quality standards and resource constraints. Continuously bang the drum of “slow down, do less, accomplish more”. This will keep the scope creep in check and allow for the completion of projects with a higher degree of quality. You can start with a time-boxed approach to this, like an experiment. If you are tracking your velocity and other metrics prior to the experiment, try it for a few sprints and compare. Don’t forget to include metrics outside the dev teams too, such as support cases and customer feedback.

Demo Day

Performing demos at the end of sprints can be a great way of showcasing achievements and progress. It is also a great time for feedback on the product and process. However, in many cases the dev teams were not inviting anyone outside of their scrum teams. When I asked why there was no one from the leadership team in the demo meetings, I was told they didn’t really get anything from it. When I talked to the leadership, they stated that they didn’t understand the technical details of what was being said.

I took the approach of the TL;DR and Executive Summary. I worked with the developers to have a short session at the beginning that was more high level and based more on functionality. We took an Agile Standup approach to it as well:
1. What were the inputs and requirements for this sprint as understood by the devs.
2. What was accomplished and then showcased.
3. Asked for feedback from the leadership.

We iterated over this approach fine tuning the inputs based on the feedback from the demo. In many cases we started to recognize that the requirements from the business were not aligned with what the devs thought. These sessions included leadership, BA’s, Product Managers and the devs. We began to have more open transparent conversations and work towards bringing the BMVP (Business Minimum Viable Product) and the TMVP (Technical Minimum Viable Product) to an aligned MVP. It took work from all teams, but the results were amazing. No one was intentionally holding information from others, but that was what was happening. Breaking down the communication barriers meant we had a unified team from input to product.

More coming in Part 2…

After much prodding from others about not doing an adequate job sharing what I’ve been learning, I created this simple blog. I will be posting multi-part blog posts around the challenges I have been facing in deploying DevOps and Agile. My goal is to share not only the successes around culture, process and tools but also the missteps and mistakes I’ve made along the way! I am always welcome to feedback on the blog posts and the ideas. The best way for me to grow is to be open to others ideas and opinions, especially if they are different from my own!