Humpty Dumpty sat on a wall,
Humpty Dumpty had a great fall.
All the king’s horses and all the king’s men
Couldn’t put Humpty together again.

That sure sounds like the last time your site went down and every team in the office started panicking, right?

When things go wrong, everybody starts pointing fingers. “The developers are saying it’s a server issue, IT is saying it’s a code issue. It’s absolute chaos,” says Emily Dowdle, of Wazee Digital.

The real truth? No-one is quite sure what’s wrong. Otherwise, they’d already be doing what’s needed to fix the problem and get the site back online. In many cases, the problem is miscommunication: person A knows something that person B doesn’t, and that’s preventing person B from coming up with a fix.

The idea behind DevOps is that facilitating interactions between teams, particularly development, operations and QA, can alleviate some of that chaos.

Silos Are Bad For Business

There’s a problem with the modern workplace that not many people talk about. Namely, that everybody works in silos. And, where that’s the case, no amount of teambuilding exercises or open office space will change things.

Let’s talk about developers and operations: they don’t always get along. Emily Dowdle, of Wazee Digital, describes the situation like this:

“There’s tension, could be described as a friction or an attitude … a general inability to tolerate each other without eye rolls and audible sighs.”

She talks about how the Operations side of things is evaluated not by the features they release, as developers are, but by quantifiable measurables like site uptime. Ops teams strive for five nines (or better) uptime, i.e. a site being up for 99.999% of the time, which works out to 5 minutes a year of downtime.

“Fundamentally, we have different priorities. It means we have conflict, we butt heads. It’s no wonder we’re natural enemies.”

But it doesn’t have to be this way.

Key DevOps Concepts

With the rise of agile development – which, of course, results in more frequent releases – embracing DevOps as a concept is not just advantageous, but a necessity.

But – like “big data”, “agile development” and other mots du jour – DevOps is such a large, often poorly defined, concept that it’s easy to get lost in.

Dowdle has a few ideas for practical implementation of DevOps in the workplace:

1. Bridge the skills gap

Acknowledging that development and operations teams don’t work in the same way, and don’t have the same priorities, is key to making changes.

No developer is ever going to fully understand the ins and outs of operations, and vice versa, but some comprehension of why the other team would want to take a certain approach is extremely valuable.

2. Share information

Everyone hates meetings, but they do have their uses.

If a new feature is in the works, it’s important for development and operations to discuss the safest way to iterate without breaking… well, everything.

From the other side, it’s helpful for developers to have access to read-only logs that let them see the impact of their deployments and any other changes they’ve made. Adding ChatOps, colorfully named so by GitHub, could be a means to better cross team communication.

3. Consistency

Whether we’re talking about the process of deployments themselves – which should be such an intuitive process that anyone on the team can do it – or how closely a staging area replicates its live equivalent, consistency is key.

Making small, incremental changes – no more bloated feature releases – is the best way to make sure everything stays recognizable while still moving forward with development timetables.

Continuous integration/delivery is, as we’ve written before, here to stay. A cornerstone of agile development, it’s also incredibly useful for trying to keep things as consistent as possible.

4. Increased accountability

In most workplaces, teams are guilty of passing the buck sometimes.

Sales and marketing teams might close deals on projects that don’t technically exist in the product yet (but the devs can take care of all that!), while developers might make a deployment that really needed more QA in order to meet a deadline.

Including devs in on-call rotation, i.e. involving them in that 4am “the site is down” conversation, is one example of a way to change the way developers think about deployments.

Again, this is as much a culture thing as it is a process thing: in her presentation, Dowdle talks about the fear of heads rolling when something goes wrong, and that fear makes people want to divert responsibility.

Making sure everyone is up to speed with best practices (e.g. security) in the workplace is key, but it’s just as vital to make sure that employees are able to do their job without worrying that one mistake will cost them it.

5. Embrace failure

Let’s think about those mistakes a little more. So the site went down? That’s bad. But it doesn’t need to be the end of the world.

It’s more important to figure out what went wrong, which automated alerts can seriously help with, and figuring out how to make sure it doesn’t happen again. Assigning personal blame isn’t helpful, but enabling a team to see where they went wrong definitely is.

Expecting and preparing for mistakes or accidents, and not setting unrealistic uptime targets of 99.99999%, can help to ease some of the tension between development and operations.

If (or maybe that should be when) you perform a postmortem on what went wrong, figuring out why and whose fault it was — without a lot of pointing fingers – is the best way to make sure that it doesn’t happen again.

DevOps and APIs

Some of the problems outlined above become even more significant when APIs are thrown into the mix: According to Dowdle, in most cases of API problems, there will be at least two development teams and two production teams in the mix.

In that respect, APIs actually offer a great jumping off point for an approach that takes DevOps more seriously; when it comes to creating a great API, creators are keen to avoid issues because they know that multiple teams will inevitably be involved.

That’s especially true if we’re talking about a public API, in which operations and testing are even more critical than one designed for internal use.

As a result, the following are key in successful API development:

• A very specific approach to requirements e.g. parameters, response codes etc.
• Thorough documentation to walk through a number of potential problems
• Need for rigid testing frameworks that functions exactly like what’s live

What API developers don’t necessarily realize is that everything they’ve applied to the development of their API is the perfect recipe for a balanced DevOps-style approach to building a core software offering too.

So, next time you’re working with your team on a new feature, ask yourself the following question: “is this how we’d be doing it if we were making changes to our API?” If the answer is no, then you might find the concepts above useful to consider.

Dowdle doesn’t come right out and say it, but her strong focus on documentation, interdisciplinary communication and parity between environments means that it’s difficult not to reach this conclusion: the way we build APIs is an effect a model for the entire software ecosystem.

Guides to API Development Management

API developers aren’t always given the same freedoms as traditional software developers. Or perhaps we should phrase that another way: API development is typically directed by feedback and requirements, or a product backlog.

In some – but not all! – workplaces, developers make additions based on what marketing/sales asks them to add, what features THEY think a product should have and sometimes (let’s be honest here) what’s trending that week on HackerNews or Reddit.

Let’s consider the following three diagrams for slightly different development models:

Accenture DevOps diagram [Source]

IBM continuous integration process [Source]

Microsoft development management cycle [Source]

You’ll notice that, while the exact order and the specifics may be a little different, the overarching ideas are very similar – requirements are designed, deployed, tested and feedback is gathered, combining both the development and operation sides in synchronicity.

The fact that the Accenture API management framework is so similar to other DevOps templates only seems to further the idea that the approach adopted during API development – which typically involves rigorous testing and continuous feedback from Operations and/or those who use the API – effectively offers a valuable DevOps model.

Putting Humpty Back Together Again

The bottom line is that much of this comes down to culture: a DevOps approach, or one that mimics the style of API development, won’t be successful if your Development and Operations teams are completely unwilling to co-operate.

The DevOps mindset isn’t a new one but it’s one that, while hugely valuable, can still be difficult to adopt because of problems with coordination or company structure. Still, it’s one that’s increasingly important as agile development and continuous delivery and continuous feedback becomes the norm.

Adopting an approach that works is even more critical when it comes to APIs because, when API development is done badly, it’s not just your own business that’s at risk; you’re also compromising the businesses of everyone who uses your API.

And if there’s one thing harder than putting Humpty back together again, it’s putting a thousand Humpties back together again with an angry mob shouting at you the whole time.