Gene Gotimer

Hi everybody. I still consider myself a developer, but nowadays I spend much less time writing Java classes than building CI/CD pipelines and writing Chef recipes. I spend most of my time at federal government clients, but I have worked with plenty of commercial clients as well. Enough to know that both commercial and government are generally their own worst enemies.

What types of problems are people generally running into when getting started with DevOps?

Don't know where to start 7

Lack of tools

No management support 2

DevOps won't work here 2

So far, "don't know where to start" seems to be popular in the poll.

"A journey of a thousand miles begins with the first step." - Lao Tzu

@Alan Crouch's "just starting the transition to DevOps" question is along those lines, and my advice would be about the same. Do a value stream analysis, ind your biggest constraint, address that, and then your next biggest constraint is your biggest constraint.

Also, starting by implementing some basic CI tools (Jenkins, Sonatype Nexus Repo, OWASP Dependency Check, SonarQube) will give you a good start to the pipeline tooling. It might give you some room to breathe will you start to tackle the cultural changes that you'll need for a successful DevOps journey. (See my answer to @Tom Stiehm's "DevOps beyond CI/CD" question.)

Kelly M

We received this question via survey monkey: Who would you include in efforts to create processes and policies for delivering applications in a DevOps environment?

Gene Gotimer

If you want all the benefits from DevOps, it has to be everyone in the organization. But "everyone" never seems to be an option when starting out, and probably wouldn't work out even if it was.

I'd start with some incremental change. Decide what part of your process is the biggest or most obvious (or most frustrating) roadblock. Then, figure out who is in charge of that and see if you can help them make it less of a roadblock. Maybe you can automate it away, or automate something else so that they have time to work on the roadblock. Maybe you find out why they are blocking the process and offer an alternative way to get them what they need.

By the time that roadblock is removed, you have an ally (or group of allies) that can help evangelize the benefits of DevOps and help you tackle the next roadblock.

owen

Hey @Gene Gotimer! Can you start DevOps without being agile?

Gene Gotimer

Sure, as long as you set expectations that a) it will be a little slower, and b) that is because you will be adopting agile practices and culture at the same time that you are starting with DevOps.

DevOps and agile aren't separate. The same principles of agile from the Agile Manifesto (https://agilemanifesto.org/principles.html) show up throughout DevOps. In fact, DevOps could be considered a focus on "Our highest priority is to satisfy the customer through early and continuous delivery of valuable software" and an extrapolation of "Business people and developers must work

together daily throughout the project" to include testing and operators as well.

The key for both is to understand that you are changing culture. Tools and automation will help, practices will keep you most disciplined and focused, but unless you change the culture to be more in line with agile principles, you won't get far with DevOps or agile adoption.

msowers

Good Afternoon @Gene Gotimer Thanks for being here. What's the best approach to integrating security into my CI/CD? Not just the tools, but also getting my security team engaged, educated and on-board?

Gene Gotimer

I almost always start with an incremental approach, trying to tackle the biggest blocker first. In fact, a lot of this answer is like my answer to the "Who would you include" question above.

So what is the security team most worried about? Is there some type of testing they are missing that they think will lead to a serious problem? Or are there some simple things that they just don't have the time or tools to address right now? If you can help make that less of an issue for them and free up some of their time, now they can be invested in helping you with the next biggest security concern. They are more engaged because they see what's in it for them.

If you don't have any "biggest" problems, or you have so many that it is overwhelming, I'd suggest starting with SCA, Software Composition Analysis (https://owasp.org/www-project-top-ten/OWASP_Top_Ten_2017/Top_10-2017_A9-Using_Components_with_Known_Vulnerabilities). SCA tools are easy to get started with early in the pipeline, remediation is straightforward and doesn't usually require much coordination with any other groups (just security and development), and can start making the application/organization more secure very quickly. Tools like OWASP Dependency Check, Contrast OSS, or Sonatype Nexus Lifecycle are all examples of tools that can be adopted in the CI/CD pipeline that will make future security efforts easier and more effective. Hopefully, that leads to the security group getting more involved with CI/CD.

Byron katz

@Gene Gotimer Let's say I'm starting a new project and have buy-in from the company to go all-out with whatever I need for good practices, no matter how seemingly-exotic. What might I expect to see that's unusual compared to my more traditional background?

Gene Gotimer

I think the biggest result that many traditional waterfall orgs are surprised by is that, done correctly, more speed can mean less risk and higher quality.

Remember the iron triangle (Fast, Cheap, Good- pick two)? If we keep the budget and people the same, we could deliver garbage faster or quality slower. At least that was the popular thinking. But CI/CD has shown us that if we most more quickly in smaller increments, we get more feedback and our process (and results) get better faster. Plus, we spend less time working on things we don't need. "Simplicity--the art of maximizing the amount of work not done--is essential."

The other big change I think would be appreciating failure. Many traditional orgs avoid failure at all costs. They are willing to spend huge amounts of effort to make sure the smallest things don't fail. Mature agile orgs are much more of the mindset of let's try it and see- if it fails, we'll learn from that. As long as the experiment is cheap (time, resources, effort, blast radius), those failures could be really good for our growth and continuous improvement. Along the lines of Edison developing the light bulb- “I have not failed 10,000 times—I’ve successfully found 10,000 ways that will not work.”

Kelly M

This question was asked in a previous slack takeover, but would like to hear your thoughts on it:

What do you think are some of the top security concerns for DevOps/DevSecOps during the COVID-19 crisis?

Gene Gotimer

I think the pipeline and AppSec security stays very similar. The two biggest things I see are a) the need for better widespread access to the pipeline and tools, and b) people forgetting that when working from home, home is a workplace.

For widespread access, companies that had tools in the cloud, that allowed employees to just log in easily, that could share information throughout the org (or possibly further), all those companies had an easy transition to a remote workforce. People already had VPNs and access they needed. They didn't need to jump through jumpbox after jumpbox and don't constantly get logged out of systems after a few minutes just because they are accessing it from outside the building. Overhead like that might have been tolerable for the occasional work-from-home day, but not when every day is WFH. Setting all that up securely in a rush isn't easy, but it is now a requirement if you want your remote workforce to be effective.

From the employee side, people working from home have to keep in mind that their home is now their office. WFH doesn't mean your work laptop can also be used for surfing the internet, sharing with your kids, playing video games. It doesn't mean public sharing sites are suddenly a good place to store business documents for convenience since your home network doesn't block them like your office firewall did. Much of the OpSec around being a good, secure, remote corporate citizen boils down to not doing things at home with corporate equipment and data that you wouldn't have done sitting in the office with the possibility that a security person would catch you doing it.

Kelly M

Thanks! That is hard for me to remember, especially with my husband also working from home, that your workplace is your home and your home is your workplace.

Gene Gotimer

@owen mentioned the other night that he no longer wants to come over to work at my house during the day, even for a change of scenery (he lives about 5 minutes away). To him, since his home is his workplace and he is stuck there so much now, my house becomes one of the few places he can hang out at and it is not a workplace. (edited)

Kelly M

haha!

Jeanette Kreutner

What do you do when your company is talking Ci/CO when they don't even have use cases/requirements covering much your company's architecture?

Gene Gotimer

Good question. The requirements and use cases exist, but maybe they aren't written down formally, or at all. You can almost always tell that someone has a good feel for some things the systems should do or maybe shouldn't do. An easy way of identifying some of those is just to offer to turn something off, or unplug it from the network, or cut it's memory in half, etc. Someone will inevitably scream "We need that because...". Boom, you've got a requirement or use case.

Of course, that type of poking and provoking isn't going to work for long, is hit-or-miss, and won't make you many friends. But it can jump start some requirements gathering and discussion around the use cases. "How would it affect our business if this feature wasn't available for 4 hours?" Make that a lunchtime discussion with a group of people from different teams, and you'll start getting your requirements. And when other groups/managers/divisions/etc. start hearing about it, hopefully they'll want to be part of the discussions, too.

In parallel with that, look at your current delivery pipeline- however you deliver software now. Doesn't matter if it is automated or manual, reliable or not, fast or slow. Just start listing out the steps it takes to get a feature from idea to delivered. John Willis calls this the A-ha to the Ka-ching. List out the times it takes for each step. When you start seeing steps that don't make sense, don't add any value, or just slow everything else down, look at starting some CI and automation efforts there. Once that isn't the biggest issue, tackle what ever is.

When you have a question about whether you need a step or not, or if you can use a workaround or a faster method in it's place, refer back to the requirement discussions you started having. Or just make an executive decision to change it and see who screams and find out why. Maybe it is a valid reason, maybe it is kingdom building from someone, or maybe it is just fear of changing the way you've done things in the past. With some empathy and education, plus small enough experiments with changing that keep the potential blast radius to a minimum, you should start making some progress towards CI /CD adoption.

Jeanette Kreutner

This really helps. My experience with software engineers is that they think the latest tool/language/process will solve the problem without having to do the work of filling in the blanks. This will give me a way to accomplish both at the same time

Gene Gotimer

"We got Jenkins installed. Are we DevOps now, or do we have to install Docker too?"

Jeanette Kreutner

Falls into the same thinking that if not writing code--not doing real work--thanks again!

Alan Crouch

Hey @Gene Gotimer, I'm curious what are your recommendations to a customer who is just starting the transition to DevOps?

Gene Gotimer

Value Stream Analysis

After that there is a big gap to continuous integration, automated deploys, other tools and practices...

The value stream stream will show you where you are investing and wasting your time. Even if you do a lite version- just list out the steps it takes to go from idea to delivery, John Willis' "A-ha to Ka-ching". Be as detailed as you can. Beside each step, mark how long you spend waiting for it (wait time, which is waste) and how much time you spend doing it (work time, which adds value). Those times can be actual times, but you can even start with t-shirt sizing for it- this step takes seconds, this one minutes, this one days, this one weeks, etc.

That will point out your biggest bottleneck. Goldratt's Theory of Constraints (https://en.wikipedia.org/wiki/Theory_of_constraints) says that you have to address the biggest bottleneck first. If you optimize before the bottleneck, you'll just get to the bottleneck quicker. If you optimize after the bottleneck, you'll just be waiting longer for work to clear the upstream bottleneck.

Evaluating your value stream will be an on-going process. It isn't a once-and-done task.

I might make some exceptions to that by attacking a visible but easily solved bottleneck just to get a quick, visible win and build some trust in the change process. But largely, you have to focus on knocking down the biggest constraint until it isn't the biggest constraint anymore. Then go after whatever the new biggest constraint is.

My other recommendations:

Find a champion with enough clout to remove (or go around) hurdles.

Focus on code quality early, using continuous integration and static analysis.

Don't forget to start doing security and performance testing (and any other non-functional testing you can do) as early in the process as possibly- if you leave them to the end you won't have time to fix any problems they find.

Understand and repeat often- DevOps is a culture change, not just adopting certain tools and practices. If you don't change the way you do business, you'll end up with the same results and problems that you have now.

Tom Stiehm

@Gene Gotimer How do you explain DevOps beyond CI/CD, what other things should be included in as you get your org to adopt DevOps?

Gene Gotimer

So not the simple tools and practices, but the important stuff that will actually make a real difference?

I forget who at Lean+AgileDC said it, but someone presented a quote to the effect of "Your organization is perfectly optimized to get exactly the results you are currently getting." In other words, if you don't change the way you do business, you won't get different results.

We sometimes joke that we get called in as consultants by clients that are willing to do whatever it takes to get to DevOps, as long as they do have to change any of their current processes. Sometimes it presents itself as "We understand that works everywhere else, but it can't work here. We are unique." Yes, just like everyone else.

So all the "beyond CI/CD" stuff are the parts that will make a difference. The cultural changes are what will get you to systems thinking, amplified feedback loops, and a culture of learning. (The Phoenix Project's Three Ways). If you at least focus on developing a culture of learning, most of the rest will start presenting themselves as important soon enough.

owen

I have a personal question for you @Gene Gotimer. If you had to rank these in order from favorite to least favorite, what would your ranking look like: Star Trek, Star Wars, Lord of the Rings

Gene Gotimer

In true nerd fashion, ranking those 3 items will result in a list of 9:

Lord of the Rings books

Lord of the Rings movies

Star Wars original trilogy

Star Trek: The Original Series

Star Trek: The Next Generation

Star Wars: Clone Wars and Rebels

Star Wars: Rogue One

Star Wars 7-9, the most recent trilogy

Other Star Trek Series

I refuse to acknowledge that the other Star Wars trilogy exists.

Alan Crouch

No mandalorian?!

Gene Gotimer

Good point. The Mandalorian comes in above ST:TOS.

owen

He’s lying. He binges watches Star Wars 1-3 and complains all the time about 4-6

ArleneAndrews

I'll ask a huge one, @Gene Gotimer: what role(s) show be delegated strictly for third-party testing services, and which should remain in local control?

Gene Gotimer

In general I think you want to balance outsourcing anything that doesn't offer you a competitive advantage with maintaining control and responsiveness. I first thought of https://continuousdelivery.com/2011/01/strategic-vs-utility-services/, but the Martin Fowler article referenced in there (https://martinfowler.com/bliki/UtilityVsStrategicDichotomy.html) is probably more to the point.

The goal of the delivery pipeline is to build confidence that you have a viable candidate for production, or identify as early as possible that you don't. If moving some testing to a third-party service slows down the feedback or reduces the fidelity, then it doesn't help. But if it isn't core to your business or if you don't have the skill set in house, then outsourcing might be a good choice.

But that's only part of third-party testing. What about Sauce Labs or LoadStorm or Blazemeter? You can outsource the testing infrastructure without giving up control of the feedback quality or timeliness. It isn't that different than outsourcing your operations infrastructure to AWS, Google Cloud, or Microsoft Azure. Why wouldn't you? They are better at it, have more resources, understand the requirements better, and allow you to focus on the pieces that are strategic to your business.

So I think I would almost always outsource something that you can't do, because someone else doing it is probably better than it not getting done. I would not outsource something that would slow down the feedback or make is less informative. I would outsource parts of any task or role that do not affect the quality of feedback, or that improve the quality. Outsourcing testing infrastructure (like functional testing on Sauce Labs or performance testing on Blazemeter) is a great example of something that actually improves the feedback without giving up control.

martinfowler.com

bliki: UtilityVsStrategicDichotomy

Separate software systems into strategic - enabling competitive advantage - and those that are utility. They have different success criteria ∴ managed differently.(102 kB)

https://martinfowler.com/bliki/images/utilityVsStrategicDichotomy/sketch.png

@ArleneAndrews Does that answer the right question?

ArleneAndrews

That it does! Now I need to figure out how to apply it to my learnings.

Ryan Kenney

@Gene Gotimer what are your thoughts on cloud hosted CI solutions (eg TracisCI, CircleCI, AWS CodePipeline) vs hosting a CI tool like Jenkins or Bamboo yourself?

Gene Gotimer

Funny you should ask. I'm hosting a virtual meetup next week and one of the speakers will be talking about exactly that topic.

https://www.meetup.com/meetup-group-YbHEIIJe/events/271040768/

Ryan Kenney

odd, what a weird coincidence

Gene Gotimer

I'd say this one fits with my answer to @ArleneAndrews about third-party services. As long as it doesn't slow down or degrade the feedback, I'd prefer to outsource anything that isn't core to my business. The CI engine could definitely be thought of as a utility, not strategic.

That said, many orgs don't like the idea of shipping their source code outside the walls of the castle, and balk at a lot of hosted services. The real (or perceived) need for control (which itself might be real or perceived) is too great for them to think they can let the source code out of their sight.

In those cases, you have to self-host. But my advice is to use as much pipeline-as-code and to spend as much money as you can afford on the infrastructure for the CI engine and build systems, since your pipeline will rely on them. They will be among the most important production systems you have, even if they live in development.

Some people shy away from pipeline-as-code thinking it isn't important or it will lock them into a particular solution. But while you might have to rewrite the pipeline code if for some reason you actually change CI platforms, the reality is the original code will be proven, written documentation of what the new code needs to do. That makes migration much easier. If you have to reverse-engineer what the UI configuration does, you'll have a much harder time.

As to the cost of the CI infrastructure, remember that every minute a developer waits for the CI system to finish is time that you are paying to at least one developer to twiddle their thumbs. People are usually much more expensive than even big machines. Also, slow machines mean less frequent feedback. Even worse, developers might wait to push multiple changes until they can batch enough to make waiting "worth it" which further reduces the feedback and increases risk.

Even if you self-host, do it in a cloud infrastructure, take advantage of dynamic system provisioning (like containers) to keep the cost of idle systems low so you can spend more on the systems you are using. Most CI platforms can use containers for build agents, which makes them fast, inexpensive, and brand new for every build which enforces a lot of discipline and saves a lot of headaches when subsequent builds pick up artifacts from earlier builds.