I sat through “one of those” meetings yesterday. The language being spoken was English, and the vocabulary sounded like words from the Agile dictionary. Yet I couldn’t parse what was going on, the words were stolen concepts, failing to match the actions and attitudes of the participants. So many attitudes from the old command and control IT mentality were surfacing that you’d swear that an “Agile transformation” was not more than a year underway, but instead had just begun.
This was a process meeting. A member of the DevOps team had asked on a number of occasions for RDP access to servers in various projects to enable deployments from Azure DevOps. Operations and Support, who are responsible for the care and feeding of virtual machines, were raising the concern that they didn’t know specifically what this DevOps team member was doing, and why they needed the access in question. So a meeting was called to develop a process for granting access and documenting server configuration changes.
The solution that the team created on the whiteboard was a set of paperwork to document the access and the changes: the who, what, and why; the preparation of a “run book” to document the configuration changes so they can be reproduced in downstream environments; and lastly, two layers of approvals. Future work included the possible addition of a configuration management product to monitor access and changes.
My skin bristled at this conversation. Why do you suppose? Was it because this new process creates more (tedious, manual) work than a simple touch of automation and a leaner, simpler, policy one-liner? In part, yes, but really what was the most disheartening was the lack of willingness to tear down the walls between project teams and Operations and Support. Participants still wanted to “own” tasks, and parts of the DevOps pipeline.
How would I solve this problem? I’m so glad you asked… Step One, a new policy:
“For application servers deployed by continuous deployment pipelines, no configuration change is to be made at any time except by scripting the change, committing that change to source control, and adding execution of that script as a step to the deployment plan.”
Bonus… there is no Step Two!
What are the consequences of choosing a simply worded policy over a set of policies and processes that are more complex?
- It deepens the experience and skill sets of everyone involved. The project team learns what sort of issues arise in misconfigured VMs, and Operations and Support get a chance to increase their skill in development and use of development tools.
- It adds auditability of who made the change, what was changed, and with proper commenting policies on check-in, why the change was made. Using a fully-integrated product like Azure DevOps, the check-in can be associated with a work item for the reconfiguration effort, further increasing auditability.
- In time, the project team can develop a trust and rapport with Operations and Support, eventually allowing the project team to self-serve for application specific configuration changes.
- If Operations and Support wish to maintain oversight of configuration changes made to servers, they can require pull requests for scripted changes as a manner of peer code review.
- Repeat after me: “The deployment plan is the run book.” One that is instantly updated without any effort or anyone needing to remember to change it. Some documentation is better than no documentation, but self-updating documentation is even yet better!
- Every time code is committed, the deployment plan is tested. For a development team that is following Agile principles regarding code commits, the deployment is tested dozens (hundreds?) of times a day!
- Instead of an engineer following a run book, and manually making changes, the deployment process takes over, reducing chances for error.
- Because the deployment plan handles change management, and you are leveraging Infrastructure as Code (ARM templates, or your Cloud provider’s equivalent) then spinning up a blank slate VM, configuring that VM properly, and installing your application should be as simple as cloning the deployment plan for a new environment and updating a few variables.
The moral of the story is that you don’t have to design complicated multi-phase processes with gates and approvals to achieve a sensible and secure state. Develop a process, diagramming it on a whiteboard. Find the step where automation gives you the biggest bang for your buck, and plan to automate it instead. Keep evaluating and planning for automation until it the effort to automate is no longer cost-effective. Distill the policies down to the simplest language that enforce the processes you’ve developed.
What are you experiences with policies and processes within your organization? Where have you had success with automation, and where did you experience problems? What sort of policies and processes work best in your environment? Please share in the comments section below.
In Starke Contrast 