Article image

The reality of DevOps

DevOps has become a subject for fictional writing about the blending of development and operational activities in Software engineering, and the advantages it can achieve. The reality is far from these illusions and not as rosy as people want to believe. Those who have real experience from manufacturing know what DevOps can turn into, when knowledge leaves the organisation, and software systems are not regularly upgraded. But they also know what it could be.

Flexibility and change is much more important In manufacturing today than it was thirty years ago. The reason is seen in shorter product series, a wider range of variants, and more general uncertainty in product development. Products last shorter time on the market, more variants are launched to be on the safer side, and technology makes flexibility possible, even desirable. Flexibility, and especially change, costs money and is harder to achieve if a manufacturing organisation wants to outsource parts of its manufacturing. There are written contracts that stipulate what a change means in terms of cost and time to deliver. Sometimes contracts need to be redrafted, a process no manufacturer voluntarily wants to go through. Subcontractors also have a tendency to make a low bid the first time and set a higher bar for subsequent changes in the manufacturing flow. When a subcontractor is deeply embedded in the manufacturing flow there is little a manufacturer can do.

Manufacturing does not want to tell how reality looks, but the software and hardware that controls assembly lines and logistics is from the ‘80s. The fact that thirty year old software and hardware drives plants is what makes Hewlett and Packard show a profit. Computer systems from the ‘80s need to be kept online or it will cost the manufacturer millions every hour they are not. Neither software, nor hardware, can be upgraded after thirty years. The software is undocumented, which means it is void of documentation, and the knowledge that exists rests with a few men and women in their 50s. IT Operations can execute the software, including start and stop, but they have a hard time finding and correcting software bugs. There is, however, no shortage of clever explanations when a bug shows up. The systems were built once and maintenance activities have slowly degraded. Faults and failures are swept under the rug, as no one knows how the software really works.

Article imageThe enterprise market for applikations has slowed to a halt the last five years. Oracle grew 1% during 2017, although Mark Hurd can see a new demand for enterprise applications. It remains for everyone else to see. A no growth in enterprise applications means that customers are not upgrading their systems. Having a thirty year old problem explains only part of the picture; manufacturing companies have not had the operating margin to be able to invest in new IT systems, even if the ambition existed. They are keeping the old IT systems in the basement, hoping that nothing will go wrong, and upgrades only the systems that are on the verge of collapse. Other problems are the CEO who doesn’t see DevOps as an asset, but as a cost. Even if manufacturing companies wanted to upgrade, it wouldn’t be easy. There is a legacy, both in terms of operations and people, which consultants and vendors can’t disregard. Building a completely fresh system from start has a tendency to end up looking like the old system. Architecture is old, maintainability is old, error reporting is old, resilience (which is pretty important in manufacturing) looks like it comes from the ‘80s. The operators who know the old system have little, almost zero, acceptance for something different. Other parts of the problem is Management, who has a tendency to go for easy solutions. If the solution fits on a powerpoint presentation, then it must work. DevOps in manufacturing isn’t sexy and it doesn’t build a career; it’s where you end your days, and people rather leave to work in purchase.

With time, DevOps becomes a patchwork. There is probably a right way to do DevOps, but the examples that exist are horrifying. Looking at organisations that have practiced DevOps for thirty years, some even longer, the results aren’t better. Employees become experts in avoiding responsibilities, to find something or someone to blame. Pretending to work has become a virtue because of the way operations are organised — there is really not that much to do. Furnishing the organisation with new knowledge is also not easy, and the few who can shuns the place. Everyone who stays develop a passive-aggressive behaviour, a kind of environment no one in their right mind want to work in. Having the ambition to improve the infrastructure or the processes is hard; the acceptance is not there and will be met with indifference or outright hostility. The whole workforce lacks in education and can’t see why an improvement should be made. New ideas are measured with yesterday’s yardstick.

When DevOps is outsourced, the company also outsources its entire knowledge of manufacturing processes and how they can be improved. To see improvement opportunities and develop strategies that work, operators and engineers need to have deep insight. The strategy for error logging is a de facto strategy, adapted from the first systems that went into service decades ago. Which makes maintenance a detective work. It is hard to know why a fault is logged and what to do to correct it. And that is only one example. Fault recovery is another. The truth is that jobs in DevOps are bottom line jobs, the kind of jobs that must be done to deliver something else. Top line jobs are about new ideas and new products, the kind of projects that get the attention from upper management.

Politics is a reality in every organisation. It’s about my job, my promotion, the attention I get. The politics are especially severe in large organisations, and people may never quite know why outcomes are the way they are, but politics can also exist in the playground amongst toddlers. Politics is especially severe when the organisation lacks in education, or the money is not there, but it can also be a telltale sign that the organisation can’t hire the competence it needs. During such circumstances, it’s important for ambitious persons to understand that they can’t change things without becoming a victim of history. Under such circumstances, the good people leave while the bad stays.

DevOps in manufacturing should be about metrics, which it isn’t today. Manufacturing processes should be measured in order to raise immediate alarms, even if manufacturing necessarily isn’t halted. The measurements should also be analysed for long-term trends: are quality of operations random or dependent on external causes, and can operations do anything about it? Measurements can be analysed with Statistical Process Control (SPC) and other tools that ought to be common in manufacturing. Artificial Intelligence should be applied to the analysis process; the analysis work can be highly automated. Money will be saved in the end and new knowledge generated how to build the plant of the future. Reality speaks, however, differently. To implement all this technology and new way of working, competence can hardly be found. There is simply no demand for it.

DevOps in manufacturing should also be about mobile. Engineers are used to build new applications like they are Facebook, despite that only a handful of personnel will use it. When the maintenance engineer carries an iPad, it would be more appropriate to build an iOS application. Instead of having to explicitly take care of credentials in the application, engineers let iOS take care of everything. But the competence is not there, neither to build or maintain it.