Prioritizing frameworks and tools only put focus on the craft and the mechanical parts of producing software; highlighting communication skills, knowledge building and creativity puts emphasis on other capabilities, that can be hard to describe or measure but are at least as important for the end result. Much discussion is put on "working code" and delivery, but if the idea is flawed, or the design doesn't work, it hardly matters to have working code and automatic test suits. Feeding user stories to programmers is as bad as giving them a poorly written specification, especially when most user stories are written by project managers. It's a weird position when teams are to build capable products but don't care if their tools and processes are equally capable. The main problem is, however, how manufacturing thinking has impregnated the school of thought on software engineering, to the extent that processes have become counterproductive and projects have forgotten what engineering is really about.

Some drivers behind manufacturing thinking are rooted in simple economics, such as lean manufacturing, just in time delivery, and waste reduction. It all comes from Taylorism and scientific management, invented at the end of the 19th century and already obsolete as an industry practice in the ’30s, but very much in people's mind to this day. Pursuit of economic efficiency is essentially pointless when a team works in the innovation process. Some would argue that lean manufacturing strives for innovation through continuous improvements, but innovation as such can hardly be achieved through a process of refinements, at least not in any material way. The core of innovation has therefore little to do with improvements and everything to do with new ideas. Economic planning as an engineering tool is thus inept and should only be used as a boundary condition that will frame the solution.

The thinking behind continuous improvements can also be found in rapid application development from the late '80s and early '90s. Although most would agree that RAD didn’t work that well, except for a few special cases, it has nevertheless become the stepping stone for many of today’s approaches, notably Agile. To achieve desirable qualities, such as adaptability and feedback, traditional planning and specification activities are ditched in favor of producing tangible output from fast iterations. Another casualty has been how projects use different abstraction levels to reason about ideas and problems without getting lost in a sea of details. Companies, like Twitter, find that most of their ideas are stuck in the backlog, despite Silicon Valley's obsession with execution and continuous delivery. For Facebook, quality has become the number one priority; teams that deliver software with failures have no future at Facebook.

Fast iterations and the delivery of high quality software is a hard equation to solve for many teams. The answer is often to go for the easy idea, the solution that can be implemented in a few weeks and that won't fail when put into service. The result is software with no distinguishable qualities, which is known as commoditization in the world of economics. The truth is that continuous improvements, otherwise known as code refactoring, is necessary in order to address software quality problems, but it won't help teams find or explore innovative ideas in any substantial way.

The concept of waste elimination has also become a central theme in contemporary software engineering. But to know if an activity, or the result it is producing, is waste cannot be asserted when innovating. It's essential for the innovation process to experiment and play with new ideas before they are implemented or pursued in any substantial way. We can therefore not let the concept of waste elimination define our software process.

What would characterize a true Software engineering process?

Before any blueprint can be presented over a true engineering process, we need to examine a few fundamental issues. The first one is to make a distinction between control mechanisms found in classically designed organizations and the drivers that accelerate creativity, innovation and understanding. All projects need control mechanisms to prevent chaos and following disaster, but control mechanisms won't make a product fly, and teams need a process that drives innovation. The second issue is to accept that a product development paradigm cannot be based on ideas that are an intrinsic part of manufacturing. Manufacturing is about standardization, marginal cost, optimization of production flows, and more times than not, a low-skilled work force. Software engineering and innovation can never be about these things, unless organizations are aiming for commoditization of software, which is a really bad idea. A true software engineering process must therefore be cognition based and embrace the idea that a problem cannot be analyzed and fully understood from one perspective only, which is the third issue. To fully understand a multi faceted problem, a process needs to support different perspectives. A business perspective may look at the market, intellectual property rights, regulations, licensing models and other commercial matters. Often these considerations are put into a product management process that is executed separately from the engineering activities, but these considerations nevertheless need to influence how a product is built. Other perspectives, such as architecture or usability, will consider technical matters or everything a user will encounter when working with the software.

A fourth issue says that abstractions are useful when analyzing a problem. Examples of abstractions are the product vision, which outlines main features and why a product should be built, and product requirements that describe the features in more detail. There are many more abstractions, such as architectural models and user scenarios, which a team may find useful when reasoning about their work. Abstractions are efficient, as they usually are easier to work with and understand, but they also have limitations an engineer should be aware of. One of them is that they are seldom updated as soon as coding activities start, another is that they shouldn’t be directly used for implementation activities.

The fifth issue is to use an exact language when describing the process and the results it delivers. A process cannot have fourteen different ways to describe the same activity, theme or artifact. Using a language that is precise will also make the whole organization appear more professional. Precise language matters when a team cares for quality, efficiency and business relations.

The sixth issue is about capabilities. Teams cannot develop a product with tools and technology that aren't as capable as the product they are building. If the gap becomes too large, a product will become a prototype rather than a commercial product intended for a market. A compiler is usually more sophisticated than the programs it compiles and links. The theory behind a compression algorithm is generally more complicated than the images and text it reduces. Capabilities are not only about tools and technology but also about processes and skills.

The last and final issue is that one process doesn't fit all. A specific problem domain or project organization can often use more than one process, but there are processes that are unquestionably inappropriate for a project to use in a specific situation. Generally, the larger a project is the more emphasis there need to be on controlling mechanisms. Processes have historically been mostly about checklists, milestones, reviews, configuration management, planning and testing, but to meet challenges that lie ahead processes need to increasingly take a cognition based approach.

It's not hard to find activities that exemplifies just that. A cognition based process would begin with an elicitation activity, where the team meets the customer or starts information collection from other sources. The research community is particularly good at applying structured methods to find relevant information, but the software community has largely ignored these practices on different grounds. The information that comes from elicitation activities are by most standards raw and unprocessed, and never validated, which means that further analysis needs to be done before the information can be used safely in implementation activities. Another example is modeling activities. A model is nothing but an abstraction of knowledge, who's purpose is to capture key characteristics of a problem or solution. Models can to great advantage be used in elicitation activities to validate processed information or help extract more information, and they can be used as input to specification and testing activities.

A cognition based approach will turn many of today's assumptions in software engineering upside down, but the best part is that it will elevate software engineering, making it innovative and fun again.