Reduce Cognitive Load in Software Engineering through Platform Engineering

Developers and DevOps professionals face an ever-increasing cognitive load, considered one of the most significant challenges in software engineering. With the proliferation of tools, frameworks, and the evolution of DevOps philosophy such as "You build it, you run it," the cognitive load has substantially increased in delivering customer-centric products and services.

In this article, we'll delve into the foundational concepts of cognitive load, exploring where developers and DevOps engineers encounter it, and how platform engineering can mitigate this burden and enhance corresponding workflows.

Understanding Cognitive load

Generally, individuals have a limited capacity to handle complexity at any given time, with the number of ideas in our minds typically ranging from 3 to 7. Unnecessary information or tasks that divert attention also contribute to the complexity we deal with.

Complexity arises from the process of integrating ideas and concepts to aid understanding, constituting the cognitive load we endure while attempting to execute tasks. In psychology, different types of burdens are categorized:

• Internal Cognitive load: Arises from inherent task difficulty.

• External Cognitive load: Arises from handling distracting or unnecessary elements.

• Additional Cognitive load: Arises from building an understanding of the task.

As familiarity with a task increases, internal cognitive load decreases. For instance, driving may initially feel overwhelming for a novice, even under optimal conditions, requiring high concentration. However, as familiarity and driving skills improve, the cognitive load diminishes. External cognitive load, like a sudden phone call while driving, still influences driving skills.

Developers' Cognitive load

Developers often discuss matters related to architecture, abstraction, and implementation details. Architecture represents overarching ideas or concepts, defining how systems come together at a macro level. Abstraction involves generalization, such as reusing code or components within the architecture. Implementation details encompass how abstractions are concretely realized.

This hierarchy allows developers to understand the system they are working on without getting bogged down by various details. While everyone involved in a software development project should be familiar with the overall architecture, it may not be advantageous for developers to delve into every detail. For instance, knowing every detail might not be beneficial when there's a bug in a specific part of the software or, worse, a flaw in the architecture.

Developing modern software is a highly complex, multi-functional, and collaborative process. Specialized developers focusing on building responsive UI for the web may not necessarily know how to configure a K8s cluster to provide services for the application. Ideally, developers should concentrate on their domain, minimizing external cognitive load, while aspects like configuring K8s clusters divert attention. When everyone can focus on their expertise, the team's value is maximized, and external cognitive load is reduced.

Cognitive load in DevOps

Despite DevOps aiming to enhance software development efficiency, DevOps engineers often face numerous challenging tasks, increasing their cognitive workload. Common examples include designing new processes like CI/CD pipelines for new software projects. DevOps engineers may also be responsible for setting up infrastructure for development environments while ensuring compatibility with the production environment. This involves managing Docker files, Helm charts, and Terraform code, which need regular maintenance as the project progresses. CI/CD pipelines need to be constructed, and while engineers can reuse some parts from previous projects, new testing or build requirements for a new project add complexity.

Existing processes must scale up or down to meet current demands. This includes expanding infrastructure to meet new processing or storage requirements and modifying existing workflows designed by a growing team of small-scale engineers.

DevOps engineers also take ownership of various aspects of the software engineering lifecycle. This includes managing third-party tools and products on top of internal code repositories and infrastructure. Other developers require code reviews and meetings to discuss potential solutions requiring specialized DevOps knowledge. Moreover, DevOps engineers must ensure the system logs correctly and can collect and analyze metrics, crucial for ensuring smooth operation and understanding potential areas that need scaling.

All these tasks bring significant cognitive pressure to DevOps practitioners. Each task and process generate additional overhead that DevOps practitioners must handle, often detracting resources from their primary goals of innovation and optimization.

As cognitive load increases, associated complexity and stress rise, leading to fatigue and increased errors. This significantly lowers team productivity, ultimately hindering innovation.

Platform Engineering Effectively Manages Complexity

Platform engineering exists to provide the abstraction needed for multifunctional teams to collaborate on the same software project. The platform offers developers the infrastructure details on which software runs as an implementation detail, while the software running on this infrastructure becomes an implementation detail for the operations team. Platform engineering effectively reduces the cognitive load of daily work, enabling developers to use resources in a self-service manner, eliminating additional cognitive load introduced by processes like ticketing systems.

One core aspect of platform engineering is the effective partitioning of complexity. Each team in an enterprise organization has its functional domain, and members of that team excel in handling complex issues within that domain, while others can safely ignore those areas. Everyone deals with implementation details based on shared abstractions and an understanding of how information comes together.

For example, architecture serves as a shared agreement for both development and operations teams, ensuring the entire system needs to function to satisfy the customer. Both teams use abstractions: containers are units running on various systems, and resources like databases are available to store data as needed. However, the implementation details differ based on responsibilities. For members of the operations team, implementation details include networking, pod policies in the cluster, and managing database instances. Developers focus on solving business problems, with the implementation details being the issues they need to address. Throughout the project, platform engineering acts as a means of communication for everyone to realize their implementation details.

Incorporating Platform Engineering into Development and DevOps

In this section, we'll combine a use case to illustrate how platform engineering can reduce the cognitive load for DevOps and developers and improve workflows.

Imagine an enterprise designing web applications following a similar architectural pattern. There's a database, an API, a web-based frontend, pre-built Docker images, and CI/CD templates. However, all these are done manually. DevOps engineers must create Docker files, implement Terraform scripts, and build Git pipelines for each project. They then collaborate with developers to ensure the environment meets their requirements and add monitoring and alerts to the production infrastructure. These tasks, regardless of complexity, all fall directly on the DevOps team. Consequently, the workload extends, delaying the delivery time for developers eager to advance their projects.

At this point, platform engineers can automate these tasks by building them into an Internal Developer Platform (IDP). For example, developers can request a repository from IDP, which then creates it instead of manually setting up a Git repository. The IDP assigns the correct user groups and automatically integrates the appropriate CI/CD template. The same pattern applies to creating development environments and deploying core infrastructure. The IDP serves as a self-service platform for developers to request services and application configurations, understanding the default built-in security best practices and monitoring. The IDP can also set up projects automatically in project tracking software and document templates.

In essence, platform engineers enhance workflows by building a standardized pattern into a self-service internal development platform. This eliminates the burden of project initialization, allowing teams to immediately start delivering business value without spending the initial weeks addressing setup issues. Additionally, as developers become more self-sufficient, platform engineers can focus on addressing larger architectural challenges and provide feedback to the IDP. This way, they can improve existing services and strengthen future systems.

In conclusion, the incorporation of platform engineering into the development and DevOps processes serves as a powerful strategy to alleviate cognitive load. By automating and abstracting common tasks and providing self-service capabilities, platform engineering enhances efficiency and collaboration within teams. This approach not only reduces the cognitive load on DevOps practitioners and developers but also fosters a more streamlined and innovative workflow, ultimately leading to improved software delivery and system reliability.

If you are interested in platform engineering, Welcome to our community:

• Discord: https://discord.gg/fXZUKK2baF

• Twitter/X: https://twitter.com/Seal_io

• LinkedIn: https://www.linkedin.com/company/seal-io

• Youtube: https://www.youtube.com/@Seal-io