Executives seeking digital transformations of their organizations require a more agile approach to architecture than traditional EA has been able to offer. Instead, architects must treat the enterprise itself as a complex system, driving business agility as an emergent property of the organization as a whole.
Perhaps the best example of a company that has succeeded with this agile approach to architecture is Netflix. Netflix NFLX -0.87% is well-known in technology circles for its industry-leading use of Public Cloud, but how they build software and in particular, how they drive their overall architecture are perhaps the most paradigm-shifting aspects of their accomplishments.
Executives at traditional enterprises may look at the Netflix case study and chalk it up to the success of a Web Scale company that would never apply to a bank, a government agency, a manufacturer, or any other traditional enterprise. To help dispel this myth that what worked for Netflix won’t work for you, I interviewed Adrian Cockcroft.
Today Adrian Cockcroft serves as Technology Fellow at Battery Ventures, but until recently, he was Cloud Architect at Netflix, well known as a thought leader for the Netflix way of doing things. Our conversation wasn’t simply an opportunity to reminisce about the Netflix days, though. At Battery, he’s seeking and supporting technology companies “who are enabling the transformation of enterprise IT” – in essence, who are able to take a page from the Netflix playbook and apply it to the enterprise. The secret to Netflix’s success? A transformative set of practices I call Agile Architecture.
The Current Status of Enterprise Transformation
As a public figure in the Cloud and DevOps worlds, Cockcroft is a lightning rod, both for professionals with problems in this space as well as people with success stories to tell. As a result, he has a unique perspective on the leading edge of progress in enterprises around the world. What are they telling him? “Enterprises have hit a turning point,” according to Cockcroft. “A year ago they still weren’t convinced about Cloud. Now they’re comfortable with Cloud and Software-as-a-Service (SaaS), and they’re trying to get DevOps to work.”
As a natural skeptic, I called him out on how enterprises are approaching DevOps: DevOps-as-buzzword, or as a true cultural shift? “People who talk about caricatures of DevOps won’t be successful,” Cockcroft explained. “DevOps is a reorganization and a cultural transformation. Many enterprises are in the middle of this transformation, and struggling.” In fact, DevOps doesn’t stand alone as a transformative force within such enterprises. “Cloud, continuous delivery, DevOps, and Agile software development are all tangled together and related,” according to Cockcroft.
In my experience, this drive toward transformation is a relatively new trend. As recently as 2013, most large enterprises were discussing digital transformation and taking early, toe-in-the-water steps, but had yet to jump in with both feet. “It does look like we’re crossing the chasm,” Cockcroft agreed, referring to Geoffrey Moore’s theory about the technology adoption lifecycle. “I’m seeing a few ahead, a lot in the middle, and a few laggards.” Perhaps the central characteristic of this chasm-crossing transition is the shift from early adopters as risk takers to the early majority who doesn’t want to be left behind. Cockcroft continued, “Enterprises face the threat of not doing such transformation – of their competitors running away from them.” As a result, “Enterprise IT has got to catch up quickly.”
In fact, this need to accelerate the business, and in particular, the development and rollout of new products and services, is a common driver across many organizations. “Enterprises want to make their OODA (observe, orient, decide, and act) loops for product development much faster,” Cockcroft said. “They want continuous learning via feedback loops and hypothesis testing for both SaaS and mobile apps.” As a result, the strategic business need for innovation at speed is percolating throughout the entire organization, and the IT shop is feeling the brunt. “Software development is the slowest thing they’re doing,” according to Cockcroft. “It has become the bottleneck.”
The Cost Optimization Paradox
On the one hand, accelerating the product development lifecycle helps the top line through improved time to market and increased market penetration, but it also reduces costs, because fewer person-hours are required between initial conception and launch. Ironically, to achieve such cost savings, organizations must bypass various policies and procedures that were intended to control costs. “Months of approvals cost more than the machine you asked for,” an example Cockcroft mentions as one reason why Cloud computing can save money. “When you optimize on cost, you slow things down, which increases costs elsewhere,” Cockcroft explains. “Externalities often aren’t taken into account.”
It is precisely those externalities that enterprises must focus on in order to achieve their digital transformation goals. Cockcroft likens such externalities to scar tissue. “Most companies are built out of scar tissue,” he posits. “Scar tissue includes processes, rules, the HR manual, the whole waterfall idea. Scar tissue prevents you from getting anything done. You need to clean stuff out, automate, streamline with fewer handoffs.”
This streamlining was one of the most important lessons Cockcroft learned at Netflix. “When you believe your process is good enough that nothing will go wrong, that’s when everything goes wrong. The process prevents the right product.” And DevOps was the key to solving this paradox. “DevOps combines externalities with core processes,” he explained. “The entire team moves at the speed of development, because they’re now one organization.”
Reinventing Architecture
Clearly, if traditional approaches to Enterprise Architecture rarely work in typical enterprises, then they certainly wouldn’t be effective at Netflix. Cockcroft agreed. “Monolithic apps that need rigid architecture with rigid architecture review boards that maintain central control” characterize the traditional enterprise approach to architecture, but Netflix was having none of it. Instead, “At Netflix, we had no central control” of the architecture, according to Cockcroft. “If we did we’d always be behind because we’d stop innovating.”
Instead, “The goal of architecture was to create the right emergent behaviors,” Cockcroft explained. Emergent behaviors are the result of the self-organization that complex systems exhibit. To achieve the desired product innovation at Netflix, Cockcroft’s team would “relax control and let it be ‘chaotic,’ but in reality it would be ‘emergent.’ Set up feedback loops and change the management style.” In fact, the difference between chaotic and emergent behavior is central to the theory of complex systems – and as a result, the central challenge of Agile Architecture is taking steps to ensure that desirable emergent behaviors result instead of the normal chaos that large organizations are typically susceptible to. “Architecture meant making people do the right things without constraining what they built,” Cockcroft explained. “At Netflix, my role as architect was to document, encourage, and evangelize architecture. For example, how we handled availability, disaster recovery, and backups.”
As a result, many architects might not recognize Cockcroft’s style of architecture. “Our architecture was changing faster than you can draw it,” he pointed out. “As a result, it wasn’t useful to try to draw it.” Furthermore, Netflix never considered their architecture to be a prerequisite to building software. Cockcroft explained that “the architecture we created was an idealized version that we put out there to get feedback. The faster we learned, the better able we were to support efforts to transform the company.” Such feedback loops are the key to driving the business agility at the heart of Netflix’s product development strategy. “The agile approach means continuous course corrections,” he explained.
Lessons for Enterprises
The most important takeaway from both Adrian Cockcroft’s experience at Netflix as well as his work in the enterprise software space is that successful digital transformation requires a solidly agile approach to architecture. Organizations must pull together Cloud, mobile, Agile development, and DevOps to drive rapid product development across multiple teams, each of which is responsible for its own continuous delivery. Trying to control this apparently chaotic environment with traditional architectural artifacts, processes, and procedures will only slow the organization down, thus hampering innovation.
And yet, organizations still require a high level of overall design and coordination that supports the agility goals of the business without hindering them. Such architecture is inherently agile in approach. “The architectural patterns at Netflix were resilient and functional,” Cockcroft explained. “Optimized but not centrally managed.” Not traditional architecture to be sure, and perhaps not even recognizable as architecture by long-term practitioners of the craft. And yet, this Agile Architecture approach is absolutely essential to the success of any digital transformation initiative.
via Agile Enterprise Architecture Finally Crosses the Chasm.
Agile Enterprise Architecture Finally Crosses the Chasm
