AKF recently engaged with a large, very successful on-premises solution space to help define and accelerate their public cloud product offering and move to an "as a service" model. As with many companies forced to play “Zone Defense” (Geoffrey Moore, Zone to Win), when their customer base and technical base is attacked, the company forklifted its existing product into the public cloud.
It’s an appropriate spur of the moment move – one we commonly endorse to provide runway for our clients to create a more cloud-native solution. AKF likes to call this a "bridge strategy" - necessary to get where you're going but not a destination in and of itself. But, as is almost always the case, multiple challenges ensued including lower than desirable availability and higher than necessary costs of goods sold; again - a fork-lift cloud move is a bridge, not a destination. The company was underinvesting in its public cloud offerings based on the demands of the large base of Fortune 300 on-premises deployed customers from which it earned the majority of its revenue.
The company needed to shift the organization to a focused Zone Defense Transformation level of investment, creating a cloud-native core product strategy in order to meet the growing needs of the market towards public cloud capabilities. Although most of the existing clients still had large on-premises platform footprints, competitive cloud offerings were winning on some smaller workloads and gradually moving up in scale capabilities to potentially impact their core revenue base.
Organizationally the engineering team was large (>1,000 people) with teams located in the United States and abroad. Most of the executive team was new in the previous 12-18 months with key searches still underway. Given the changes still in flight at the senior management level, multiple executives sponsored a search for an interim CTO to help drive the engineering organization while other senior executive searches were being conducted.
Over the last 12 years, AKF Partners has been assisting in developing a holistic cloud acceleration strategy from Business, Product and Technology perspectives. We had identified key short, medium and long term actions to focus on in order to execute a product and organizational transformation including establishing a public cloud dedicated team to ensure adequate focus and autonomy – and to optimize speed above all else towards the public cloud market objectives. An expert was hired to lead the public cloud dedicated team, while I was placed in the Interim CTO role to drive and transform the core engineering organization to the needs of the future evolving in-market platform.
Following is a summary of key challenges the organization faced.
Product and Business:
- Suboptimal overall performance benchmarked to competitive products for the core platform.
- Slow, highly manual – and error prone – process to provision customers initially on public cloud platforms.
- Margin erosion as the product was not architected to take full advantage of inherent elasticity of public cloud
- Subpar availability and resilience in public cloud based on on-premises architectural limitations built into product after decades of evolution on tightly controlled, integrated and tuned hardware (as is the case with most on-premises providers)
- Customers frustrated by release quality issues brought on by process changes meant to speed up delivery – but without the necessary agile, automation and architecture foundations in place yet
The obstacles to success within the engineering organization followed directly from the broader product and business challenges.
People & Organization:
- Lack of accountability and ownership of outcomes at all levels – affective conflict regarding who got to make what decisions.
- Siloed teams with a mixed bag of platform responsibility areas – resulting in finger pointing and a culture of ‘it’s not my job/not my fault’
- Shoot the messenger culture resulting in lack of accurate information regarding issues with quality and delivery (and therefore regular ‘bad surprises’) - and also contributing to a culture of fear
- Highly toxic culture as regularly self-described by employees.
- Lack of adequate skills in relation to public cloud offerings
- Isolated public cloud skills concentrated in one cloud team
Process:
- Lack of focus on the most important initiatives and peanut buttering of resources to priorities
- Lack of clarity and transparency regarding what Engineering was working on. Resource allocation analysis required heavy lifting that was immediately out of date
- Inconsistent quality of epics and user stories
- Deployment to cloud near the end of the release cycle rather than cloud first, resulting in slower time to market for public cloud as issues with deploying and running in cloud were found at the very end of each (generally large) release
- Heavily waterfall release process overall with long QA cycle times in the final phase of release
Architecture:
- Antiquated cloud last approach to development as one separate team was charged with optimizing for deployment to public cloud as opposed to building natively to capitalize on cloud capabilities
- No broader cross-engineering plan to move to a cloud native architecture
- Highly manual and fragmented development pipeline
- As with many on-premises deployments, the underlying infrastructure was optimized to run on-premises on tightly controlled and tuned specialty hardware and required significant foundational changes to run well on commodity public cloud platform(s)
The Fix
I was brought in as the interim EVP of Engineering to accelerate the team towards the public cloud success along all necessary dimensions including organization/culture, process and technology architecture. Requests to fill interim CTO roles are a common ask from our clients based on our teams’ in the trenches experience.
The need for an Interim CTO can arise for a variety of reasons and given this, an important first step is alignment on outcomes for the interim assignment. Therefore we started my assignment by working with the CEO and executive team to agree on priorities based on this top level product cloud acceleration, as well as current challenges identified in the AKF Assessment recently delivered to the Executive Team and Board of Directors. Key actions taken along these dimensions are summarized below.
... put simply a fork-lift cloud move is a bridge, not a destination ...
Organization/Culture:
- Reorganized full engineering team within 6 weeks to directly align to product outcomes and therefore be a foundation to drive clear ownership and accountability with engineering leadership
- Exited leaders who were ineffective in their role in the first 30 days
- Emphasized open communication and set up ongoing channels for dialogue with myself, management and the entire organization
- Conducted weekly skip level meetings in order to ensure ongoing input broadly and deeply and help counteract the previous perception of closed and ivory tower management
- Held regular cadence of engineering all-hands meetings, emphasizing open dialogue to re-establish trust and share progress towards a short list of key objectives
- Defined modern cloud skills needed going forward and updated recruiting approach accordingly for ALL roles going forward
- Developed plan to move some engineers and hire cloud skills into all product engineering teams rather than being concentrated in one separate cloud specialist team
- Worked with the new leader to properly skill the team working on modernizing the core of the platform while the broader engineering team continued to deliver on near/medium term product priorities for both on-premises and public cloud customers (with the existing in-market products
- Established stronger open and transparent dialogues between engineering, product and customer facing teams and made this visible to the broader organization as part of larger strategy and information sharing meetings and forums
Process:
- Established the importance of simple time tracking to key initiatives and elevated transparency of reporting what engineering was working on so everyone understood how the information was used to drive investment and product prioritization decisions
- Reviewed data quality for time tracking, epics/stories and prioritization in order to improve quality and completeness for ongoing use as part of the prioritization and development lifecycle.
- Established a rhythm of Operating reviews to manage engineering quality and delivery including:
- Joint Customer quality reviews with Product and Client facing teams
- Regular execution focused reviews on key priorities
- Regular operational reviews to review quality and craftsmanship and to drive continuous improvement across engineering
- Established basic guidance and standards for use of tools to set a baseline of quality and consistency on the foundations of agile execution
- Used out-of-the-box Jira reports during progress reviews wherever possible to eliminate duplicate/busy work on powerpoint and report manipulation
- Created top level engineering scorecard to be shared with executive team on a weekly basis which covered indicator metrics for: customer quality, agile execution and key engineering improvement initiatives (e.g. indicators of code quality, security etc.)
Technology Architecture:
- Moved team from central cloud ‘lift and shift after the fact approach’ to cloud first at the product component level by reviewing cloud feature roadmap and determining which features should be built into the components themselves and developing a plan to shift this work based on skilling up and adjusting teams as needed to distribute this work
- Established plan to automate build/deploy pipeline to all clouds such that public cloud deployments happen throughout development rather than at the end of releases. Emphasized the need for teams to work autonomously, including owning their own development pipeline (based as appropriate on common tools and enabling frameworks)
- Instilled a sense of urgency with focus on delivery and specific engineering improvement initiatives: CI/CD, moving to cloud first, ongoing deployments to ALL target environments
The Results….
Over a 6-month period we made significant progress in accelerating the entire organization to focus on its most important business imperative to build a competitive public cloud offering. By quickly organizing the team to focus on product outcomes, we set the foundation for faster progress, a more accountable engineering attitude and improved morale. In parallel we educated the organization on what cloud first meant and paved a path towards this evolution both organizationally and architecturally.
Morale also began to improve based on modeling open, honest and transparent dialogue across the organization. Perhaps most importantly we began to improve the perspectives on the engineering teams’ capabilities and potential based on building strong partnerships across engineering, product and customer facing teams and extending the push towards engineering transparency to all parts of the organization. In addition, we ensured the team continued to make urgent progress while the search for the ongoing CTO was executed as we created a strong foundation for the new leader. Once the new leader was onboarded we executed a seamless transition with proactive communication to the engineering team and key stakeholders. AKF has performed dozens of interim executive positions from CEO of multi-billion dollar companies, to VP of Engineering at startups.
We've helped companies across the spectrum from rapidly scaling under high demand to fighting off competitors in long tenured and established markets. Need some help at an executive level? Give us a call.