BoomTown, www.boomtownroi.com, is looking for a VP of Engineering to join their talented team in Charleston, S.C. We have had the pleasure of working with BoomTown on several occasions and think this is a great opportunity. Please email Mike Paylor, email@example.com if interested.
The Vice President of Engineering will join the BoomTown team and report to Founder & CEO, Grier Allen. S/he will lead the engineering and technology practice companywide and will have all duties and authorities commensurate with the position.
BoomTown’s current engineering team is a group of roughly 30 engineers is operating without a centralized engineering leader. There is also the opportunity to lead a smaller team of designers and UI/UX personnel depending on the leader’s experience/interests.
BoomTown has spent the last five years building and solidifying its position as an innovator in real estate technology. At the same time, the company has established itself as a trusted partner to real estate professionals in a time when disruption of the real estate business model is a consistent theme. With the massive shift to online home search, brokers and agents must find new ways to effectively engage those consumers, establish longterm relationships, and reestablish their value, which is under continuous attack.
As the real estate market has rebounded, brokers and agents are scrambling to invest in technology that will diminish the threat of disruption. BoomTown is in a unique position to capture the mindshare of both agents and brokers at both the SMB and Enterprise/Franchise level. BoomTown believes that the ability to truly impact agent sales productivity is by connecting the right agent with the right consumer, at the right time, and with the right information is the key to why agents and brokers will adopt the next generation of real estate technology.
Central to BoomTown’s strategy is the focus on predictive datadriven intelligence. By combining consumer behavior with realtime listing activity and agent performance, BoomTown can focus agents on the most productive activity at any given moment. This enables agents to rapidly and effectively engage consumers throughout the lifecycle. BoomTown’s strategy — to marry predictive analytics with a mobilecentric experience — is key to becoming the dominant technology in the future of real estate.
Just over a month ago, the WebscaleSQL collaboration project was launched. This project aims to create a community-developed branch of the popular MySQL DBMS that incorporates features to make large-scale deployments easier. As many of our clients run large clusters of MySQL on commodity hardware (as a means to reduce costs and improve scalability) the WebScaleSQL project naturally drew our attention.
The project is currently developed by a small collaboration of engineers from Facebook, Google, Twitter, and LinkedIn. Certainly no strangers to scalability challenges, the developers from these web giants all face similar challenges and must work continuously to improve performance and scalability of their MySQL deployments to remain competitive. Aimed at minimizing the duplication of effort across these engineering teams, WebScaleSQL’s development is run as an open collaboration between these major contributors. Contributions aren’t limited to major companies, however, and participation from outside engineers is encouraged.
Despite having only been around a few weeks, the WebScaleSQL project boasts some significant improvements overs its upstream parent (Oracle’s MySQL ver 5.6). These advances include an automated testing framework, a stress testing suite, query optimizations and host of other changes that promise to improve the performance, testing, and deployment of large-scale DBs.
As WebScaleSQL matures, we’ll continue to track its development and report on our clients’ experiences (both good and bad) working with this new “scale friendly” branch of MySQL.
WebScaleSQL source is currently hosted on GitHub (https://github.com/webscalesql/webscalesql-5.6) and released under version 2 of the GNU public license.
In his book, The Lean Start-Up, Eric Ries defines a start-up as a “human institution designed to deliver a new product or service under conditions of extreme uncertainty.” Operating in uncertainty is inherently risky. What makes many start-ups successful, among other things, is their ability to mitigate risk. One way a company can remain adaptive and responsive during uncertainty, and therefore reduce risk, is employing an agile development model. A good example of such an organization is Spotify, which continues to embrace, practice, and improve on agile fundamentals even after growing to over 30 development teams in three different locations.
There are striking similarities between the way Spotify organizes their teams and the way U.S. Army Special Forces organizes their Operational Detachments-Alpha, aka “A-Teams.” It makes sense that the two would be similar because the A-Team was in fact an innovation designed to deliver a “service” under conditions of extreme uncertainty. In this case, the “service” was military assistance to the struggling Republic of Vietnam, and later to guerrilla forces and resistance movements waging unconventional warfare against the North Vietnamese. The U.S. Army had never undertaken a mission similar in objective and scope. They did not know the conditions, the support, or the communication capabilities that their soldiers would find. The U.S. Army needed a self-sustaining team with all the skills necessary to complete a mission with a broad and evolving scope. Moreover, the team would have to operate autonomously and would have to be trusted to make decisions normally left to upper echelons.
The U.S. Army assessed and selected twelve of their most seasoned veterans, and then trained them each in one of six specialties: medicine, engineering, communications, weapons and tactics, or intelligence. Having redundancy in each position was desired in case one soldier got injured, and it also gave the team the ability to pair a new member with a more senior member of the same specialty. Redundancy also facilitated the ascension of the team’s most senior soldier to the role of Operations Sergeant. The A-Team Operations Sergeant is comparable to the role of Spotify’s Agile Coach, or sometimes called a “Scrum Master” in other organizations. He no longer fills a specialty function on the A-team. He is the team’s mentor and draws on his experience to coach the team. He wants his team to be efficient and safe, which equates to a quality product in their realm. It is the also the Operations Sergeant’s job to advise and informally train the A-team’s commander on the team’s competences and potential with respect to their assigned mission. He ensures that the commander does not over commit the A-team to the stakeholders.
The A-Team was designed to operate deep behind enemy lines. They would not maintain frequent contact to their command. Because the complexity of the mission at hand, the A-Team worked not only for their command, but on the on behalf of many other stakeholders, such as the U.S. State Department, CIA, and other government agencies. In the absence of guidance from their command, and other key stakeholders, it is crucial that the commander, with input from the team, manage the team’s priorities with respect to the mission’s goals. Just like Spotify’s Product Owner, it is the A-team’s commander who ensures the team is on track to accomplish the stakeholders’ desired end state by prioritizing projects and associated tasks.
The way that Spotify tweaked the idea of a Matrix Org to facilitate product-centered teams while managing professional development and adherence to standards across functional areas resembles the method used by A-teams. Spotify organizes their development teams or “Squads,” into “Tribes” of 100 contributors of varying disciplines such as software engineering, hardware engineering, DBAs, etc. Each Tribe has a “Chapter” in which contributors with the same competencies are grouped. Chapters are led by “Chapter Leads” who are responsible for the quality of work produced by his or her chapter. Special Forces A-Teams are organized into “Companies” of six multi-disciplinary teams (around 70 people). A-Team “Companies” have a headquarters unit that is responsible for the professional development and standards of each specialty on the team. As an example, the medics on each team practice medicine under the license of a medical doctor assigned to the headquarters unit. The doctor’s function is similar to that of a Chapter Lead because he is responsible for each medic’s continued professional growth, refresher training, and the overall standard of care that is provided by his chapter.
The A-Team model’s structure has not significantly changed since their inception. During the Vietnam War, and most recently in Iraq and Afghanistan, Special Forces A-Teams have produced tangible results where larger, conventional military units have not. Their self-sufficient and decentralized approach allows the team to focus on mission accomplishment. The conventional Army’s centralized and hierarchical structure doesn’t allow for the flexibility needed to mitigate risk in uncertain environments. Results often become secondary to strict adherence to protocol for protocol’s sake, and the approval of long reporting chains. Similarly, in the technology industry, companies like Spotify who adhere to agile fundamentals are able to scale and grow successfully amid the uncertainty, while larger, less “agile” organizations become bogged down in bureaucracy, and can not navigate the evolving and uncertain market.
- Why Caching is Key to Optimizing Software Performance
- Why Software Companies Need to Adopt a Services Mindset
- One Big Warning Sign Your Organization Isn’t Structured to Scale
- Build or Buy? How to Decide if You Should Outsource Development
- Own the Goal! Tips for Empowering Teams to Innovate
- Companies should encourage customers to misbehave with their products and services
- How We’d Fix HealthCare.gov: Scalability Experts Speak
Make sure you follow Marty and Mike and AKF’s Facebook Pages to keep up to date with the latest.
As a young Army Lieutenant leading a Platoon in Iraq in 2006, I was faced with a crisis. Our platoon was continually getting ambushed during our patrols in sector. The uncertainty of waiting to be attacked, the physical, mental and emotional strain on the men, not to mention the damage to our vehicles and equipment, started to take its toll. The ensuing battles would generally last for hours. We would stay out in sector for upwards of 16 hours. This would only leave 8 hours to come back to the forward operating base, refit and conduct maintenance on our vehicles. This type of operation couldn’t be sustained. Our morale was starting to fall, tensions were starting to rise between my platoon and the maintenance crews servicing our vehicles. We’d bring in leaking, limping, bullet ridden trucks and expect to jump back in them and roll out in a matter of hours. In military terms, we had “lost the initiative.” That is to say, the enemy was dictating where and when and how the battle would go. We were only reacting. We were compelled to react to the enemy’s strengths and capabilities. We were treading water trying to stay afloat. In this crisis, my men depended on me to come up with a plan to take back control, or “seize the initiative.” The importance of having a plan is emphasized in Sun Tzu’s famous quote, “ Victorious warriors win first and then go to war, while defeated warriors go to war first and then seek to win.” My platoon and I were at war, and trying to win.
The crisis I experienced in Iraq is similar to the challenges many executives face with regard to scaling. Granted, no one is shooting at them, but the enemy can represent any or all of the variables and uncertainty in the industry; the customer, the competition, the software, the hardware, the market, etc. Some executives find they are compelled to react to daily incidents, i.e. “lets just make it another 24 hrs without an outage,” and it’s a struggle to maintain, let alone scale effectively. Just like with my platoon, this type of operation takes a toll on an organization. Aside from being bad for business, it sows the seeds of despair, brings down morale, and possibly creates tension between the engineers and the business folks similar to that between my troops and our maintenance crews. The leader needs to take a step back, understand what is going on and figure out a plan to get his or her team out of it.
For me, the solution was counter-intuitive at first. I knew we needed to take time to regroup, but I hesitated to not keep pressing the enemy. I feared that we’d lose what little ground we had. Then, I remembered something my instructor told me while learning to navigate in the woods with a map and a compass. He warned us, “If you don’t know where you are going, don’t walk as fast. You’ll only get lost in a hurry.” In our eagerness to get out into sector and fight the enemy we were getting lost in a hurry. We had gone to war, and were now trying to win. It is a difficult decision to halt operations, or take a product offline. Aggressive leaders are often very successful because they persevere in the face of adversity. However, there is a fine line between perseverance and stubbornness. I made the call to postpone operations. I needed time to develop an effective sustainable strategy.
“Seizing the initiative” requires leaders to make sound decisions under uncertain conditions and to anticipate events so their organization can recognize and leverage opportunities. The key to anticipating events is to understand the problem. In my case, we really did not understand the enemy. From an institutional standpoint, we had been training to fight the soviets for the last 40 years, and here we were faced against an adaptive and evolving enemy. We scaled our operations back at first, focused on reconnaissance and understanding the enemy. In doing so we were better able to anticipate events, and began setting the conditions of the fight. We were engaging the enemy when and where we wanted. We had seized the initiative. Regardless of your type of organization, if you are struggling just to maintain, if you have lost the initiative, the best solution is to take the necessary pause in operations, understand the problem, and develop a strategy for a sustainable effective approach.
After several friends recommended that I read the book The Phoenix Project, I at last downloaded it to my kindle and read it on one of my transcontinental flights. I really enjoyed it and thought it was both entertaining and enlightening which is difficult to accomplish. However, I don’t believe it goes far enough in describing how the most modern and efficient technology shops run. More about that later, first a recap.
The story line is that the fictional company, Parts Unlimited, is failing to keep up with competition and at risk of being split up and sold off. The root cause of the company’s issues are IT. We join the story after two IT executives are fired and the hero of our story, Bill, is asked to take over as the VP of IT Operations. He is introduced to his sensei, Erik, a tech / business / process genius, who leads Bill through the path to enlightenment by comparing IT Operations to a manufacturing plant. Through this journey the reader is introduced to concepts such as kanban, work-in-process (WIP), work centers, Theory of Constraints, Lean Startup, and the overarching concept of DevOps. In fact the book ends with most of the main characters at a party where Bill reflects “Maybe everyone attending this party is a form of DevOps, but I suspect it’s something much more than that. It’s Product Management, Development, IT Operations, and even Information Security all working together and supporting one another.”
I’m a fan of DevOps but to me the integration of tech ops and development is one small step for technology driven companies. In order to truly be nimble and effective, teams need to incorporate development, tech ops, product management, quality assurance, and any other function such as marketing or analytics that are needed to make the team autonomous. We’ve written about this concept several times The Agile Org, The Agile Org Solution, and The Agile Exec. We call these teams PODs and most importantly they need to be aligned to swim lanes in the architecture. By doing this, you essentially create mini-startups. These teams can make decisions themselves, deploy independently, and are responsible for their service from idea to production support.
I really enjoyed the book The Phoenix Project and I too highly recommend that you read it, especially if you are not familiar with the concepts mentioned above. As the authors are obviously talented writers and technologist, I would love to see them write another book that takes Parts Unlimited to the next level by creating autonomous teams. The authors hint at this concept when they state
“You’ve helped me see that IT is not merely a department. Instead, it’s pervasive, like electricity. It’s a skill, like being able to read or do math. Here at Parts Unlimited, we don’t have a centralized reading or math department—we expect everyone we hire to have some mastery of it. Understanding what technology can and can’t do has become a core competency that every part of this business must have. If any of my business managers are leading a team or a project without that skill, they will fail.”
Hopefully this is foreshadowing of another book to come.
The Power of Customer Misbehavior is now available for the Kindle. Here is a short video about the book.
Its Monday morning and past Saturday evening issues in one of your datacenters triggered a failover to your second data center for service restoration. In other words, all customer traffic has been routed to a single datacenter. The failover was executed flawlessly and the team went back to bed waiting for Monday morning to permanently fix the issue so traffic could once again run out of both datacenters. On Monday morning, we are expecting a flash sale and will make close to $8000 a minute at peak. All is well and there is nothing to worry about. Right?
Hopefully you cringed at the above scenario. What if the data center you are running out of suffers from a failure? Or what if the only data center and its components that is now live for all of your traffic simply wasn’t sized correctly for acceptable performance during a traffic spike?
If it hasn’t happened yet, it will. If that were the case, your business would stand to lose significant revenue. We see it over and over again with many clients and have also experienced it in practice. Multiple datacenters can serve as a false sense of security and teams can become complacent. Remember, assume everything will fail as a monolith. If you are only running out of a single data center and the other is unable to take traffic, you now have a SPOF and as a whole the DC is a monolith. As a tech ops leader you have to drive the right sense of urgency and lead your team to have the right mindset. Restoring service with a failover is perfectly acceptable. However, the team cannot stop there. They must quickly diagnose the problem and return the site to normal service, which means you are once again running out of two datacenters. Don’t let the false sense of security slip into your ops teams. If you spot it, call it out and explain why.
To help combat complacency from setting in, we recommend considering the following:
- Run a Morning Ops meeting with your business and review issues from the past 24 hours. Determine which issues need to undergo a postmortem. See one of our earlier blogs for more information: http://akfpartners.com/techblog/2010/08/29/morning-operations-meeting/
- Communicate to your team and your business on the failure and what is being done about it.
- Run a postmortem determine multiple causes and actions and owners to address the causes: http://akfpartners.com/techblog/2009/09/03/a-lightweight-post-mortem-process/
- Always restore your systems to normal service as quickly as possible. If you have split your architecture along the Y or Z axis and one of the swim lanes fails or an entire datacenter fails, you need to bring it back up as quickly as possible. See one of our past blogs for more details on splitting your architecture: http://akfpartners.com/techblog/2008/05/30/fault-isolative-architectures-or-“swimlaning”/
“There is a land called Crete in the midst of the wine-dark sea, a fair land and a rich, begirt with water, and therein are many men innumerable, and ninety cities.” - Homer (fl. 850 B.C.), Odyssey, Book XIX
“And some one shall some day say even of men that are yet to be, as he saileth in his many-benched ship over the wine-dark sea…” – Homer (fl. 850 B.C.), Iliad, Book VII
The phrase “wine-dark sea” appears dozens of times in the Iliad and the Odyssey resulting in much debate at what Homer actually meant by the phrase. One theory is that he was describing an outbreak of red-colored marine algae. Another theory put forward by researchers Wrignt and Cattley, was that the Greeks mixed highly alkaline water with their wine, resulting in blue-wine.
The explanation that sounds most plausible to me is that ancient Greeks did not have a word for “blue”. At the time of Homer’s writing there were only five colors (metallics, black, white, yellow-green, and red). Lacking the appropriate term to describe the world, they used what they knew.
Many of us are not unlike the ancient Greeks. We can’t describe a new architecture or we can’t imagine our business differently because we don’t have words for it. Whether you are just out of school and don’t have a ton of experience or your more senior but haven’t seen highly scalable system architectures, both leave you blind to “seeing” a different architecture or business model. Another way we’re blinded is just by being at a business for a number of years. Companies and institutions have collective beliefs, cultures, and even memories that become our own. This is completely normal. If you don’t adapt to the standards and norms of the company you’re going to have a rough go of it. Just like the body tries to reject transplanted organs because the body doesn’t recognize the foreign object, companies reject employees and even leaders who don’t fit or adapt to fit.
So, what is one to do if you know you suffer from a blind spot? The solution to this is to bring in new people or, occasionally, consultants who have seen this before and can help you learn to describe the new world. As made famous by many twelve-step programs, the first step is to “admit there is a problem”. If you can’t admit that you might not see or be able to accurately describe everything, you’ll eventually get blindsided by either the inability to scale or, even worse, a competitor able to see things differently.