AKF Partners

Abbott, Keeven & Fisher PartnersPartners In Hyper Growth

Category » Interviews

Outbrain’s CTO on Scalability

I had the opportunity to speak with Ori Lahav, Outbrain’s co-founder and CTO, about his experience scaling Outbrain to handle 1.5 billion page views in just three years.  Outbrain is a content recommendation service that provides for blogs and articles what Netflix does for videos or Amazon does for products.

Ori is oversees the R&D center for the company located in the heart of Israel’s technology center. Prior to founding Outbrain, Ori led the R&D groups at Shopping.com (acquired by eBay) in search and classification. Before joining the internet revolution Ori led the video streaming Server Group at Vsoft.

Below are my notes from the interview.  Here is the link to the audio version but the quality is fairly poor.

What is your background and how did you come to start Outbrain?
Outbrain was founded by Yaron Galai and myself (Navy friendship).  Before that I spent 3.5 years at shopping.com, which was acquired by eBay, leading software groups.  I liked the challenges of a start-up and joining forces with Yaron was a great opportunity.

How has Outbrain grown over the past couple of years?
We did some false starts with several directions but when we started with recommendations on content sites it started catching fire.  We started with self serve blogs, then professional blogs, small publishers, national publishers, and now we are international.  In 3 years we grew from 0 to over 1.5B page views, 3 production servers to over 150, 0 system administrators to 4, and from 3 developers to 15 today.

How has your architecture changed over this period?
Surprisingly… not much, it was first 2 application servers and a MySQL database.  Then we started adding more application servers and replicating the MySQL.  As the product grew, we added more and more components including memcached, Solr, TokyoTyrant, and Cassandra.  We recently added layer that warms the caches and is being notified by ActiveMQ.

On the backend, we have machines that are fetching content from the sites we are on.  We gather article text and images and save them to MogileFS where we have indexers that index them in Solr.
We started investing in reporting infrastructure and started with MySql but with the amount of data we have we shortly understood that Hadoop is the way to go – we now have a cluster of more than 10 nodes in our Hadoop cluster.

What is Outbrains view of scalability?
First – scalability is culture – if you think big – you will be big.  The regular rules of scalability apply here too:

  • No singles
  • Scale out instead of up
  • Replicate data to ease the load
  • Shard data to scale
  • Utilize commodity hardware

Specifically for Outbrain I would add:

  • open-source is crucial for scalability
  • be cost conscious
  • build many simple/small environments and not a single big one

How have you managed data centers to be so cost effective?
We simplify it!  We create many small datacenters and not a big one.  We have no upfront costs of network gear (stacking) and real-estate (Cages).  We use open-source for our network (load balancer and firewalls) and infrastructure (OS).

In order to ease the step function of cost, grow as you go. Our data center provider, Atlantic Metro, has helped with metered power instead of paying by the circuit.  This way we’re motivated to power down servers not needed.

What has been the most difficult scalability challenge?
Scaling the team…we are more of a patrol boat DNA not an aircraft Carrier DNA.  Technology challenges are fun – never too hard but the team and process are much more challenging.

We have also started using the Continuous Deployment process and this has been a great help. By empowering the team members to act and change the system as they need – you can grow to be an Aircraft carrier and still keep the maneuverability of a patrol boat.

What do you think of the NoSQL movement?
We are big fans.  We use Hadoop, Hive, Cassandra, TokyoTyrant, MySql, and others.  This helps us maintain our low serving cost and attract the type of talent we need on the team.
Outbrain is proud to be 100% open-source.  We use open-source from the office telephony system to all platforms and network infrastructure.

And… we are hiring top technology talent in Israel, so contact Outrbain if you are interested.

1 comment

Puzzle Questions in Interviews

Jeff Atwood of Coding Horror posted his thoughts on using puzzle questions in interviews. He is not a fan of the puzzle question and prefers to “have the candidate give a 10 minute watercooler presentation to your team on something they’ve worked on.” His reason is that he thinks communication is a key trait for the strongest engineers. We agree.

Jeff then goes on to say that you would probably want to complement this presentation with some hands on programming to make sure the engineer knows their stuff. This sounds like a good approach to interviewing. We combined the two, presentation and programming, in our recommended approach to the technical interview. You probably can’t go wrong with either approach.

Comments Off on Puzzle Questions in Interviews

Interviewing Engineers Continued – Part III (The Cultural Interview)

The most often overlooked aspect of interviewing a candidate is interviewing the candidate for a cultural “fit” within your team.   An employee who doesn’t fit in with the company culture can ruin team dynamics and actually reduce your engineering throughput rather than raising it.  The radio show This American Life did a show about bad apples in December 2008, Ruining it for the Rest of Us, which does a great job covering the downside of employees with bad attitudes.  

The biggest failure most managers have in the area of interviewing for cultural fit is that they far too often spend somewhere between 1 and 3 total hours with a candidate before making a hiring decision.  On top of this, they may have 2 to 5 people spend an hour each with the candidate. None of this is sufficient to get a good feel for the candidate and whether they will work well within the team.  You and your team probably spend more time researching major purchases such as a car, a house or even a flat screen TV set.  You and your team are probably going to spend 9 to 12 hours a day, 5 days a week with the newly hired employee.  There’s a good chance you don’t spend that much time with your significant other, your family or your friends within a week.

The most obvious aspects of this interview is to ensure that the candidate has the right work ethic, shares similar goals and aspirations, and works and plays well with others.  This is honestly easier said than done, but it is nevertheless an absolute necessity if you are to be happy with your hiring decision.  Here are some suggestions to get you started:

  1. Make a list of your current cultural aspects and those to which you aspire.  Is your team a bunch of workaholics?  Do they often get together after work and “blow off steam” or are they largely socially introverted?  Do you have a culture where people talk openly about issues?  Do people work mostly at work, or do they take work home with them?  Does the team joke around a lot, or are they “buttoned up”?   Is this person a “grandstander” or does he or she have a “hero mentality” or are they all about making the team perform better?
  2. Identify the folks on your team who exemplify the culture you have or the culture to which you are attempting to move the team and have them be part of the interview team.  Be careful only to select “A” players as you don’t want “B” and “C” players kicking out a potential “A” player for the wrong reason.
  3. Distribute a set of cultural questions that everyone should ask as a baseline so that you are reviewing the same questions.  You should encourage people to create their own as well as you want both innovation in the process and a good set of “control” questions.

Be careful not to confuse the need for diversity with the need for cultural fit.  You absolutely want different perspectives and approaches in your team.  This diversity leads to the abolition of “group think” and creates a higher performing team.  But you don’t want to bring a person who is constantly pointing fingers into a team that looks to solve problems together.

Before you hire anyone, each member of the interview team should have spent enough time with the individual to be able to answer the question of whether they “like” the person.  You aren’t really interviewing the person to see whether they could be a friend as you are still their boss, but you had better be able to answer the following questions at a minimum before offering someone a job:

  1. Can I be with this person 9 to 12 hours a day and still enjoy working with them?
  2. Can this person offer something to the team as a contributor and will the team accept the person given the behaviors I’ve perceived during my hours of interview and discussion?
  3. Can I learn from this person?  Note – this doesn’t mean that the person is smarter or has more experience than you much as it is a question of whether you can form a relationship with this person such that when they have valuable information or experience, you will be willing to accept it from them.
  4. Can I coach this person?

We propose that if the answer to any of the above questions is “No”, then the person is not a fit for you or your organization.

1 comment

Interviewing Engineers Continued – Part II (The Technical Interview)

The technical interview is important because besides the code test and the phone screen, which both can be misleading, it is the only chance you get to see what the engineer brings to the table in terms of their technical prowess.  First we should probably address why the code test and phone screen can be misleading.  Depending on how you conduct each of these there is a very real possibility that unsavory candidates will cheat to some degree on these.  During the phone screen you may hear frantic typing in the background, a possible hint that they are googling your questions.  If the coding test is done remotely or if the same one is given to everyone it’s likely, at least according to this article and this one, that a high percentage of candidates will cheat on it.  So that leaves us with the in person technical interview as the one remaining check we have of how good they are technically.

To start with we think there are at least seven areas of knowledge in which every engineer should have at least a solid understanding if not proficiency.   These areas are: 

  1. Algorithms, big-O or asymptotic notation used to describe upper and lower bounds for algorithms and how growth can be constant, linear, or logarithmic.  
  2. Object oriented programming, the fundamentals of inheritance, encapsulation, abstraction, and polymorphism and how these are implemented or not in their particular language of choice. 
  3. Syntax, data structures, and type of a primary language, be able to write a method on the board without glaring syntactic errors.
  4. Design patterns, why they exist and at least a couple of the more common ones such as factory and bridge. 
  5. PDLC, know the difference between Waterfall and Iterative lifecycles and be able to argue for the appropriate uses of each.
  6. Source Code Control, should believe in the concept and have knowledge of at least one platform.
  7. Compiler / Interpreter, should have an understanding of the debugger and that they be able to read and step through the compiler output (assembly equivalent) in the debugger.

How do you put these into a technical interview?  We recommend start off by picking your favorite sorting algorithm, such as a bubble sort.  Ask the candidate something similar to the following:

  • What are the steps involved in a bubble sort?
  • What is the worst and average complexity (big-O notation) of a bubble sort? 
  • In the expected language that will be used every day, write a method implementing a bubble sort on a white board.
  • What OOP properties are displayed in the code?
  • If the sort was written in C++ with an iterator what design pattern is being followed?  {or some language appropriate design pattern question}
  • What PDLC methodology did they most closely demonstrate when writing the bubble sort?
  • How would you check in this piece of code?
  • What are the steps that compiler goes through when compiling the bubble sort code? 

In a few quick questions you have covered in a concrete fashion the seven areas of knowledge that an engineer should possess.  Do you have any favorite engineering interview questions?


Interviewing Engineers Continued – Part I (The Recruiting Process)

We wrote almost a year ago about how to interview engineers.  It has been one of our most popular posts so we thought we’d expand on it a little.  In that post we outlined the general process that we recommend to hiring managers to follow.  These basic steps included: 

  1. Recruiting, hopefully with a knowledgeable technical recruiter if not give whoever is doing your recruiting a set of questions to ask.
  2. Phone screening, in order to save time do at least one phone screen with all candidates before bringing them on site.
  3. Code test, we recommend you use one for all engineering candidates so that everyone is treated the same.
  4. In person interview, conduct at least a technical, cultural, and logic interview for each candidate.
  5. Offer or decline, make a decision on each candidate quickly it reflects on your company.

Since writing the original post, we’ve decided that there are a number of ways that we can expand upon the topic.    This will be a three part series, covering in order the three areas that we believe to be most important when making a decision about having someone join your “family”.

This post will cover recruiting.  Article number 2 will be a deeper dive into the technical evaluation and article 3 will cover the “cultural fit” aspects of interviewing.

You should view your recruiting process as a pipeline that needs to remain full at all times if you want to quickly turn around and hire new engineers.  We’ve found that the five most successful sources of quality recruits, in no particular order are:

  1. Your own extended network
  2. Internally sourced candidates (candidates offered up by your engineering team)
  3. College sourced candidates
  4. Candidates from a professional recruiter
  5. Recruiting “fairs”

In the days before social networking sites, your personal network consisted of the people with whom you had contact information in your online and offline contact “rolodexes”.  Today they include the 1st, 2nd and 3nd order contacts within such popular business networking sites as LinkedIn, Plaxo and Facebook.  Call your close acquaintances, email those with whom you have some form of relationship and post messages to the remainder of your lesser known contacts when you have new jobs to fill.  Moreover, and to our point on pipelines above, work your network all the time for potential candidates and keep their resumes or contact information. 

You should also consider keeping a list of candidates that have been mentioned to you over time.  This might be in an excel spreadsheet or maybe even a little Access database.  Your HR team may have a tool for you to use to keep track of potential candidates whether or not you have a job open.  Or your company may have a relationship management system that you can use for this purpose, something like a Salesforce or Bullhorn implementation.

The important point to remember here is that recruiting is an activity you should ALWAYS be performing.  Don’t wait for an employee requisition to source qualified candidates.  You might not have a job to offer today, but if you are always looking to upgrade your team you may find an individual that is twice as good as the worst performer on your team.

The next source of qualified candidates is those recommended by your team.  During one on ones, talk to your superior performers and find out the names of folks with whom they’ve worked.  “A” players tend to run in packs and are often willing to recommend other “A” players.  “B” and “C” players do not often recommend people better than them, so be careful about sourcing candidates from these people.  Once again, this should be a career long activity – not just one you dread when you have a requisition.  Take people to lunch or breakfast or go out and have a beer and find out their interests.  Keep track of them and bring them in when you need them.

Colleges and universities are another source of candidates.  Far too many companies go to these places at the last minute when they need people.  Unfortunately for these companies, the great candidates are often picked up well before their graduation dates.  Some of the best recruiters in the industry scout for talent in graduate programs as if they were a baseball scout.  If you want quality candidates from universities, you need to forge a long term relationship with the university.  Start an intern program, wherein you are bringing in college sophomores and juniors and evaluating them over the course of 1 or 2 summers.  Then make the best ones offers before they go back for their senior year or during the holiday break.  Select one to three colleges with which you will partner closely.  You might have a tiered system consisting of a top-tier school (MIT, CMU or the like), a good public school (Wisconsin, University of Washington, University of Florida, University of Illinois, etc) and a local school.  You can get good talent at nearly any school, but the better the school the higher the likelihood that the school has already selected great talent for you.  Consider making a donation through your company to the school to forge a strong recruiting relationship, present at school events and get to know the students.

Professional recruiters either retained or contingent based, are another source of candidates.  This approach has the highest relative cost of all the options.  It’s probably not an option to keep your pipeline full constantly, as these companies hope to turn around engagements quickly.  They are a good source for when you have requisitions open and your other paths aren’t working well.  If you are a larger company, you might have full time recruiters.  You should keep the resumes of past individuals you thought of as “good” but did not hire for future needs.

Finally, when you have a lot of hiring to do and you need a large source of candidates, consider putting on a “fair”.  These events are productions, where you advertise in advance of the fair, have a company presentation, cater food and have a large number of your staff (only folks who are qualified to interview candidates) around to run potential candidates through a screening process.  

Remember – keep your pipeline full at all times.  In so doing, you’ll find that you can quickly turn candidates into employees and keep your team strong.

Comments Off on Interviewing Engineers Continued – Part I (The Recruiting Process)

How to interview engineers

The wrong selection for a new hire can make your life miserable whether you are a manager having to deal with counseling and possibly firing of the employee or a coworker and having to pick up the slack or deal with a wacko in a cube next to you.  Since the interview process is incredibly time consuming for the organization, the question we all have is:  How do you make the right decision?  Passing on great candidates out of fear can be just as detrimental as hiring the wrong person.  Here is our outline for how the interview process should flow:

Recruiting – Having a knowledgeable technical recruiter is a blessing.  They can start right away weeding out the candidates that are not up to speed on the technical skills that you need.  If your recruiter isn’t technically savvy, write up five questions for them to ask and the acceptable range of answers for each.

Phone Screen – This is a must it because it saves a ton of time and money.  We prefer to see two separate phone screens:  a technical screen performed by an engineer or manager and focused on the core technologies in which you are interested, and a second one focused on culture and fit.  For the technical interview prepare 5 – 10 questions ahead of time and try them on a coworker over lunch to see if someone can explain the answer sufficiently without writing anything down.  Engineers are notorious for being visual so this is a good check of your questions.  For five essential phone-screen questions try Stevey’s blog.  The second phone screen can be done by HR, a manger or an engineer and should be focused on culture and fit.  The goal is to compare this candidate’s style of work and play with that of the team’s. 

Code Test – This is can be a sensitive topic.  Some people swear by the code test, claiming it separates the wheat from the chaff, while others abhor it, thinking it too sophomoric.  Personally, we say that you have to do it.  Even if you’ve worked with someone in the past, treat all candidates the same and make them participate.  If you don’t want to do a separate test to evaluate an individual’s code, just incorporate the questions in the phone or in person steps of the process.

In Person – If the candidate has gotten past your recruiter, two phone screens, and possibly a code test, they are ready for the in person interviews.  I recommend three or more separate interviews with each interviewer assigned a specific subject matter to cover.  At a minimum you want one technical, one culture, and one logic (think puzzles) interview.  I like the technical interview broken out into two sections if possible; one for the specific programming language and the other focused on related technologies such as SQL.

Offer or Decline –The most important thing whether you are giving someone an offer or declining them is that you give them the feedback as soon as possible.  This will increase your chances of getting the candidate or leave them with a positive experience.  Dragging out the decision or the communication about that decision for days or weeks is guaranteed to leave a bad feeling with the candidate.  All offers should come with a drop dead date; this gives you the advantage over those other companies that can’t put offers together quickly.

Pitfalls – One of the most common reasons for a “mis-hire” in our experience is that not enough time was spent with an individual before offering him/her a job.  It is critically important that the hiring manager spend enough time with the candidate to “know” that they will be comfortable with him/her.  This is usually more than an hour.  The hiring manager must feel as comfortable as possible that the person is not only a good technical fit, but a good cultural fit as well.  A really talented engineer can completely ruin the culture of a high performing organization and cause it to underperform.  Unfortunately, it is not so easy for an underperforming organization to be changed culturally for the better with one really good hire.

We also suggest that you not give into the urge to have “everyone” interview a candidate.  If you’ve been in an interview process where you meet 8 to 10 people in 1 to 2 days for an hour each you know what we mean.  Wearing the candidate down with numbers doesn’t increase your probability of a good hire – more likely it will give you false positives or negatives.  Per our suggestion above, fewer people with more time spent with the candidate will likely yield a better result.