AI News, The Next Big Thing You Missed: Airbnb's Human Brains Crunch Data Better Than Computers

The Next Big Thing You Missed: Airbnb's Human Brains Crunch Data Better Than Computers

But as Newman's ultimate solution goes to show, sometimes the best resource for solving data problems is the one that gets the least credit: the human brain.

For startups especially–even though so many of their brethren love to hawk automated thinking–the most efficient tool for the job might be made of flesh and blood.

At Airbnb, Newman first did some fairly standard computer-aided number-crunching to learn that most calls came either when customers were booking rooms or heading out on their trips.

To find out more about what was going on, Newman assembled a small team that became known as the "Air Divers"–the people who would dive deep into the individual complaints and surface with answers.

Each was given a couple hundred support tickets connected to a specific issue that the data had identified as a hot-button topic.

During a later presentation at the conference, a leader of the IBM team that oversees Watson, its Jeopardy-winning supercomputer, described the effort to model human behavior in computers as a shift toward probabilistic calculations.

The approach might seem quick and dirty, but the data-processing might of the human brain was enough to bring customer service calls down 75 percent.

The Next Big Thing You Missed: Airbnb's Human Brains Crunch Data Better Than Computers

But as Newman's ultimate solution goes to show, sometimes the best resource for solving data problems is the one that gets the least credit: the human brain.

For startups especially–even though so many of their brethren love to hawk automated thinking–the most efficient tool for the job might be made of flesh and blood.

At Airbnb, Newman first did some fairly standard computer-aided number-crunching to learn that most calls came either when customers were booking rooms or heading out on their trips.

To find out more about what was going on, Newman assembled a small team that became known as the "Air Divers"–the people who would dive deep into the individual complaints and surface with answers.

Each was given a couple hundred support tickets connected to a specific issue that the data had identified as a hot-button topic.

During a later presentation at the conference, a leader of the IBM team that oversees Watson, its Jeopardy-winning supercomputer, described the effort to model human behavior in computers as a shift toward probabilistic calculations.

The approach might seem quick and dirty, but the data-processing might of the human brain was enough to bring customer service calls down 75 percent.

At Airbnb, Data Science Belongs Everywhere

At that time, the few people who’d even heard of the company were still figuring out how to pronounce its name, and the roughly 7 person team (depending on whether you counted that guy on the couch, the intern, and the barista at our favorite coffee shop) was still operating out of the founders’ apartment in SOMA.

The trick has been to manage scale in a way that brings together the magic of those early days with the growing needs of the present — a challenge that I know we aren’t alone in facing.

So I thought it might be worth pairing our posts on specific problems we’re solving with an overview of the higher-level issues data teams encounter as companies grow, and how we at Airbnb have responded.

This will mostly center around how to connect data science with other business functions, but I’ll break it into three concepts — how we characterize data science, how it’s involved in decision-making, and how we’ve scaled it to reach all sides of Airbnb.

The foundation upon which a data science team rests is the culture and perception of data elsewhere in the organization, so defining how we think about data has been a prerequisite to ingraining data science in business functions.

While this may seem obvious, it doesn’t happen naturally — when data scientists are pressed for time, they have a tendency to toss the results of an analysis ‘over the wall’ and then move on to the next problem.

This isn’t because they don’t want to see the project through, but with so much energy invested into understanding the data, ensuring statistical methods are rigorous, and making sure results are interpreted correctly, the communication of their work can feel like a trivial afterthought.

We began with the centralized model, tempted by its offering of opportunities to learn from each other and stay aligned on metrics, methodologies, and knowledge of past work.

While this was all true, we’re ultimately in the business of decision-making, and found we couldn’t do this successfully when silo’d: partner teams didn’t fully understand how to interact with us, and the data scientists on our team didn’t have the full context of what they were meant to solve or how to make it actionable.

Over time we became viewed as a resource and, as a result, our work became reactive — responding to requests for statistics rather than being able to think proactively about future opportunities.

So we made the decision to move from a fully-centralized arrangement to a hybrid centralized/embedded structure: we still follow the centralized model, in that we have a singular data science team where our careers unfold, but we have broken this into sub-teams that partner more directly with engineers, designers, product managers, marketers, and others.

And by not fully shifting toward an embedded model we’re able to maintain a vantage point over every piece of the business, allowing us to form a neural core that can help all sides of the company learn from one another.

Once situated within a team that can take action against an insight, the question becomes how and when to leverage the community’s voice for business decisions.

Over time, we’ve identified four stages of the decision-making process that benefit from different elements of data science: Sometimes a step is fairly straightforward, for example if the context of the problem is obvious — the fact that we should build a mobile app doesn’t necessitate a heavy synopsis upfront.

Measuring the impact of a data science team is ironically difficult, but one signal is that there’s now a unanimous desire to consult data for decisions that need to be made by technical and non-technical people alike.

This has been trickier than one might expect because Airbnb’s ecosystem is complicated — a two-sided marketplace with network effects, strong seasonality, infrequent transactions, and long time horizons — but these challenges make the work more exciting.

How we scaled data science to all sides of Airbnb over 5 years of hypergrowth

At that time, the few people who’d even heard of the company were still figuring out how to pronounce its name, and the roughly seven-person team (depending on whether you counted that guy on the couch, the intern, and the barista at our favorite coffee shop) was still operating out of the founders’ apartment in SoMA.

The trick has been to manage scale in a way that brings together the magic of those early days with the growing needs of the present —

So I thought it might be worth pairing our posts on specific problems we’re solving with an overview of the higher-level issues data teams encounter as companies grow, and how we at Airbnb have responded.

It was construed purely as a measurement tool, which paints data scientists as Spock-like characters expected to have statistics memorized and available upon request.

While answering questions and measuring things is certainly part of the job, at Airbnb we characterize data in a more human light: it’s the voice of our customers.

Over time, our colleagues on other teams have come to understand that the data team isn’t a bunch of Vulcans, but rather that we represent the very human voices of our customers.

This isn’t because they don’t want to see the project through, but with so much energy invested into understanding the data, ensuring statistical methods are rigorous, and making sure results are interpreted correctly, the communication of their work can feel like a trivial afterthought.

But there’s also a strong belief in cross-functional collaboration at Airbnb, which brings up questions about how to structure the team within the broader organization.

We began with the centralized model, tempted by its offering of opportunities to learn from each other and stay aligned on metrics, methodologies, and knowledge of past work.

While this was all true, we’re ultimately in the business of decision-making, and found we couldn’t do this successfully when siloed: partner teams didn’t fully understand how to interact with us, and the data scientists on our team didn’t have the full context of what they were meant to solve or how to make it actionable.

So we made the decision to move from a fully-centralized arrangement to a hybrid centralized/embedded structure: we still follow the centralized model, in that we have a singular data science team where our careers unfold, but we have broken this into sub-teams that partner more directly with engineers, designers, product managers, marketers, and others.

Doing so has accelerated the adoption of data throughout the company, and has elevated data scientists from reactive stats-gatherers to proactive partners.

And by not fully shifting toward an embedded model we’re able to maintain a vantage point over every piece of the business, allowing us to form a neural core that can help all sides of the company learn from one another.

Once situated within a team that can take action against an insight, the question becomes how and when to leverage the community’s voice for business decisions.

But the reality of a hypergrowth startup is that the scale and speed at which decisions need to be made will inevitably outpace the growth of the data science team.

Early in the year, we were still a small company based entirely in San Francisco, meaning that our army of three data scientists could effectively partner with everyone.

Six months later, we opened more than 10 international offices simultaneously, while also expanding our product, marketing, and customer support teams.

We needed to find a way to democratize our work, broadening from individual interactions, to empowering teams, the company, and even our community.

Measuring the impact of a data science team is ironically difficult, but one signal is that there’s now a unanimous desire to consult data for decisions that need to be made by technical and non-technical people alike.

At a recent event, Airbnb's head of data science said it's important to listen to user-generated data, the voice if your customer.

At one point Riley Newman, Airbnb head of data science, took a look the company's booking numbers and compared it to the number of customer service inqueries.'People were contacting us at a rate that was like one to one,' Newman said Monday at DataBeat in San Francisco.

Riley Newman on data science for startups

When Riley Newman joined Airbnb nearly six years ago, he was one of 10 employees and working from the co-founders’ apartment.

Today he leads a team of 70-plus data scientists, analysts, and engineers, who are challenged with empowering more than 2,000 Airbnb employees with global insights.

recently caught up with Riley to chat about why it’s crucial for startups to invest in data early, the difficulties of keeping data accessible at scale, how his field’s findings represent the voice of the customer, and more.

Alongside helping to foster that growth I’ve built the data science function in the company, which is now roughly 60 data scientists and analysts, 10 data engineers, and a few others.

My career arc has been relatively quantitative, but it began in the field of economics and has since moved into the application of theory within a business context.

As the company has scaled and our product has reached a greater level of sophistication, the types of people that we hire have started to skew a little bit more toward computer science and statistical backgrounds – thinking about problems like machine learning.

As I mentioned the term data science and big data hadn’t really taken off yet, so it was especially anomalous that the founders brought me in that early.

They did bring me in very early, and increasingly startups are looking to build data teams earlier than you’d see some other functions kick off within the company.

There are lots of very young companies that have reached out and asked for advice about how to build the data science team from scratch, because they’re looking to do it straight away.

It’s actually more important that early stage startups build a strong data culture early, because in many cases it can be the difference between death and success.

There’s always this risk, particularly in a business like ours that’s primarily offline, that you will spend all of your time optimizing the online portion of the business and forget the actual experience that the guests and hosts of Airbnb.

The data science team can identify an anomaly and pass that anomaly off to the user research team, who can go deeper than the data lets us.

They can bring back a hypothesis in the form of a solution to the anomaly, which we can help put into production and test through a controlled experiment.

To give you a concrete example, we were looking at the conversion rates of our payments page and were cutting it by the country of origin of the person who’s trying to book.

Brian Chesky and Joe Gebbia, two of the three co-founders, talk a lot about how in the early days they would fly out to New York to meet with the first hosts of the Airbnb community and talk to them about what was happening, what worked for them about the experience, and what didn’t work.

I view the data science team as carrying that culture forward as the company has scaled – trying to keep us very much in touch with the people who are using the product.

It’s no longer possible to meet with every single person individually, but we do try to host meetups and stay connected to guests and hosts.

This really resonated with me, because when I was in grad school I focused on economic geography – how economic trends take place across space and how those different spaces or clusters influence one another.

We have a data science team, which is where people’s careers unfold, but that team is broken into these sub-units that are embedded with engineers, designers, and project managers.

Another one is there’s a long funnel of activity that has to take place before a data scientist can really do impactful work, and it’s important for startups to understand and respect that.

A lot of investment is required into the way data is logged and the infrastructure surrounding where it’s housed – the ETL pipelines for transcribing it from a raw log event into meaningful information.

Adam: If you do find yourself as a data scientist at a young company and you are running into those walls or feeling handicapped, how can you demonstrate the value of analytics across the company?

When you’re working with a group that is meant to take action based on a finding from a data scientist, they don’t really worry about the statistical voodoo behind the work you’ve done.

From that perspective something that we did early on that made a big difference is we placed equal weight on a data scientist’s ability to communicate their work as the technical rigor behind it.

So that when they came in we could ensure they would be very impactful and help teams understand why this way of thinking makes sense, why we use experiments to roll out products, and how we interpret those experiments.

We spend a lot of time trying to look for ways to create leverage for the team and scale the work of every individual within the team.

Every time you’d get back to a given question, it’d be like “How did I do this last year?” You’d dig around in Git for code and look in your email for the keynote presentation.

From my perspective as head of a data science team, you have to look as much for opportunities for leverage within the team as opportunities for impact within the company.

Riley: Many of the designers and PMs at Airbnb, when they think about the world a couple years ago, they would’ve been just as frustrated as data scientists with the difficulty of accessing data.

It speaks to the need for investment in infrastructure and tools surrounding data so that the company is as empowered as possible and the team is able to focus on the highest impact work.

It’s the role of the people within that organization to expand your knowledge of the ecosystem of your business, alongside partnering with engineers to build data products.

In each business case it’s slightly different, but it will be very obvious to the head of this function where those gaps lie, and that person should take responsibility for addressing them in doing whatever they can to democratize their team’s understanding of what’s going on and help create leverage for the people on their team.

find the most leverage from my work these days is really empowering the team of data scientists that I work with and ensuring they’re set up for success.

As we have gotten to a point where we have lots of data scientists, information, and technicality within the team, I still find us trying to answer the question of what is going on and how do we frame it in a way that is digestible to a fast-growing business?

We’re building a team that can serve as thought leaders within the data science organization and think about how to frame problems different ways and how to communicate a compelling and consistent narrative across different projects as well as across the business as a whole.

Chris Wallace interviews Russian President Vladimir Putin

Putin dismisses claims of Russian interference in the 2016 election, says Russia would react negatively to the expansion of NATO, blames terrorists for civilian ...

Shawn Rochester: "The Black Tax: The Cost of Being Black In America​" | Talks at Google

CEO of Good Steward LLC, Financial Coach, and Author Shawn D. Rochester discusses his new book, "The Black Tax: The Cost of Being Black in America".

Knowledge Graphs: The Path to Enterprise — Michael Moore and AI Omar Azhar, EY

Michael Moore, Ph.D. — Executive Director, EY Performance Improvement Advisory, Enterprise Knowledge Graphs + AI Lead, EY and Omar Azhar, M.S. ...

Tom Gayner: "The Evolution of a Value Investor" | Talks at Google

About the Talk Tom Gayner is the CIO of Markel Corp, where he manages the company's investment portfolio. He talks about his journey as an individual and ...

Genius of Things: Changing the way we live and work

Speakers: Harriet Green, GM Watson IoT; Bob Wolpert, Corporate Senior Vice President and President, Logistics for Golden State Foods; Doug Voda, Global ...

Sinead Kaiya: The marriage of data science and global enterprise

How big data technologies can make the world run better. At the second annual Women in Data Science conference, Sinead Kaiya, COO of Products ...

Megan Smith: Perspectives on artificial intelligence from the White House

The government is using artificial intelligence in tasks ranging from planning space missions to forecasting job growth. Given the potential effects of these ...