A key dilemma in the Big Data market has been the required level of skill and knowledge to successfully build business applications of value. Enter Big Data start-up Continuuity, who seeks to address this issue with their Java-based infrastructure Reactor, giving companies the power to more efficiently build big data apps. We spoke with CEO Jonathan Gray about how Continuuity has drastically altered the Big Data industry in its short period of existence, constructing a Platform-as-a-Service (PaaS) Hadoop framework designed by developers, for developers.
When you moved from Facebook to found Continuuity, what gap did you see in the Big Data market that you were aiming to address?
I knew I wanted to start Continuuity before I went to Facebook, but at the time I didn’t think the infrastructure – the open-source software, specifically HBase — was mature enough to build the company around it.
At Facebook, we were very successful with building a lot of really cool, powerful, large-scale applications on top of Hadoop and HBase and putting those into production. The problem is it took a team at Facebook of 15 of the best engineers I’ve ever worked with to build an application and run that application in production. The gap, and what Continuuity is trying to address, is that the rest of the industry does not have 15 top-notch engineers to dedicate to their infrastructure. There are 9 million Java developers in the world, but probably just a few thousand people who are actually proficient with building applications on Hadoop.
How do we address that gap? How do we empower those 9 million Java developers to be able to solve their business problems – to focus on their business problem rather than spend two years understanding the low-level infrastructure software? We’re really trying to shorten the amount of time it takes for this market to mature so we can start to focus on applications of business value rather than infrastructure.
Do you think your initial vision is still very important to the direction your company is heading?
Absolutely. When we first started the company, we were really talking about applications on top of Hadoop. We were talking about real time on top of Hadoop. And we were pretty early as far as being in the market and talking about those things. Now we hear it all the time — lots of people are talking about applications, lots of people are talking about real time. And the shift is starting to go toward developers who have largely been ignored in this ecosystem so far.
We’re getting validation from people who are starting to do things like us. We’re getting validation in how our positioning and messaging in our product are resonating in the market through our customers and our engagement. I think we were right to set out to focus on the developer. It’s a unique focus for us — it’s differentiated, and there’s a need for it in the market that’s currently under-served. We did pick the right vision, and we’ve stayed true to that vision pretty well so far.
How does Reactor differ from other Big Data applications available in today’s market?
The differentiator for us is the fact that we’re oriented for developers. There are companies out there delivering off-the-shelf solutions, and there are companies out there delivering BI tools, visualization tools, things for an analyst or a scientist to work on. But a lot of the companies that we deal with want customization.
The out-of-the-box application does not work for them, so they’re coming to us to enable them to develop their own solutions internally. What we’re selling to them is reducing time to market, reducing the number of developers you have to have, reducing the talent those developers need to have. It’s about being able to get your application to market faster, which is much easier to do on our platform versus doing it straight on the infrastructure.
What’s the most common issue you see new clients trying to solve by switching to the Reactor framework to store their data?
I’d say the biggest thing would be to move to real time. Most of our customers already use Hadoop. Largely, they’re using it just for an offline MapReduce application. And a lot of times, our customers are trying to get more real time. If they’re doing analytics, they want to be able to do them in real time. If they’re doing targeting, they want to be able to actually change their models based on what’s happening right now.
How would you describe your ideal customer? Do you have a specific type of customer in mind when you’re working on new developments for Reactor?
We have two main types of customers. We have large enterprises, Fortune 100–type companies, where we’re an internal developer pad. These companies have a lot of developers, and we function as an internal platform layer on top of Hadoop. On the other side are the more Web 2.0, Web 3.0, Internet, media, advertising, gaming-type companies. They have already adopted Hadoop for the most part, and we’re helping them to do more real time and build more dynamic applications.
I think we are very customer and use-case oriented in terms of how we determine what to build, what’s in our roadmap, what features we have. We engage with our customers through pretty close partnerships and around what we do with POCs – proof of concepts. We only came out to market at the end of March, but we’re working very closely with our customers to make sure that what we’ve built is solving their problems.
Considering the exponential growth of the modern Big Data market and the number of companies joining the game, how does Continuuity plan to remain relevant in the industry?
Developers-first is the number one reason why I think we’ll be successful. Right now we’re the only people who are myopically focused on the developer first. We’ve built a rock-star team that came from the Hadoop community– we have been there and seen it and done it, and we know what the pain is of doing it. If that’s our opportunity, then that’s our goal to be in that position and to be the de-facto development platform for Big Data.
We’re also very active in the existing open-source communities, and we have new open-source projects that we’ll be releasing. Our plan is to stay relevant in the community and open-source side of Big Data as well, which I think is very critical. Focusing on what we’re great at and partnering for gaps that we’re not focused on is ultimately how we can stay relevant in the market.
What challenges have you faced trying to compete in a robust industry that seems to be expanding exponentially?
I think for us and for virtually everybody else in our industry, recruiting is probably the biggest challenge. We need really, really great people with a lot of expertise in Hadoop, and finding those people is pretty hard. We’re lucky because we have a great story to tell and we’re a for-developer-by-developer company. But it’s still a constant challenge that everyone is dealing with in our space.
I think the second thing is Hadoop and Big Data are really hard. Most people out there do not have the expertise or experience to be able to build apps, so there’s a lot of hand-holding that needs to be done. A company like ours does not want to be in the solutions business, but we can very easily get dragged into being a solutions company, say if we go to a customer and they want us to build apps for them. It’s up to us to stay true to being a platform company and stay true to our vision.
Were there any business ideas or practices that you adapted from Facebook when starting Continuuity?
I learned a lot about culture and how you run an engineering organization, and there’s definitely a big influence of that in our company. We’re not totally to the same extent as Facebook, but we’ve adopted that same culture of trust and empowerment and of not lowering your hiring bar, even if it means you’re understaffed. That’s something that Facebook really stayed true to, and it was one of the best decisions they ever made. I took a lot of influence from how you get things done, how you create happy engineers, how you have a good company work culture.
The software that I built at Facebook really also informs the software we’re building now. One of the things that I did at Facebook was build the system that moved their MapReduce to real-time apps, and now I get to bring all the expertise that I gained here. I’d say the kind of techniques that I learned there about how you deal with scalable systems and scalable software have also been really valuable for us.
Where do you envision Big Data heading in the near future? Have you noticed any developing trends that users can expect to see in coming months?
If I’m going to look at one technology that’s coming out this year and how it’s going to drastically change what people call Hadoop and what people are doing on Hadoop in 2014, it’s YARN (the new version of Hadoop). I think you’re going to see a lot of interesting things happen on top of Hadoop – where people see it today as a kind of internal developer PaaS – but for unstructured data, map and scale, it’s cheaper, better, faster. I think the move towards applications, the move towards operational analytics, and the move towards real time are going to really expand what people think about Hadoop.
Continuity is adopting YARN heavily — one of our biggest developments is our stream processing engine, which is an application for Hadoop. Right now that’s part of our platform and our customers are using that production, but it’s something we’ll be bringing into open source. We’re hoping the rest of the Big Data ecosystem can also adopt this new technology. It’s a new type of application — it’s very real time, and it supports a very different set of applications than what people know Hadoop as today.
Learn more about Continuuity and their Reactor framework at www.continuuity.com.