Airbyte, an open source data integration platform for building ELT data pipelines, today announced the general availability of its cloud service. The company is also detailing its new compensation plan for open source developers who write and maintain data connectors for the platform and it plans to launch its new command-line interface later this week, too.
The promise of Airbyte is that it allows businesses to easily create resilient ELT (extract, load and transform) pipelines by either using the company’s pre-built connectors or by building their own. The fast-growing company, which recently raised a $150 million Series B funding round, first released its cloud service into a limited beta last October and has since improved a number of user experience aspects of it, including the onboarding process, and taken the time to learn about how its cloud users’ needs are different from those of its open source users.
Image Credits: Airbyte
“I don’t think you can just translate open source to cloud. There are some adaptations to do on the product,” explained Airbyte’s co-founder and CEO Michel Tricot. The open source users, he noted, tend to be a bit more technical and willing to get their hands dirty to make things work. The cloud users, unsurprisingly, just want things to work.
Earlier this year, the company announced that it saw its deployments grow by 6x in 2021, going from 400 per month in Q1 to 2,500 by the end of the year. It now offers over 165 data connectors and expects to expand this to 500 by the end of the year.
Compensating open source contributors is a bit of a hot issue right now. Airbyte is opting for an interesting model here by opting to compensate those developers who write, maintain and fix connectors for its platform.
Image Credits: Airbyte
“We’ve always had contributions — and a growing number of contributors every week,” Airbyte co-founder and COO John Lafleur said. “One way to make sure that part of the community is focused on what has the most impact for the user community is to incentivize them. We’ve been doing a beta test for that over the last few weeks and it’s about giving some kind of bounty, some rewards, to address some issues in priorities. Meaning getting a connector from alpha to beta and from beta to production and address what is needed to get to GA certification — or bug fixing.”
In practice, this means a developer who adds an API stream to a connector can get up to $300, for example, with more incremental updates earning up to $200.
As for the new command-line access, Tricot noted that Airbyte tends to have to kinds of users. The more technical data engineers who want programmatic access to Airbyte and then data analysts who want a UI-based service. Airbyte always offered programmatic access, but Trico noted that the community was asking for a way to access the service programmatically that incorporated all of the GitOps best practices to access the service. With this new command line, users can keep their configuration and the state of their connection on Git and incorporate those into their existing CI pipelines.
Looking ahead, the Airbyte team noted that it is working on making it possible for its cloud users to manage both the data and control plane. “It’s part of our general north star forever, which is ubiquity and reliability. Ubiquity means we need to be able to access data wherever it is, whether it’s on our cloud, whether it’s on AWS, Azure, GCP or in different geographic zones,” Tricot said.
He also noted that while Airbyte right now focuses on the centralization of data into data warehouses, the next part of the journey will include more emphasis on moving that data into APIs to operationalize it (basically, the other side of the coin of data pipelines). The first step for many data teams is to build a foundation, which is what Airbyte is focusing on now, but the natural next step then is to unlock new use cases for that data.
“With Airbyte Cloud, we remove the headache of building and maintaining custom data infrastructure by providing a simple, economical way for enterprises to move data as needed,” said Tricot. “Just as importantly, we’re able to assure that as Airbyte data connectors enter general availability, they meet quality goals, especially for the long tail of data connectors that would otherwise be orphaned, by providing compensation for maintainers. We also feel it is critical, considering the recent Linux Foundation findings that contributors not just be recognized, but be compensated.”
