Introducing Pipedream

pipedream (noun)
(1) an integration platform for developers delivered as a single app
(2) a fantastic plan to make glue coding fun

We are excited to launch the beta release of Pipedream. In this post, we will share what Pipedream is, how it works, and what inspired its creation.

What is Pipedream?

Pipedream enables developers to easily build and execute workflows — often called “glue code” — that integrate apps, APIs and data, for free.

• “Easily build” = write production code in minutes with built-in state management, authentication, observability, and error handling
• “Execute” = we host and run your code on our infrastructure so you don’t have to manage any servers or cloud resources
• “Workflows” = compiled Node.js code, authored using code steps and pre-built actions, that runs on any trigger or schedule
• “Integrated Apps” = Google, AWS, Github, Gitlab, BitBucket, Heroku, Netlify, Twilio, Slack, Discord, Reddit, Yahoo! Fantasy and many more
• “Free” = no cost

Today, code is developed in our UI purpose-built for rapid development. Local development and a CLI are coming soon.

Video

This video explains how Pipedream works:

View and fork the workflow from the video, and a few more tutorials, here:

• Send new user signups to Slack, email 👋
• r/doggos subreddit top posts to Slack 🐶
• Save HTTP payloads to Amazon S3 for later analysis 🔎
• Save data to the Pipedream Data Warehouse, run SQL on it 📈

You can also read our docs to learn more.

What you can build

Developer use cases vary widely and we have already seen hundreds of customers use Pipedream to solve a wide range of problems:

• Query PostgreSQL on a schedule
• Send Webhook Data to Amazon S3
• Real-Time Sentiment Analysis
• “Google Alerts” for Hacker News
• New Subreddit top posts -> Slack
• Email yourself when someone stars your Github repo
• Fan out HTTP body to multiple webhooks
• Explore public workflows or build your own

The power of rolling your own + low code solutions

In an ideal world, you’d spend no time worrying about the machine that runs your app. You would simply focus on the code.

Serverless platforms like AWS Lambda demonstrate the power of this idea and you can spin up simple services in minutes without touching a server.

But running a service on Lambda is an investment. To build a Lambda that runs on HTTP requests, for example, you’ve got to understand the basics of API Gateway, IAM, and CloudWatch. As your app grows, you might need queues (SQS), state (Dynamo), and more.

Example Lambda that processes Github webhooks

When you’re building a complex product, and each service serves a purpose, this flexibility is awesome. But when you’re writing glue code, AWS is too low-level and complex.

On the other end, tools like Zapier and IFTTT facilitate integrations without writing any code. They manage the execution of tasks, so there’s also no server to run, and they provide pre-built actions you can run on triggers from virtually any SaaS app.

But as a developer, programming in these tools is unnatural and you are limited when you try to do anything complex or run high-volume tasks.

Pipedream implements the best of both approaches. We manage the server, execution environment, deployments, and scale, and provide you with thousands of pre-built actions across hundreds of integrated apps and services.

Furthermore, you can run any Node code, and require any npm package, at any step. (Yes, Python is coming.)

Instead of cloud functions or web services, you build workflows. A workflow is a linear sequence of steps triggered by some event (like an HTTP request), a cron schedule an email, events from a SaaS app, and more.

Example workflows from Pipedream

Building a developer community

We’ve spent the last year building the foundations of the platform: the workflow UX, app integrations, pre-built actions, and more.

But with thousands of apps and APIs, and innumerable potential workflows, we’re a long way from solving all the problems you encounter every day. We can’t develop all the integrations and actions necessary to solve every use case.

Nearly every open-source and commercial effort to enable developers to easily integrate apps, APIs and data, has failed to engage and inspire a large community of developers.

So, why do we believe Pipedream will be different?

1. Focus on developer experience and productivity. Pipedream should feel intuitive for those writing their first lines of Node, or those writing no code at all. But it should be powerful enough for experienced developers to write complex integrations quickly.
2. Support low volume, edge use cases. Most workflows are low volume, and will never justify a paid, enterprise product. Many of these workflows are custom, where you may need the control code affords. We believe providing a free, yet powerful, tool to developers solving real problems will help us integrate a wide range of services quickly.
3. Code transparency improves peer production. All code on Pipedream is public, all data is private. We believe public code and shared actions will improve your individual productivity and benefit the broader community.
4. We need your feedback. For this vision to succeed, we need you to write and run code on Pipedream. Tell us what you like and what we can improve. Where is the programming interface awkward? What apps are missing? Is there anything we can build to save you dev time? We listen to every piece of feedback we receive and ship multiple times a day to deliver the best possible product for you.

How you can help

Join our Slack, send us an email, or find us on Twitter — we want any and all feedback. And when you build a workflow that solves a real problem, share it with us and the rest of the community.