wtf is beast?

Beast is a tool that moves data from Kafka into Google BigQuery. Kafka is a popular system for handling real-time streams of data, like user clicks, app events, or transaction logs. BigQuery is Google's large-scale data warehouse where companies run analytics and queries. Beast bridges the gap: it reads batches of messages from Kafka and writes them into BigQuery tables so your data is ready for analysis. Under the hood, the system has three main pieces working together. A consumer pulls messages from Kafka in batches and holds them in internal queues. Workers then take those messages and push them into BigQuery. Once a batch is confirmed as successfully written, a committer tells Kafka that the data has been handled, so Kafka can mark it as done. If BigQuery rejects messages because they contain invalid timestamps, Beast can optionally set those aside in Google Cloud Storage so processing continues without stalling. This tool was built at Gojek, a large ride-hailing and payments company, so the primary audience is engineering teams dealing with high-volume event streams. A concrete example would be a product team that sends millions of ride-booking or payment events per day into Kafka and needs those events to land cleanly in BigQuery for dashboards and reporting. Beast handles the plumbing, retries, and error routing so the team doesn't build that pipeline from scratch. One thing to note is that Beast expects data to be in Protocol Buffer format, a structured way of serializing data. It is tightly coupled to the Google Cloud ecosystem, specifically BigQuery and optionally Cloud Storage. The project is designed to run on Kubernetes, with built-in metrics monitoring. The README doesn't go into detail on exactly how to configure every parameter, but it does point to Helm charts for deployment.