Working Group: Stream Processing with Flink + Pulsar/Kafka

Learning a new solution or building an architecture for a specific use case is never easy, especially when you are trying to embark alone on such an endeavour – thus in 2020 started a new way of learning specific big data solutions/use cases: working groups. In 2020 we started 3 working groups:

  • Spark Structured Streaming + NLP (completed)
  • Building live dashboards with Druid + Superset (completed)
  • Understanding Decision Trees (running until December)

And with 2 of the groups completed and the Decision Trees one to be completed in December, we are now opening registration for a new working group – this time focused on Apache Flink and Pulsar: How to process streaming data with Apache Flink and Apache Pulsar/Apache Kafka. The working group aims to take place December – February and will bring together a team of 5-6 participants that will define the scope, select the data (open data), install the needed components, implement the needed flow.       

Details of the working group are listed below. If you are interested to participate in this group please register using the form at the bottom of the page. 

What will this working group mean:

A predefined topic: How to process streaming data with Apache Flink and Apache Pulsar/Apache Kafka (we decide in the group if we use Kafka or Pulsar for this group).

A group of 5-6 participants and one predefined driver per group – the scope of the driver is (besides being part of the group) to organize the groups and provide the cloud infrastructure needed for installing the studied solution;

5 online meetings every 2 weeks (thus a 10 weeks time window for each working group, we will use Google Hangouts/Zoom). The meetings will take place Monday-Friday, in the interval 6PM – 9PM;

Active participation/contribution from each participant, for example each participant will have to present in at least 2 of the meetings to the rest of the group;

Some study @ home between the sessions;

The fee for participating in these working groups is 100 Euro/participant and will cover the costs with cloud infrastructure and other tools/logistics costs for the group meetings. 

Driver of Flink + Pulsar/Kafka working group: Valentina Crisan