We published recently 2 new dates for open courses and 1 new workshop, see their details below.
Workshop: Getting started with Apache Kafka Connect, November 6th, online
Open Course: Understanding Apache Kafka, November 15 – 17, online
Open Course: Big Data Architecture overview, December 7 -10, online
We are launching in April two online workshops that should get you started with both Python and Machine Learning in the easiest way. Instead of local installations (in order to have a Python environment like Apache Jupyter) – we will use Google Colab (an online notebook environment that allows writing and executing of Python code). As for Machine Learning library we are going to use scikit-learn – probably the most known and used ML open library. The workshops are build independently so that one can participate to either one or both of them, but in case you want to get started in this field we recommend both getting familiar with Python and the data science libraries and then learning how to build and test and ML model.
More info on the workshops can be found below:
Workshop 1, April 3rd: An Easy Slide into Python for Data Science , trainer Ionut Oprea
Workshop 2, April 17th: Start learning ML with Sklearn and Google Colab , trainer Maria Catana
Learning a new solution or building an architecture for a specific use case is never easy, especially when you are trying to embark alone on such an endeavour – thus in 2020 bigdata.ro started a new way of learning specific big data solutions/use cases: working groups. In 2020 we started 3 working groups:
And with 2 of the groups completed and the Decision Trees one to be completed soon, we are now opening registration for a new working group – this time focused on Apache Flink: How to process streaming data with Apache Flink and Apache Pulsar/Apache Kafka. The working group aims to take place December – February and will bring together a team of 5-6 participants that will define the scope (Kafka/Pulsar and the exact use case), select the data (open data), install the needed components, implement the needed flow.
More details and registration here.
Open Course: Understanding Apache Kafka
Timeline & Duration: November 20th – December 15th, 4 X 5 hours online sessions, during 4 weeks (1 sessions/week on November 20 & 27, Dec 4 & 15, to be decided the actual hours). An online setup will be available for exercises/hands-on sessions for the duration of the course, but also local installations will be guided for the course project completion.
Main trainer: Valentina Crisan
Location: Online (Zoom)
Price: 350 EUR (early bird 300EUR if payment is done until November 15th)
Pre-requisites: knowledge of distributed systems, SQL syntax for the KSQL part of the course
More details and registration here.
Following our bigdata.ro Spark (completed) and Druid+Superset working group (ongoing) , we are now opening registration on a new working group with the topic: Machine Learning with Decision Trees and Random Forest.
Timeline: September 27th – December 4th (1 online meeting/ 2 weeks, 5 meetings)
Predefined topic: Solving a regression/classification problem using decision trees and random forest.
We will use: Python, Jupyter notebooks, sklearn, dtreeviz for tree visualisation, matplotlib/seaborn for data visualization
The group’s purpose is to go through the entire ML flow while learning about decision trees and random forest.
You can check out the group’s description and register here.
Authors: Cosmin Chauciuc, Valentina Crisan
Throughout our training journeys with bigdata.ro one of the most encountered questions at the end of a course (that usually runs in the cloud on an already installed/configured setup) is: “how do I now setup my own small big data system”? Most of the time the question refers to a local installation – not something in the cloud – but to something that can be used for testing different small use cases and get someone started with their big data curiosity/exploration.
Now, getting started with big data in general can have 2 completely different approaches depending of the intended use case:
The blog post below has been drafted and revised by me (Valentina Crisan) but actually all steps were described and detailed by one of my former students in the Big Data Architecture and Technologies open courses: Cosmin Chauciuc. I am so proud to see someone taking forward the basis laid by my course and wanting to expand and become proficient in the big data field.
The blog post will have 3 parts:
We assume the reader already knows something about the solutions mentioned in this post (Kafka, Cassandra, Spark, Zeppelin) – thus we won’t go into the basics of what these are.
We will use Ubuntu Linux distribution for this installation – thus please follow Step 0 for one of the alternatives of installing Ubuntu on your system. Please note for a complete setup you will need a minimum of 8GB RAM – see in next chapter the RAM requirements for each service.
Ubuntu is an open source linux distribution, based on Debian, very easy to set up and have a full linux operating system. There are various ways to have Ubuntu on your laptop with your current windows installation.
Now, why 8GB RAM? See below an estimated RAM for each of the services :
4. Create a virtual hard disk, VDI, with dynamically allocated size. Depends on what you plan to install, allocate 20-30 GB size.
5. The last step is to attach to the newly created virtual machine, the ISO file: settings , storage : Live CD. This will let you boot from the ISO file.
Ubuntu can be used as a Live system or it can be installed on the VDI hard drive. The difference is that the live OS allows you to use it, but it is not persisting changes/files/new software installed, at reboot. If we want to proceed with installation of big data tools, we have to install the system.
After you have Ubuntu OS installed you need to process with the next Installation steps :
1. Confluent Kafka
2. Apache Cassandra
3. Apache Zeppelin – with embedded Apache Spark local
4. Apache Zeppelin configurations for Spark interworking with Kafka and Cassandra configurations
Note: you might observe that all solutions above are Apache besides Kafka that we chose to be a Confluent installation. The reasoning is simple – Confluent has a one node installation available that has all Kafka services in one node: Apache Kafka (Zookeeper + Kafka broker), Schema Registry, Connect, KSQL to name a few. This one node installation is meant for testing purposes (not for commercial/production ones). In production you will need to install these services from scratch (unless you choose a commercial license installation from Confluent), but this is a setup we found best to get you started and give you a good glimpse into what the Kafka ecosystem can give you as functionality. Nonetheless, if you don’t want to go for Confluent Kafka one node installation at the end of this post you have as well the Apache Kafka version – in this case you will have only Kafka – without the Connect, SchemaRegistry and KSQL.
sudo apt install curl
sudo apt install openjdk-8-jdk openjdk-8-jre
$ java -version openjdk version "1.8.0_265" OpenJDK Runtime Environment (build 1.8.0_265-8u265-b01-0ubuntu2~20.04-b01) OpenJDK 64-Bit Server VM (build 25.265-b01, mixed mode)
curl -O http://packages.confluent.io/archive/5.5/confluent-5.5.1-2.12.zip
unzip confluent-5.5.1-2.12.zip cd confluent-5.5.1/
We’ll use the default configuration files.
<path-to-confluent>/etc/
~/confluent-5.5.1/bin$ ./confluent local start
The local commands are intended for a single-node development environment
only, NOT for production usage. https://docs.confluent.io/current/cli/index.html
Using CONFLUENT_CURRENT: /tmp/confluent.kCNzjS0a
Starting zookeeper
zookeeper is [UP]
Starting kafka
kafka is [UP]
Starting schema-registry
schema-registry is [UP]
Starting kafka-rest
kafka-rest is [UP]
Starting connect
connect is [UP]
Starting ksql-server
ksql-server is [UP]
Starting control-center
control-center is [UP]
echo "deb https://downloads.apache.org/cassandra/debian 311x main" | sudo tee -a /etc/apt/sources.list.d/cassandra.sources.list
curl https://downloads.apache.org/cassandra/KEYS | sudo apt-key add -
sudo apt-get update
sudo apt-get install cassandra
sudo service cassandra start
sudo service cassandra stop // only if you want to stop the service
nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 127.0.0.1 70.03 KiB 256 100.0% 1c169827-bf4c-487f-b79a-38c00855b144 rack1
$ cqlsh Connected to Test Cluster at 127.0.0.1:9042. [cqlsh 5.0.1 | Cassandra 3.11.7 | CQL spec 3.4.4 | Native protocol v4] Use HELP for help. cqlsh>
$ wget http://mirrors.m247.ro/apache/zeppelin/zeppelin-0.9.0-preview2/zeppelin-0.9.0-preview2-bin-all.tgz
$ tar xzvf zeppelin-0.9.0-preview2-bin-all.tgz
$ cd zeppelin-0.9.0-preview2-bin-all $ ./bin/zeppelin-daemon.sh start Log dir doesn't exist, create /home/kosmin/zeppelin-0.9.0-preview2-bin-all/logs Pid dir doesn't exist, create /home/kosmin/zeppelin-0.9.0-preview2-bin-all/run Zeppelin start [ OK ]
Default login is with an anonymous user. The configuration for users is found in conf folder from <zeppelin_path>
$ ~/zeppelin-0.9.0-preview2-bin-all$ cd conf/ $ mv shiro.ini.template shiro.ini $ vi shiro.ini #uncomment the line with admin = password1, admin save restart zeppelin service
sc.version res3: String = 2.4.5
An Apache Zeppelin interpreter is a plugin that enables you to access processing engines and data sources from the Zeppelin UI. In order to connect our installed Zeppelin to Kafka we will need to add some artifacts in the Spark interpreter in Zeppelin configurations. More info on Zeppelin interpreters can be found here:
sc.version - in the zeppelin notebook res1: String = 2.4.5
org.apache.spark:spark-sql-kafka-0-10_2.11:2.4.5
sc.version - in the zeppelin notebook res1: String = 2.4.5
com.datastax.spark:spark-cassandra-connector_2.11:2.5.1
Version compatibility for the Spark-Cassandra connector can be found here.
kosmin@bigdata:~/confluent-5.5.1/bin$ ./kafka-topics --create --zookeeper localhost:2181 --replication-factor 1 --partitions 3 --topic bigdata Created topic bigdata. kosmin@bigdata:~/confluent-5.5.1/bin$ ./kafka-console-producer --broker-list localhost:9092 --topic bigdata enter some words here :) CTRL+C // if you want to close producing events in the topic
val kafkaDF = spark.readStream.format("kafka")
.option("kafka.bootstrap.servers", "localhost:9092")
.option("subscribe","bigdata")
.option("startingOffsets","earliest")
.load().select($"key".cast("STRING").as("key"),
$"value".cast("STRING").as("value"),
$"topic",
$"partition",
$"offset",
$"timestamp")
val query_s0 = kafkaDF.writeStream.outputMode("append").format("console").start()
query_s0.awaitTermination(30000)
query_s0.stop()
import org.apache.spark.sql.functions._
val tables = spark.read.format("org.apache.spark.sql.cassandra").options(Map( "table" -> "tables", "keyspace" -> "system_schema" )).load()
tables.show(10)
kosmin@bigdata:~/confluent-5.5.1$ ./bin/zookeeper-server-start ./etc/kafka/zookeeper.properties
In another terminal window, we can check if Zookeeper started. By default, it listens on 2181 port.
$ ss -tulp | grep 2181
tcp LISTEN 0 50 *:2181 *:* users:(("java",pid=6139,fd=403))
For starting Kafka service, run this in another terminal :
kosmin@bigdata:~/confluent-5.5.1$ ./bin/kafka-server-start ./etc/kafka/server.properties
Check if Kafka started in another terminal window. Default Kafka broker port is 9092.
kosmin@bigdata:~/confluent-5.5.1$ ss -tulp | grep 9092
tcp LISTEN 0 50 *:9092 *:* users:(("java",pid=6403,fd=408))
kosmin@bigdata:~/confluent-5.5.1/bin$ ./kafka-topics --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic Hello Created topic Hello. kosmin@bigdata:~/confluent-5.5.1/bin$ ./kafka-topics --list --zookeeper localhost:2181 Hello __confluent.support.metrics _confluent-license kosmin@bigdata:~/confluent-5.5.1/bin$
At this moment you should have your own private small cluster ready for learning some of the solutions in Big Data Architectures. Although with such setup you cannot see the wonder of distributed systems, you can though understand the purpose of Kafka/Cassandra/Spark/Zeppelin in a big data architecture.
Next post in these series will be about building an end to end case with this setup.
Finding trending Meetup topics using Streaming Data, Named Entity Recognition and Zeppelin Notebooks – a tale of a super enthusiastic working group during the pandemic times.
Author : Andrei Deusteanu
Project Team: Valentina Crisan, Ovidiu Podariu, Maria Catana, Cristian Stanciulescu, Edwin Brinza, Andrei Deusteanu
We started out as a working group from bigdata.ro. Our main purpose was to learn and practice on Spark Structured Streaming, Machine Learning and Kafka. We built the entire use case and then the architecture from scratch.
This is a learning case story. We did not really know from the beginning what would be possible or not. For sure, looking back, some of the steps could have been optimized. But, hey, that’s how life works in general.
Since Meetup.com provides data through a real-time API, we used it as our main data source We did not use the data for commercial purposes, just for testing.
The problems we tried to solve:
For these we developed 2 sets of visualizations:
In order to annotate the data, we need to transform it into a certain format, from text to numbers, and then back to text.
We used a machine from Hetzner Cloud with the following specs:
These visualizations are done on data between 8th of May 22:15 UTC and 4th of June 11:23 UTC.
As you can see, most of the RSVPs occur in the United States, but the city with the highest number of RSVPs is London.
As you can see, most of the RSVPs are in the largest cities of the country. This is probably due to the fact that companies tend to establish their offices here and therefore attract talent to these places.
* This was produced with Grafana using RSVP data processed in Spark and saved locally.
* This was produced with Grafana using RSVP data processed in Spark and saved locally.
This visualization is done on data from July.
This visualization is done on almost 1 week of data from the start of August. See Issues encountered chapter, point 5.
All of these are mentioned in the published Notebooks as well.
This study group was a great way for me to learn about an end-to-end solution that uses Kafka to ingest streaming data, Spark to process it and Zeppelin to build visualizations on it.
It was great that we all had different backgrounds and we had interesting debates about how to approach problems and how to solve them. Besides working with and getting to know these nice people, I got to learn the fundamentals of Kafka, how to use Spark Streaming to consume Kafka events, how a Named Entity Recognition system works, and how to use Spark NLP to train a NER model and make predictions with it. I’ve especially enjoyed one of our last meetings, where we worked together for a couple of hours and managed to build some great visualizations for meetup keywords and RSVPs using Zeppelin and Grafana.
On the downside, the project took almost twice as long to complete than we originally planned – 18 weeks instead of 10. Towards the end this made it hard for me to work on it as much as I would have wanted to because it overlapped with another project that I had already planned for the summer.
All in all, I would recommend this experience for anyone interested in learning Big Data technologies together with other passionate people, in a casual and friendly environment.
Course date duration: August 8th, 2020, 9:30 – 14:00, 30 min break included
Trainer: Lucian Neghina
Location: Online (using Zoom)
Price: 150 RON (including VAT)
Number of places: 10 3 places left
When you need to design a data solution one of the earliest questions is where your data is coming from and how you will make it available to the solution/solutions that processes/stores the data. Especially since data we might deal with IoT data, thus various sources, and data will be as well processed and stored by several components of your solution. Even more nowadays that we work mainly with streams not with static data such a solution that is able to design and run the flow of events from the source/sources to the processing/storage stage it’s extremely important. Apache NiFi has been built to automate that data flow from one system to another. Apache NiFi is a data flow management system that comes with a web UI that helps to build data flows in real time. It supports flow-based programming.
You can check out the agenda and register here.
Open Course: Understanding Big Data Architecture E2E (Use case including Cassandra + Kafka + Spark + Zeppelin)
Timeline & Duration: July 27th – August 14th, 6 X 4 hours online sessions, during 3 weeks (2 sessions/week, Monday + Thursday) . An online setup will be available for exercises/hands-on sessions for the duration of the course.
Main trainer: Valentina Crisan
Location: Online (Zoom)
Price: 250 EUR
Pre-requisites: knowledge of distributed systems, Hadoop ecosystem (HDFS, MapReduce), know a bit of SQL.
More details and registration here.
Learning a new solution or building an architecture for a specific use case is never easy, especially when you are trying to embark alone on such an endeavour – thus in 2020 bigdata.ro started a new way of learning specific big data solutions/use cases: working groups. And with the first working group (centered around Spark Structured Streaming + NLP) on its way to completion in July, we are now opening registration for a new working group – this time centered around Apache Druid: Building live dashboards with Apache Druid + Superset. The working group aims to take place End of July – October and will bring together a team of 5-6 participants that will define the scope, select the data (open data), install the needed components, implement the needed flow. Besides the participants for this group we will have a team of advisors (with experience in Druid and big data in general) that will advise the participants on how to solve different issues that will arise in the project.
Find more details of the working group here.
bigdata.ro 