bigdata.ro

In march we will restart our bigdata.ro sessions, workshops mainly aimed to help participants navigate easier through big data architectures and get a basic understanding in some of the possible components of such architectures. We have discussed in the past Cassandra, HDFS, Hive, Impala, Elasticsearch, Solr, Spark & Spark SQL, generic big data architectures and on March 16th we will continue our journey with one of the unusual children of noSQL: the graph database Neo4j. Not quite similar with the other noSQL siblings, this database is not derived from the likes of DynamoDB or BigTable like others do, but instead addresses relationship between data not just the data itself. The result is amazing, the use cases are incredible and Calin Constantinov will guide us through the basics of this interesting solution.   

See below a few questions and answers in advance of the workshop, hopefully these will increase your curiosity towards Neo4j.

Valentina Crisan – bigdata.ro

Calin Constantinov – trainer “Intro to Neo4j” workshop, March 16th

—————————————————————————————————————————————

What is a graph database and which are the possible use cases that favour such a database?

Calin: They say “everything is a graph”. Indeed, even the good old entity-relationship diagram is no exception to this rule. And graphs come with a great “feature” which us humans tend to value very much: they are visual! Graphs can easily be represented on a whiteboard and immediately understood by a wide audience.

Moreover, in a traditional database, explicit relationships are destroyed the very moment we store data and need to be recreated on-demand using JOIN operations. A native graph database has preferential treatment for relationships meaning that there are actual pointers linking an entity to all its neighbors.

I remember the first time I needed to implement a rather simple Access Control List solution that needed to support various inheritable groups, permissions and exceptions. Writing this in SQL can quickly become a nightmare.

But of course, the most popular example is social data similar to what Facebook generates. For some wild reason, imagine you need to figure out the year with the most events attended by at least 5 of your 2rd degree connections (friends-of-friends), with an additional restriction that none of these 5 are friends between them. I wouldn’t really enjoy implementing that with anything other than Neo4j!

However, not all graphs are meant to be stored in a graph database. For instance, while a set of unrelated documents can be represented as a graph with no edges, please don’t rush to using Neo4j for this use-case. I think a Document store is a better persistence choice.

In terms of adoption, 75% of the Fortune 100 companies are already using Neo4j. As for concrete use-case examples, Neo4j is behind eBay’s ShopBot for Graph-Powered Conversational Commerce while NBC News used it for uncovering 200K tweets tied to Russian trolls. My personal favourite is the “Panama Papers” where 2.6TB of spaghetti data, made up of 11.5M heterogeneous documents, was fed to Neo4j. And I think we all know the story that led the investigative team to win the Pulitzer Prize.

What graph databases exist out there and how is Neo4j different from those?

Calin: Full disclosure, given the wonderful community around it (and partially because it happened to be the top result of my Google search), it was love at first sight with me and Neo4j. So, while I’m only using Neo4j in my work, I do closely follow what’s happening in the graph world.

JanusGraph, “a graph database that carries forward the legacy of TitanDB” is one of the most well-known alternatives. A major difference is that JanusGraph is more of a “graph abstraction layer” meaning that it requires a storage backend instead of it being a native graph.

OrientDB is also popular do its Multi-Model, Polyglot Persistence implementation. This means that it’s capable of storing graph, document and key/value data, while maintaining direct connections between records. The only drawback is that it might have not yet reached the maturity and stability required by the most data-intensive tasks out there.

More recently, TigerGraph showed impressive preliminary results, so I might need to check that out soon.

Is the Neo4j architecture a distributed one? Does it scale horizontally like other noSQL databases?

Calin: The short answer is that Neo4j can store graphs of any sizes in an ACID-compliant, distributed, Highly-Available, Causal Clustering architecture, where data replication is based on the state-of-the-art Raft protocol.

In order to achieve best performance, we would probably need to partition the graph in some way. Unfortunately, this is typically a NP-hard problem and, more often than not, our graphs are densely connected which can really make some form of clustering quite challenging. To make matters worse, coming back to the Facebook example, we need to understand that this graph is constantly changing, with each tap of the “Like” button. This means that our graph database can easily end up spending more time finding a (sub-)optimal partition than actually responding to queries. Moreover, when combining a complex query with a bad partitioning of the data, you wind up with requiring a lot of network transfers within the cluster, which will most likely cost more than a cache miss. In turn, this could also have a negative effect on query predictability. Sorry to disappoint you, but this is the reason for which Neo4j doesn’t yet support data distribution. And it’s a good reason too!

So, a limitation in the way Neo4j scales is that every database instance has a complete replica of the graph. Ideally, for best performance, all instances need to have enough resources to keep the whole graph in memory. If this is not the case, in some scenarios, we can at least attempt to achieve cache sharding by identifying all queries hitting a given region of the graph and always routing them to the same instance. As a starting point, there is a built-in load-balancer which can potentially be extended to do that. Additionally, we can easily direct I/O requests intelligently in a heterogeneous environment, designating some Read Replicas for handling read queries while only writing to instances packing the most power. This is a good thing for read operations which can easily scale horizontally. Write operations are however the bottleneck. Nevertheless, the guys over at Neo4j are always coming up with clever ways to significantly improve write performance with every new release.

Does Neo4j work with unstructured/flexible structured data?

Calin: A graph is composed of nodes and relationships. We are able to group similar nodes together by attaching a label, such as “User” and “Post”. Similarly, a relationship can have a type, such as “LIKED” and “TAGGED”. Neo4j is a property graph meaning that multiple name-value pairs can be added both to relationships and nodes. While it is mandatory in Neo4j for relationships to have exactly one type, labels and properties are optional. New ones can be defined on-the-fly and nodes and relationships of the same type don’t all necessarily need to have the same properties. If needed, Neo4j does support indexes and constraints, which can, for instance, improve query performance, but this is as close as you get to an actual schema.

In regards to the question, the whole point of a graph database is to structure your data in some way. Keep in mind that the true value of your data lies within uncovering relationships between entities. If you feel like this doesn’t fit your use-case, coming back to the example of having a number of unrelated free-text documents or even some form of semi-structured data, while Neo4j now supports full text indexing and search, there are clearly better alternatives out there, such as key-value and document stores.

How is it best and easiest to get started with Neo4j?

Calin: Apart from attending my workshop, right? I think the best way to get up to speed with Neo4j is to use the Neo4j Sandbox. It’s a cloud-based trial environment which only requires a browser to work and which comes preloaded with a bunch of interesting datasets. If you’re into a more academic approach, I highly recommend grabbing a copy of “Graph Databases” or “Neo4j in Action“.

Can you detail how can users interact with Neo4j?  What about developers, are there pre-built drivers or interfaces?

Calin: Neo4j packs a very nice Browser that enables users to quite easily query and visualize the graph. This comes with syntax highlighting and autocompletion for Cypher, the query language used by Neo4j. It also features a very handy way to interact with query execution plans.

Developers can “talk” with the database using a REST API or, better yet, the proprietary binary protocol called Bolt, which is already uniformly encapsulated by a number of official drivers, covering the most popular programming languages out there.

However, as I don’t want to spoil the fun, that’s all you’re getting out of me today. But do come and join us on the 16th of March. Please.

Dates: 18-20 May, 2018

In preparation for October 10th Bucharest Big Data Meetup here’s a short interview with Cristina Grosu, Product Manager @ Bigstep, company that has built a platform of integrated big data technologies such as Hadoop, NoSQL, MPP Databases as well as containers, bare-metal compute and storage, offered as a service. Cristina will be talking “Datalake architectures seen in production” during the meetup and we discussed a bit about the Data Lake concept and the possible solutions part of its architecture.

 
[Valentina] What is a data lake and how is it different from the traditional data warehouse present today in majority of bigger companies?

[Cristina] Think about the data lake as a huge data repository where data from very different and disparate sources can be stored in its raw, original format. One of the most important advantages of using this concept when storing data comes from the lack of a predefined schema, specific data structure, format or type which encourages the user to reinterpret the data and find new actionable insights.

Real-time or batch, structured or non-structured, files, sensor data, clickstream data, logs, etc can all be stored in the data lake. As users start a data lake project, they can explore more complex use cases since all data sources and formats can be blended together using modern, distributed technologies that can easily scale at a predictable cost and performance.

[Valentina] What (Big Data) solutions usually take part in a data lake architecture? Tell us a little bit regarding the generic architecture of such a solution.

[Cristina] The most common stack of technologies that I see in a data lake architecture is built around the Hadoop Distributed File System.

Since this is an open source project it allows customers build an open data lake that is extensible as well as well integrated with other applications that are running on-premises or in the cloud. Depending on the type of workload, the technology stack around the data lake might differ, but the Hadoop ecosystem is so versatile in terms of data processing (Apache Spark), data ingestion (Streamsets, Apache NiFi), data visualization(Apache Kibana, Superset), data exploration(Zeppelin, Jupyter) and streaming solutions(Storm, Flink, Spark Streaming) that can be fairly easy interconnected to create a full-fledged data pipeline.

What I have seen in production with a lot of our customers are “continuously evolving data lakes”, where users are building their solutions incrementally with a long-term vision in mind, expanding their data architecture when needed and leveraging new technologies when a new twist on their initial approach appears.

Other architectures for the data lake are based on NoSQL databases (Cassandra) or are using Hadoop as an extension of existing data warehouses in conjunction with other BI specific applications.

Regardless of the technology stack, it’s important to design a “forward-looking architecture” that enables data democratization, data lineage tracking, data audit capabilities, real-time data delivery and analytics, metadata management and innovation.

[Valentina] What are the main requirements of a company that would like to build a data lake architecture?  

[Cristina] The truth is that in big data projects, the main requirement is to have a driver in the company, someone with enough influence and experience that can sell the data lake concept in the company and can work together with the team to overcome all obstacles.

At the beginning of the project building the team’s big data skills might be necessary to make sure everyone is up to speed and create momentum within the organization by training, hiring or bringing in consultants.

Oftentimes, this leader will be challenged with implementing shifts in mindsets and behaviors regarding how the data is used, who’s involved in the analytics process and how everyone is interacting with data. As a result, new data collaboration pipelines will be developed to encourage cross-departmental communication.

To ensure the success of a data lake project and turn it into a strategic move, besides the right solution, you need a committed driver in your organization and the right people in place.

[Valentina] From the projects you’ve run so far, which do you think are the trickiest points to tackle when building a data lake solution?

[Cristina] Defining strategies for metadata management is quite difficult especially for organizations that have a large variety of data. Dumping everything in the data lake and trying to make sense of it 6 months into the project can become a real challenge if there is no data dictionary or data catalog that can be used to browse through data. I would say, data governance is an important thing to consider when developing the initial plan, even though organizations sometimes overlook it.

In addition, it’s important to have the right skills in the organization, understand how to extract value from data and of course figure out how to integrate the data lake into existing IT applications if that is important for the use case.

[Valentina] Any particular type of business that would benefit more from building a data lake?

[Cristina] To be honest the data lake is such a universal solution that it can be applied to all business verticals with relatively small differences in the design of the solution. The specificity of the data lake comes from every industry’s business processes and the business function it serves.

The greatest data lake adoption is observed among the usual suspects in retail, marketing, logistics, advertising where the volume and velocity of data grow day by day. Very interesting data lake projects are also seen in banking, energy, manufacturing, government departments and even Formula 1.

[Valentina] Will data lake concept change the way companies work with data in your opinion?    

[Cristina] Access to data will broaden the audience of the organization’s data and will drive the shift towards self-service analytics to power real-time decision making and automation.

Ideally, each person within a company would have access to a powerful dashboard that can be customized and adapted to run data exploration experiments to find new correlations. Once you find better ways to work with data, you can better understand how trends are going to affect your business, what processes can be optimized or you can simulate different scenarios and understand how your business is going to perform.

The data lake represents the start of this mindset change but everything will materialize once the adoption increases and technologies mature.

Interview by: Valentina Crisan

Two of the biggest struggles for a Big Data beginner are – most of the time – the practicality of data, meaning the understanding the applicability of the data in different businesses ( what kind of data and what kind of results those businesses obtain for it) and where to start from when thinking of a new idea.   

We are kicking off today a couple of posts that are meant to help understanding the applicability of data in business and the architectures that need to be built. Also, we will gather suggestions/recommendations on how to start when thinking of a new project.

Our first interview is with Tudor Lapusan, Big Data/ Hadoop Devops @ Telenav, who also runs Big Data Meetup in Cluj and is part of the team developing Zimilar – the startup that aims to recognize and match any piece of clothing & accessories out there.   

[Valentina] What kind of Big Data / Data are you analyzing at Telenav (at least in the area you’re working on)? Please mention type, volume, batch/real time – if possible.

[Tudor] I’m working for Telenav and playing around with big data technologies for more that 5 years. Telenav is a car navigation company and as you already may guess, our majority datasets are geospatial related (ex. anonymous lat, lon coordinates indicating where our clients are driving). Sharing the exact amount of data we have it’s confidential, but what I can disclose is that we store tens of terabytes of data and the amount of data is growing continuously.

Also my view is that the info on the size of one’s data on is relative because it depends on how it’s stored. You can store it in raw/serialized format or you can store it compressed. Also depends what schema/solution you choose to store the data, you can used HDFS or a NoSql database. All of these choices have a big impact in the size of your data.

In Telenav we started our Big Data project using batch processing. Five-six years ago was that moment when MapReduce and HDFS were the key components from Hadoop ecosystem and – at that momentum – with a weak adoption for small/medium companies. I believe that majority of early adopters were doing batch processing in that time. Currently although our main analysis processes are still batch processing, we are also working on prototypes for real time projects.

[Valentina] What kind of analytics are you performing on this data?

[Tudor]  Some of Telenav projects are using Open Street Map (OSM) as the map behind the car navigation. Because OSM is a free map, where anyone from this world can edit it, there are still some things to improve. We are analyzing our dataset to predict the real speed from the streets to improve the car driving experience. Also, based on our data, we improve the OSM by finding streets which are missing from the original map, we detect wrong direction information from the streets, one ways, double ways, turn-restriction which are mapped incorrect. These are one of the main analyses we are doing and all have the goal to improve the OSM and the car navigation experience.

We built even an web application to expose all of these results and to help OSM community : https://improveosm.org/.  Also, we are doing a lot of internal analyses to help us understand where the majority of our clients are coming from and their behavior.

[Valentina] Can you describe the architecture used (the solutions inside and their role)?

[Tudor] As I said before, in the beginning we started with classical architecture, MapReduce and HDFS (stored the data in HDFS and processed it with MapReduce). After that we needed also random access to our data, so after some research we choose HBase as a random data access solution.

Our current architecture is different though nowadays from the first one we built. Now we use Spark to analyze the data and because we don’t need random access on entire data, we don’t use HBase anymore. All the data is stored in HDFS as parquet files which is a perfect match for Spark and batch processing. We use ElasticSearch to import small/medium datasets when we need random access, mostly useful for debugging and internal products.

[Valentina] How did you choose the different components? What were the main requirements to be fulfilled?

[Tudor] I read and researched at lot at the times about what it’s going into Big Data world. Also I’m part of a  passionate team and this helps a lot in our decisions.

The world of Big Data is more stable now than in the past years. So, if you are a beginner in this space and you have a Big Data use-case for your project, after few days of research you will see that all tutorials go through one or two technologies to choose from. Of course, if you need to set up an entire pipeline, things get complicated and maybe you would need few advices from someone with more experience.

Others important aspects are the community around the frameworks, resources (tutorials, books, stack-overflow) and its maturity. Majority of the big data technologies are working well when you analyze few gigabytes. The real challenge is when you give them to analyze tens of terabytes of data or even more. Until now I didn’t find the perfect technology to work the same on small and big data. Depends on everyone how deep you want to search and understand that technology.

[Valentina] When building an architecture which are – in your experience – the main things to consider when choosing different solutions. For example, choosing different Hadoop distributions: like Cloudera or Map-R?

[Tudor] When we started to work with Hadoop, if I remember correctly, there was only Cloudera which offered solutions to deploy and monitor your cluster and it wasn’t free (as is it now). So we initially decided to manually deploy and manage our cluster. And all was fine while we had only nine servers in our cluster and only few services to manage. The main advantage of the free solution was that it forced us to really understand how each service is working, the big disadvantage was that it took time, efforts and sometimes a dedicated team to manage it.

Jumping to nowadays, to deploy a Hadoop cluster become a really easy process. You can do it even after a few days of research. All of these thanks to the big players which entered into the big data world like Amazon, Cloudera, Hortonworks, Map-R, Microsoft and many others. At Telenav we are using Cloudera as a solution to deploy and monitor our cluster and we are pretty happy with it.

If you plan to start working with big data technologies, I strongly recommend to use the solutions of one of the above mentioned companies. I can’t comment if one is better than the others, since I have more experience with Cloudera. But I have friends from other companies which are happy with Hortonworks or AWS.

[Valentina] What is the best use case that you see for HDFS currently?

[Tudor] HDFS is a distributed filesystem thought to store large volume of data. The default block size is, if I remember correctly, 128 MB in the newest versions and this size is customizable. These characteristics make HDFS the perfect solution to store your data at a minimum cost and in the end to do some big analysis on it (batch processing). At Telenav we store as parquet files all our historical data into HDFS and use Spark to process it.  The book “Hadoop, The definitive guide” (HDFS chapter) , includes one of the best explanations regarding HDFS characteristics. I strongly recommend our readers to read it if they want to find out more about HDFS and others use-cases for it.

[Valentina] You are as well the main driver of Cluj big data community, can you tell me how do you see the take off or adoption of Big Data solutions/architectures in Cluj and as well Romania level?

[Tudor] I remember when I initiated the Big Data community in Cluj, three years ago, that only 2-3 companies were then playing with Hadoop. I used to use Linkedin to look for other people from Cluj with ‘hadoop’ skills and found very few people at the time. Now, there are definitely more companies in Cluj working with Big Data solutions, some of them also supporting our community nowadays. From meetups and workshops, I met a lot of people which are working with big data technologies and even more people which are interested in this field. Plus more and more companies started to work on machine learning also. I don’t know the level for Romania, but I assume that it’s a good one. I heard about companies working with big data technologies in Bucharest, Timisoara, Iasi, Brasov.

Interview by: Valentina Crisan @bigdata.ro