About big data lakes & architectures

In preparation for October 10th Bucharest Big Data Meetup here’s a short interview with Cristina Grosu, Product Manager @ Bigstep, company that has built a platform of integrated big data technologies such as Hadoop, NoSQL, MPP Databases as well as containers, bare-metal compute and storage, offered as a service. Cristina will be talking “Datalake architectures seen in production” during the meetup and we discussed a bit about the Data Lake concept and the possible solutions part of its architecture.

 
[Valentina] What is a data lake and how is it different from the traditional data warehouse present today in majority of bigger companies?

[Cristina] Think about the data lake as a huge data repository where data from very different and disparate sources can be stored in its raw, original format. One of the most important advantages of using this concept when storing data comes from the lack of a predefined schema, specific data structure, format or type which encourages the user to reinterpret the data and find new actionable insights.

Real-time or batch, structured or non-structured, files, sensor data, clickstream data, logs, etc can all be stored in the data lake. As users start a data lake project, they can explore more complex use cases since all data sources and formats can be blended together using modern, distributed technologies that can easily scale at a predictable cost and performance.

[Valentina] What (Big Data) solutions usually take part in a data lake architecture? Tell us a little bit regarding the generic architecture of such a solution.

[Cristina] The most common stack of technologies that I see in a data lake architecture is built around the Hadoop Distributed File System.

Since this is an open source project it allows customers build an open data lake that is extensible as well as well integrated with other applications that are running on-premises or in the cloud. Depending on the type of workload, the technology stack around the data lake might differ, but the Hadoop ecosystem is so versatile in terms of data processing (Apache Spark), data ingestion (Streamsets, Apache NiFi), data visualization(Apache Kibana, Superset), data exploration(Zeppelin, Jupyter) and streaming solutions(Storm, Flink, Spark Streaming) that can be fairly easy interconnected to create a full-fledged data pipeline.

What I have seen in production with a lot of our customers are “continuously evolving data lakes”, where users are building their solutions incrementally with a long-term vision in mind, expanding their data architecture when needed and leveraging new technologies when a new twist on their initial approach appears.

Other architectures for the data lake are based on NoSQL databases (Cassandra) or are using Hadoop as an extension of existing data warehouses in conjunction with other BI specific applications.

Regardless of the technology stack, it’s important to design a “forward-looking architecture” that enables data democratization, data lineage tracking, data audit capabilities, real-time data delivery and analytics, metadata management and innovation.

[Valentina] What are the main requirements of a company that would like to build a data lake architecture?  

[Cristina] The truth is that in big data projects, the main requirement is to have a driver in the company, someone with enough influence and experience that can sell the data lake concept in the company and can work together with the team to overcome all obstacles.

At the beginning of the project building the team’s big data skills might be necessary to make sure everyone is up to speed and create momentum within the organization by training, hiring or bringing in consultants.

Oftentimes, this leader will be challenged with implementing shifts in mindsets and behaviors regarding how the data is used, who’s involved in the analytics process and how everyone is interacting with data. As a result, new data collaboration pipelines will be developed to encourage cross-departmental communication.

To ensure the success of a data lake project and turn it into a strategic move, besides the right solution, you need a committed driver in your organization and the right people in place.

[Valentina] From the projects you’ve run so far, which do you think are the trickiest points to tackle when building a data lake solution?

[Cristina] Defining strategies for metadata management is quite difficult especially for organizations that have a large variety of data. Dumping everything in the data lake and trying to make sense of it 6 months into the project can become a real challenge if there is no data dictionary or data catalog that can be used to browse through data. I would say, data governance is an important thing to consider when developing the initial plan, even though organizations sometimes overlook it.

In addition, it’s important to have the right skills in the organization, understand how to extract value from data and of course figure out how to integrate the data lake into existing IT applications if that is important for the use case.

[Valentina] Any particular type of business that would benefit more from building a data lake?

[Cristina] To be honest the data lake is such a universal solution that it can be applied to all business verticals with relatively small differences in the design of the solution. The specificity of the data lake comes from every industry’s business processes and the business function it serves.

The greatest data lake adoption is observed among the usual suspects in retail, marketing, logistics, advertising where the volume and velocity of data grow day by day. Very interesting data lake projects are also seen in banking, energy, manufacturing, government departments and even Formula 1.

[Valentina] Will data lake concept change the way companies work with data in your opinion?    

[Cristina] Access to data will broaden the audience of the organization’s data and will drive the shift towards self-service analytics to power real-time decision making and automation.

Ideally, each person within a company would have access to a powerful dashboard that can be customized and adapted to run data exploration experiments to find new correlations. Once you find better ways to work with data, you can better understand how trends are going to affect your business, what processes can be optimized or you can simulate different scenarios and understand how your business is going to perform.

The data lake represents the start of this mindset change but everything will materialize once the adoption increases and technologies mature.

Interview by: Valentina Crisan

Big Data Projects and Architectures (Part 1)

Two of the biggest struggles for a Big Data beginner are – most of the time – the practicality of data, meaning the understanding the applicability of the data in different businesses ( what kind of data and what kind of results those businesses obtain for it) and where to start from when thinking of a new idea.   

We are kicking off today a couple of posts that are meant to help understanding the applicability of data in business and the architectures that need to be built. Also, we will gather suggestions/recommendations on how to start when thinking of a new project.

Our first interview is with Tudor Lapusan, Big Data/ Hadoop Devops @ Telenav, who also runs Big Data Meetup in Cluj and is part of the team developing Zimilar – the startup that aims to recognize and match any piece of clothing & accessories out there.   

[Valentina] What kind of Big Data / Data are you analyzing at Telenav (at least in the area you’re working on)? Please mention type, volume, batch/real time – if possible.

[Tudor] I’m working for Telenav and playing around with big data technologies for more that 5 years. Telenav is a car navigation company and as you already may guess, our majority datasets are geospatial related (ex. anonymous lat, lon coordinates indicating where our clients are driving). Sharing the exact amount of data we have it’s confidential, but what I can disclose is that we store tens of terabytes of data and the amount of data is growing continuously.

Also my view is that the info on the size of one’s data on is relative because it depends on how it’s stored. You can store it in raw/serialized format or you can store it compressed. Also depends what schema/solution you choose to store the data, you can used HDFS or a NoSql database. All of these choices have a big impact in the size of your data.

In Telenav we started our Big Data project using batch processing. Five-six years ago was that moment when MapReduce and HDFS were the key components from Hadoop ecosystem and – at that momentum – with a weak adoption for small/medium companies. I believe that majority of early adopters were doing batch processing in that time. Currently although our main analysis processes are still batch processing, we are also working on prototypes for real time projects.

[Valentina] What kind of analytics are you performing on this data?

[Tudor]  Some of Telenav projects are using Open Street Map (OSM) as the map behind the car navigation. Because OSM is a free map, where anyone from this world can edit it, there are still some things to improve. We are analyzing our dataset to predict the real speed from the streets to improve the car driving experience. Also, based on our data, we improve the OSM by finding streets which are missing from the original map, we detect wrong direction information from the streets, one ways, double ways, turn-restriction which are mapped incorrect. These are one of the main analyses we are doing and all have the goal to improve the OSM and the car navigation experience.

We built even an web application to expose all of these results and to help OSM community : https://improveosm.org/.  Also, we are doing a lot of internal analyses to help us understand where the majority of our clients are coming from and their behavior.

[Valentina] Can you describe the architecture used (the solutions inside and their role)?

[Tudor] As I said before, in the beginning we started with classical architecture, MapReduce and HDFS (stored the data in HDFS and processed it with MapReduce). After that we needed also random access to our data, so after some research we choose HBase as a random data access solution.

Our current architecture is different though nowadays from the first one we built. Now we use Spark to analyze the data and because we don’t need random access on entire data, we don’t use HBase anymore. All the data is stored in HDFS as parquet files which is a perfect match for Spark and batch processing. We use ElasticSearch to import small/medium datasets when we need random access, mostly useful for debugging and internal products.

[Valentina] How did you choose the different components? What were the main requirements to be fulfilled?

[Tudor] I read and researched at lot at the times about what it’s going into Big Data world. Also I’m part of a  passionate team and this helps a lot in our decisions.

The world of Big Data is more stable now than in the past years. So, if you are a beginner in this space and you have a Big Data use-case for your project, after few days of research you will see that all tutorials go through one or two technologies to choose from. Of course, if you need to set up an entire pipeline, things get complicated and maybe you would need few advices from someone with more experience.

Others important aspects are the community around the frameworks, resources (tutorials, books, stack-overflow) and its maturity. Majority of the big data technologies are working well when you analyze few gigabytes. The real challenge is when you give them to analyze tens of terabytes of data or even more. Until now I didn’t find the perfect technology to work the same on small and big data. Depends on everyone how deep you want to search and understand that technology.

[Valentina] When building an architecture which are – in your experience – the main things to consider when choosing different solutions. For example, choosing different Hadoop distributions: like Cloudera or Map-R?

[Tudor] When we started to work with Hadoop, if I remember correctly, there was only Cloudera which offered solutions to deploy and monitor your cluster and it wasn’t free (as is it now). So we initially decided to manually deploy and manage our cluster. And all was fine while we had only nine servers in our cluster and only few services to manage. The main advantage of the free solution was that it forced us to really understand how each service is working, the big disadvantage was that it took time, efforts and sometimes a dedicated team to manage it.

Jumping to nowadays, to deploy a Hadoop cluster become a really easy process. You can do it even after a few days of research. All of these thanks to the big players which entered into the big data world like Amazon, Cloudera, Hortonworks, Map-R, Microsoft and many others. At Telenav we are using Cloudera as a solution to deploy and monitor our cluster and we are pretty happy with it.

If you plan to start working with big data technologies, I strongly recommend to use the solutions of one of the above mentioned companies. I can’t comment if one is better than the others, since I have more experience with Cloudera. But I have friends from other companies which are happy with Hortonworks or AWS.

[Valentina] What is the best use case that you see for HDFS currently?

[Tudor] HDFS is a distributed filesystem thought to store large volume of data. The default block size is, if I remember correctly, 128 MB in the newest versions and this size is customizable. These characteristics make HDFS the perfect solution to store your data at a minimum cost and in the end to do some big analysis on it (batch processing). At Telenav we store as parquet files all our historical data into HDFS and use Spark to process it.  The book “Hadoop, The definitive guide” (HDFS chapter) , includes one of the best explanations regarding HDFS characteristics. I strongly recommend our readers to read it if they want to find out more about HDFS and others use-cases for it.

[Valentina] You are as well the main driver of Cluj big data community, can you tell me how do you see the take off or adoption of Big Data solutions/architectures in Cluj and as well Romania level?

[Tudor] I remember when I initiated the Big Data community in Cluj, three years ago, that only 2-3 companies were then playing with Hadoop. I used to use Linkedin to look for other people from Cluj with ‘hadoop’ skills and found very few people at the time. Now, there are definitely more companies in Cluj working with Big Data solutions, some of them also supporting our community nowadays. From meetups and workshops, I met a lot of people which are working with big data technologies and even more people which are interested in this field. Plus more and more companies started to work on machine learning also. I don’t know the level for Romania, but I assume that it’s a good one. I heard about companies working with big data technologies in Bucharest, Timisoara, Iasi, Brasov.

Interview by: Valentina Crisan @bigdata.ro

 

 

 

Distributed search with Apache Cassandra and/or Apache Solr?

One is a noSQL solution, the other one is primarily a search engine, each also adding up search or scaling functions thus increasing their potential use cases. Talking about the integration of Cassandra and Solr and how is best to perform a distributed search, we (Valentina Crisan & Radu Gheorghe) decided that this might be a good topic for one of Bucharest Big Data meetup presentations thus on June 7th we will debate the use cases in which these 2 powerful solutions might be best on their own or integrated.        

Below interview article is the base of the presentation and discussion in the upcoming June 7th Bucharest Big Data meetup: Distributed search with Apache Cassandra and/or Apache Solr.  

Best use cases for Cassandra + Solr/Lucene or Solr + Cassandra/noSQL?

[Valentina] From Cassandra perspective is pretty simple, any use case that needs as well – besides its main use cases  – queries with full-text search capabilities ( facets, highlights, geo-location…), also requiring one interface (CQL) for launching all queries, then Solr integration or any Lucene index will be the solution. The main data store would be Cassandra and the indexes would be handled by Lucene/Solr engine. Please also see below answer about Cassandra search capabilities.     

[Radu] Solr is great for either full text search (think books, blog posts) or structured search (products, logs, events) and for doing analytics on top of those search results (facets on an online shop, log/metric analytics, etc). Backups aside, you would benefit from a dependable datastore like Cassandra to be the source of truth. Also, some use-cases have big field values that don’t need to be shown with results (e.g. Email attachments), so they might as well be stored in Cassandra and only pulled when the user selects that result – leaving Solr with only the index to deal with for that field.

What are the optimal use cases for Apache Cassandra?

[Valentina]  When your requirement is to have very heavy write, distributed and highly scalable system and you might want as well to have quite responsive reporting system on top of that stored data, then Cassandra is your answer. Consider use case of Web analytics where log data is stored for each request and you want to built analytical platform around it to count hits by hour, by browser, by IP, etc in real time manner. Thumb rule of performing real time analytics is that you should have your data already calculated and should persist in the database. If you know the reports you want to show in real time, you can have your schema defined accordingly and generate your data at real time. Cassandra is not an OLAP system but can handle simple queries with a pre-defined data model and pre-aggregation of data, supporting integration with Spark for more complex analytics: joins, groupby,window functions.

Most of Cassandra uses cases are around storing data from next domains: Product Catalog/Playlist, Recommendation/Personalization Engine, Sensor Data/Internet of Things, Messaging, Fraud Detection.

Which are the Cassandra search capabilities?

[Valentina]  Traditionally Cassandra was not designed to support search cases (it can only do matches on partition key), but since v3.4 there has been some improvement, through the introduction of SASI feature (see below).

In general the standard option in Cassandra when dealing with search type of queries is to use an index, Cassandra supporting a pluggable interface for defining indexes, and thus there are multiple implementations available:

  • The native solution: SASI ( SSTable-Attached Secondary Index), implemented over the native secondary indexes in Cassandra, this feature is not only an extension of the native secondary indexes in Cassandra, but an improved one in the sense that shortens access time to data versus the standard indexes implementation. The are still some things to consider with SASI when compared with a search engine ( see Ref 1):
    • It is not possible to ask for total ordering of the result (even when LIMIT clause is used), SASI returns result in token range order;
    • SASI allows full text search, but there is no scoring applied to matched terms;
    • being a secondary index, SASI requires 2 passes on disk to fetch data: first to get the index data and another read in order to fetch the rest of the data;
    • native secondary indexes in Cassandra are locally stored in each node, thus all searches will need to hit all cluster nodes in order to retrieve the final data – even if sometimes this means only a node will have the relevant information;

  Through SASI Apache Cassandra supports:

– search queries including more than 1 column (multiple columns indexes)

– for data types (text, varchar & ascii) supports queries on:

– prefix using the LIKE ‘prefix%’ syntax

– suffix using the LIKE ‘%suffix’ syntax

– substring using the LIKE ‘%substring%’ syntax

exact match using equality (=)

– For other data types (int, date, uuid …) supports queries on:

– equality (=)

              – range ( <, ≤, >, ≥ )

  • Stratio’s Cassandra Lucene Index:  is an open plugin for Apache Cassandra that extends its index functionality to provide near real time search such as ElasticSearch or Solr, including full text search capabilities and free multivariable, geospatial search. It is achieved through an Apache Lucene based implementation of Cassandra secondary indexes, where each node of the cluster indexes its own data. Through Cassandra Lucene index next queries are supported (see Ref 2):
    • Full text search (language-aware analysis, wildcard, fuzzy, regexp)
    • Boolean search (and, or, not)
    • Sorting by relevance, column value, and distance
    • Geospatial indexing (points, lines, polygons and their multiparts)
    • Geospatial transformations (bounding box, buffer, centroid, convex hull, union, difference, intersection)
    • Geospatial operations (intersects, contains, is within)
    • Bitemporal search (valid and transaction time durations)
  • DSE ( Datastax Enterprise Search  – commercial edition): it’s a Datastax proprietary implementation ( you can get access to it only through DSE ) which uses Apache Solr and Lucene on top of Cassandra. It maintains search cores co-located with each Cassandra node, updating them as part of Cassandra’s write path so that data is indexed in real-time. In addition to supporting the text and geospatial search features, DSE Search also supports advanced capabilities like search faceting.
  • Other solutions: 
    • Of course you can integrate separately Cassandra and Solr (application layer integration) by using an analytical tool that joins the data from both systems and a solution like Kafka in order to synchronize the data. This has nothing to do with Cassandra search capabilities, just pointing the option. Spark also could be use as an aggregation and analytics layer for the data.       

Why would anyone, in your opinion, use Cassandra for search instead of just using Solr?

[Valentina]  The best implementation always depends on the use case and its history. If the primary use case is building an OLTP system, storing the data reliably, writing it really fast in distributed environments with simple queries that usually rely on the partition key and just sporadically support more complex queries: text search, joins, groupby, moving average, then having Cassandra as a base system and use it’s SASI or Lucene index or even DSE Solr implementation plus Spark for full analytics might be a solution. Depending on the search complexity you can decide on the 3 alternatives provided by Cassandra.

If your use case is a heavy analytics one that includes as well evolved search capabilities then having an analytics tool that combines the fast retrieval of data in Cassandra (queries by partition) and the indexes in Solr might be a solution.    

So, depends which is your primary use case for the solution you’re trying to build. For sure comparing Cassandra and Solr as noSQL or search engines is not optimal since Cassandra was not meant to be search engine and Solr a noSQL system ( although since SolrCloud there have been use cases for Solr as a data storage system) thus your main requirements should be the ones that drive the solution design not just a bare comparison between these 2.

How can Solr be used as a data storage system (besides a search engine)? What features are available for data redundancy and consistency?

[Radu] Solr supports CRUD operations, though updates tend to be expensive (see below) and a get by ID is in fact a search by ID: a very cheap search, but usually slower than getting documents by primary key from a more traditional storage system – especially when you want N documents back.

For redundancy, replicas can be created on demand and if leader goes down, a replica is automatically promoted. Searches are round-robined between replicas, and because replication is based on the transaction log (on SolrCloud at least), we’re talking about the same load no matter if a shard is a leader or a replica. Lastly, active-passive cross-DC replication is supported.

As for consistency, Zookeeper holds the cluster state, and writes are acknowledged when all replicas write to transaction log: the leader writes a new document to its transaction log and replicates it to all the replicas, which index the same document in parallel.

What are the preferred types of input data for Solr use cases (e.g. documents)? Which type of data is the most optimal for its functionality?  

[Radu]  Structured documents (e.g. products) and/or free text (books, events) are ideal use-cases. Solr doesn’t work well with huge documents (GB) or huge number of fields (tens of thousands).

Some solutions owners do have a separate storage system connected to Solr. How the integration looks in this case (all data is replicated twice or only the indexes reside in Solr)?

[Radu]  Usually Solr indexes data (for searching) and some fields have docValues (for sorting and facets). Most also store the original value for fields where docValues aren’t enabled (so one could get the whole document without leaving Solr). Storing a field is also required for highlighting. If you need all these features, you need to replicate the data twice (the worst case is times:separate storage, docValues, stored fields. Plus the inverted index). If you don’t, you can skip some of those, and rely on the separate storage to get the original data. That said, fetching data from a different storage is often slower because it requires another [network] call.

How does data update look like in Solr? Is the data updated or rather added up (with different versions)?

[Radu]  There are three main options:

  • Index a new document over the old one. Versioning can be used here for concurrency control.
  • Update a value of a document. This will trigger Solr to fetch the document internally, apply the change and index the resulting document over the old one. You can still use versions here, either assigned by Solr or provided by you (e.g. a timestamp).
  • For fields that only have docValues (not indexed, not stored), an in-place update of that value is possible.

When might be ok to use Solr as an analytics tool?

[Radu]  When you filter (search) on data before analyzing, because that’s where Solr shines. For example, live dashboards on logs, events, metric data. Or faceting for product search (e.g. Amazon).

What are Solr’s capabilities when it comes to full text search?

[Radu] I can’t be exhaustive here, but let me go over the main categories:

  • Fast retrieval of top N documents matching a search. It can also go over all matches of a search if a cursor is used.
  • Capable and pluggable text analysis. For example, language detection, stemming, synonyms, ngrams and so on. This is all meant to break text into tokens so that “analysis” in your query matches “analyze” in your document, for example.
  • Sorting by a field works, of course, but Solr is also good at ranking results by a number of criteria. For example, per-field boosting (including boosting more exact matches over more lenient ones, like those that are stemmed), TF/IDF or BM25, value of a field, potentially with a multiplier. Multiple criteria can be combined as well, so you can really tweak your relevancy scores.
  • Highlighting matches. It doesn’t have to go over the text again, which is useful for large fields like books.

 

This discussion will be be continued on June 7th, Bucharest Big Data Meetup.  See you there. 

Ref 1: http://www.doanduyhai.com/blog/?p=2058#sasi_perf_benchmarks

Ref 2: https://github.com/Stratio/cassandra-lucene-index 

Big Data in Tg Mures

Looks like Big Data topic is getting more and more attention and more cities are starting events that include this topic, thus we decided we will have a new bigdata.ro page where we will present these events. The first event we will feature will take place in Tg Mures on June 2nd, Kodok Marton will talk about “Complex Realtime Event Analytics Using BigQuery”.

Join the event if you are in the Tg Mures area, you can see the event details here.

Looking forward to see a Big Data community forming in Tg Mures as well.

Hadoop MapReduce and Spark training

With the occasion of completing the Big Data Romanian Tour in Bucharest we planned a light intro and hands-on session for MapReduce and Spark. Together with the trainers – Tudor Lapusan and Andrei Avramescu – we are planning a dynamic and interactive session that is going to go from theory to practice. We have only 4 hours so we will need to be pretty fast – thus we will send in advance some materials to get familiar to. But, the good thing is that at the end of this session you should be able to start your own Hadoop or Spark cluster, pump data into it and start the analysis. Tudor and Andrei will also talk about the differences between Spark and Hadoop MapReduce. We will have a limited number of places available for this training – since we will need to work with everybody on the hands on part of the training – thus, if you are interested, don’t wait up. Register here for the training and tell us your expectations so that we can prepare better in advance. See you all soon & if you have any questions contact us on contact at bigdata.ro or comment this post.

Romanian Big Data Roadshow: Cluj, Timisoara, Bucharest

colaj big data meetups

3 Romanian communities, more then 700 members, incredible energy and subjects, these are just a few of the ingredients of the Romanian Big Data Roadshow that brings together speakers and members of three dynamic communities that have in common an extremely high interest in Big Data & Data Science.

A bit of a background for the event 

I met (virtually) Alex Sisu and Tudor Lapusan, the organizers for Timisoara and Cluj Big Data meetups, early 2014 and was not long after that we decided we need to organize a Big Data event, given:

– the amount of knowledge but also of interest that exists in Romania on this particular area

– but as well our similar thoughts on keeping such events technology/companies independent as much as possible, in order keep the content genuine and not transform presentations in sales pitches.

To get started and see how our collaboration goes first we started with this roadshow. Follow us on this page (or on the respective pages of the local communities for the detailed agendas, presentations, pictures) and see how the roadshow looks like, but as well stay tuned for what we’re planning next.