Big Data — What it is and Why it matters most in today’s era??

3 min readSep 16, 2020

In today’s world, the most important thing for any industry to survive is the data that they collect from their users. Any of the big companies like Google, Facebook, Amazon, and others collect lots of data in a single day.

What is Big Data??

As the term describe itself, Big Data is defined as data that is huge in size. Big Data is not a technology name, it is actually a set of problems that we face in the industry to store a large amount of data.

Big data is a term used to describe a collection of data that is huge in size and yet growing exponentially with time. Social Networking sites like Facebook reveals in one of his surveys, that they collect more than 4+ petabytes of data in a single day. Managing and securing this much data is the most difficult task for any company. And the success of Facebook is because of the data that they collect and utilize efficiently, they still increase day by day.

We have multiple examples like Facebook by which you understand the importance of data on this earth. Big Data helps organizations to create new growth opportunities and entirely new categories of companies that can combine and analyze industry data. These companies have ample information about the products and services, buyers and suppliers, consumer preferences that can be captured and analyzed.

Let’s see some stats…

Some BigData Stats:

If we talk about Google, we submit 40,000 search queries per second. That amounts to 1.2 trillion searches yearly!
Each minute, 300 new hours of video show up on YouTube. That’s why there are more than 1 billion gigabytes (1 exabyte) of data on its servers!
Every minute, users send 31 million messages and view 2.7 million videos on FaceBook.
The amount of data created each year is growing faster than ever before. By 2020, every human on the planet will be creating 1.7 megabytes of information… each second!
In only a year, the accumulated world data will grow to 44 zettabytes (that’s 44 trillion gigabytes)! For comparison, in the starting of 2020, it’s about 4.4 zettabytes. This is really hilarious…
In last two years, nearly 90% of all data has been created. That’s why Companies that use big data analytics are already gaining momentum on their competition.
We are using BigData in almost each of the industries like Medicine, Media, Banking, IOT devices, etc.

Let’s talk about some more concepts of Big Data.

What are the problems which companies facing in storing Big data?

We have multiple problems, but I’m here discussing two important V’s factors that lead us to the Big Data world.

1. Volume :

Let’s think like you have 50 TB storage device and you need to store 100 TB. This much data we can store in one single hard drive but this is going too much expensive for us. And this leads to some other problems too which we discuss next.

2. Velocity :

When we use a large size hard disk, it leads to Input/Output problems. Reading or Writing in HardDisk takes a lot of time in comparison to RAM. This is because reading data from the RAM is much faster than reading data from the hard drive. Running programs from the RAM of the computer allows them to function without any lag time.

Solution:

Distributed Storage is the technology/concept which we use in storing big data. We have some tools like Hadoop, Spark, BigQuery, etc.

A distributed storage system is an infrastructure that can split data across multiple physical servers, and often across more than one data center. It typically takes the form of a cluster of storage units, with a mechanism for data synchronization and coordination between cluster nodes. By this way, you can effectively boost I/O speed too.