What do you qualify as "Big Data"? Does it have more to do with the amount of data or the type of data?
Sort by:
Big data refers to large and complex data in varied formats generated through multiple sources and cannot be processed using standard application softwares.
Big Data is data for business. Several types of data like video, audio, images, logs etc.
For every transaction, end points call data generated and all of this kind of data becomes big data, complexity and size both make it big.
To me, and most of my customers big data is data-points of their business. Often a representation of the physical world (images, video, sensor, logs, telemetrics, and seismic data), the virtual world, or the metaverse. It dosn’t matter if it’s digitalization of human tissue, a city or building, simulation of a software or making of the next Starwars series… but as soon as you get help from machines, sensors, or devices to generate data-points and you also get help from computers and software to innovate from the data, then you have entered the world of big-data.
It has to do with both. If you have a small but complex set then it is still easier to work with than a large set that is complex as it takes more to understand the outcomes you can get from it. I always think of big data as large sets of unstructured data that you are trying to derive decisions from.
With the ability to superpose LLM on top of the data lyfecycle, maybe big data as a description has lost a bit of importance, given that any data format can be pre-processed by these multi-modal models to assist in discovering, cataloging, labeling and analyzing any data.