The massive amounts of data collected over time that are difficult to analyze and handle using common database management tools. The data are analyzed for marketing trends in business as well as in the fields of manufacturing, medicine and science. The types of data include business transactions, email messages, photos, surveillance videos, browsing behavior, activity logs and unstructured text from blogs and social media, as well as the huge amounts of data that can be collected from sensors of all varieties (see
machine-generated data).
Big Data Storage and Big Data Analytics
The storage industry is continuously challenged as big data increases exponentially. While the physical storage can be enhanced with more multi-terabyte drive arrays, the software infrastructure must be flexible enough to quickly and economically accommodate ever greater volumes of transactions and queries. See
Hadoop,
Spark and
cloud computing.
The analytical challenge is deriving meaningful information from data in petabyte and exabyte volumes. Big data analytics breaks down the data sets into smaller chunks for efficient processing and employs parallel computing to derive intelligence for effective decision-making (see
MPP). Increasingly, AI and machine learning are the tools used to analyze big data. See
AI.
Big Data vs. Market Research
Analyzing big data may be able to forecast trends and people's habits without the kind of explanation one would derive from traditional market research. Big data patterns show the next probable behavior of a person or market without a logical explanation as to why. Right, wrong or indifferent, big data was a hot buzzword in the 2010s. See
Petabyte Age.
The Biggest Big Data
Google has amassed the largest trove of data in the history of the world. Over the years, Google's Web crawlers, virtual assistants and search requests have captured more data about more people and more things than any other entity. Combined with its Web venues (YouTube, Maps, Photos, etc.), Google Analytics, software applications and billions of Android users, Google "is" big data. See
Google Zeitgeist and
Google.