The massive amounts of data collected over time that are difficult to analyze and handle using common database tools. Data are analyzed for marketing trends in business as well as in the fields of manufacturing, medicine and science. The types of data include business transactions, email messages, photos, surveillance videos, browsing behavior, activity logs and unstructured text from blogs and social media, as well as the huge amounts of data that are collected from sensors of all varieties (see
machine-generated data).
Big Data Storage and Big Data Analytics
The storage industry is continuously challenged as big data increases exponentially. While the physical storage can be enhanced with more multi-terabyte drive arrays, the software infrastructure must be flexible enough to quickly and economically accommodate ever greater volumes of transactions and queries. See
Hadoop,
Apache Spark and
cloud computing.
The analytical challenge is deriving meaningful information from data in petabyte and exabyte volumes. Big data analytics breaks down the datasets into smaller chunks for efficient processing and employs parallel computing to derive intelligence for effective decision-making (see
MPP). Increasingly, AI and machine learning are used to analyze big data, and AI has also transformed the capturing, structuring and storing of the data.
Big Data vs. Market Research
Analyzing big data may be able to forecast trends and people's habits without the kind of explanation one would derive from traditional market research. Big data patterns show the next probable behavior of a person or market without a logical explanation as to why. Big data was a hot buzzword in the 2010s. See
Petabyte Age.
The Biggest Big Data
Google has amassed the largest trove of data in the history of the world. Over the years, Google's Web crawlers, virtual assistants and search requests have captured more data about more people and more things than any other entity. Combined with its YouTube, Maps and Photos venues, Google Analytics, Google software applications and billions of Android smartphone users, Google "is" big data. See
Google Zeitgeist and
Google.