{"id":559,"date":"2021-08-12T13:40:41","date_gmt":"2021-08-12T13:40:41","guid":{"rendered":"https:\/\/sites.rutgers.edu\/joann-ordille\/?p=559"},"modified":"2021-08-27T17:35:05","modified_gmt":"2021-08-27T17:35:05","slug":"where-is-all-the-data","status":"publish","type":"post","link":"https:\/\/sites.rutgers.edu\/joann-ordille\/where-is-all-the-data\/","title":{"rendered":"Where is all the data?"},"content":{"rendered":"

As you learn data science, and even when you\u2019re a long-time data scientist, you will eventually ask: \u201cHow can I acquire the data I need?\u201d\u00a0 Sometimes, you need to obtain the data within your enterprise. \u00a0Other times, you need data for analyzing broader trends, e.g., how fires in the West are impacting the supply chain, or whether there will be enough trained people to build a staff in a remote location.<\/p>\n

So, where is all the data?\u00a0 A good first step is querying Google Dataset Search<\/a>.\u00a0 Early in its development Dataset Search provided results that hid the real gems of open data in a haystack of for-fee datasets. \u00a0No more!\u00a0 Choose the free dataset filter to benefit from the massive number of open datasets on the Internet.<\/p>\n

Perhaps you are interested in acceptance rates at business schools?\u00a0 You can enter the following query:<\/p>\n

\"Google
Google Dataset Search Query<\/strong><\/figcaption><\/figure>\n

The results list several datasets at a popular dataset sharing site, Kaggle.<\/p>\n

\"Three
Partial Results from Google Dataset Search Query<\/strong><\/figcaption><\/figure>\n

But these results are for all universities, not just business schools.\u00a0 Another query for business schools will produce datasets with lists of business schools.\u00a0 The information you want is buried in these two datasets, and you are now moving on\u00a0 in the data science pipeline to managing and wrangling your data.<\/p>\n

\"Diagram
The Data Science Pipeline<\/strong><\/figcaption><\/figure>\n

You need to combine the datasets together to meet your need.\u00a0 That might not be straightforward.\u00a0 The names of the universities may be spelled differently in the two datasets.\u00a0 I\u2019ll address the next steps in preparing this data in subsequent posts.\u00a0 Note that this process is very similar to the Extract, Transform and Load (ETL) process for building a data warehouse.\u00a0 In ETL, data is extracted from a company\u2019s operational databases, and then restructured and loaded into a data warehouse for analytical processing.<\/p>\n

Your search for datasets has begun!\u00a0 You may discover data sharing sites you wish to join to take advantage of their instructional and data compilation resources.\u00a0 In addition, Kaggle runs data science and machine learning competitions to help you sharpen your skills.\u00a0 Many of the sites provide space for you to share the results of your analysis and other datasets you\u2019ve collected.<\/p>\n

Learn more about Google Data Set Search in the article Discovering millions of datasets on the web<\/em><\/a>.\u00a0 Some other very powerful data source sites are:<\/p>\n