2.3 Web Scraping and Social Media Data Collection

Regular Expressions in Python

Regular expressions, also known as regex, are a set of symbols used to define a search pattern in text data. In Python, these expressions are supported by the re module (function), and their syntax is similar to other programming languages. The use of regular expressions offers researchers a robust method for identifying and manipulating patterns in text. With this powerful tool, specific words, characters, or patterns of characters can be searched and matched. Typical applications include data parsing, input validation, and extracting targeted information from larger text sources. Common use cases in Python involve recognizing various types of data, such as dates, email addresses, phone numbers, and URLs, within extensive text files. Moreover, regular expressions are valuable for tasks like data cleaning and text processing. Despite their versatility, regular expressions can be elaborate, allowing for advanced search patterns utilizing meta-characters like *, ?, and +. However, working with these expressions can present challenges, as they require a thorough understanding and careful debugging to ensure successful implementation.

    ## import the regular expression module
    import re
    # create a string with story problem
    story = "Samantha is a sixth-grade student who uses the popular coding language Python to collect and analyze weather data for science project. She creates a program that collects data from an online weather API and stores it in a CSV file for 10 days. With the help of her teacher, she uses Python to visualize the data and discovers patterns in temperature, humidity, and precipitation."
    # create a regex pattern to match a selected word
    pattern = "Python"
    # use the "findall" to search for matching patterns in the data string
    words = re.findall(pattern, story)
    
    # print the number of repeated python
    print ("The word 'python' is repeated", len(words), "times in the story.")
    
    # print the matched patterns
    print(words) # output: ['Python']
  

The word 'python' is repeated 2 times in the story.
['Python', 'Python']

Parsing and Extracting Data

Splitting and slicing are two methods used to manipulate text strings in programming. Splitting a string means dividing a text string into smaller parts or substrings based on a specified separator. The separator can be a character, string, or regular expression. This can be useful for separating words, phrases, or data values within a larger string. For example, the string "Data Science" can be split into two substrings "Data" and "Science" by using a space as the separator.

Slicing a string refers to extracting a portion or section of a string based on a specified range of indices. An index refers to the position of a character in a string, starting from 0 for the first character. The range specifies the start and end indices for the slice, and the resulting substring includes all characters within that range. For example, the string "Data Science" can be sliced to extract "Data" by specifying the range from index 0 to 4, which includes the first four characters. Slicing can also be used to manipulate strings by replacing, deleting, or inserting new content into specific positions within the string.

Parsing and extracting data involves the analysis of a given dataset or string to extract specific pieces of information. This is accomplished using various techniques and functions, such as splitting and slicing strings, which allow for the structured retrieval of data. This process is particularly valuable when working with large and complex datasets, as it provides a more efficient means of locating desired data compared to traditional search methods. Note that parsing and extracting data differs from the use of regular expressions, as regular expressions serve as a specialized tool for pattern matching and text manipulation. In contrast, parsing and data extraction offers a comprehensive approach to identifying and extracting specific data within a dataset.

Parsing and extracting data using Python involves using the programming language to locate and extract specific information from a given text. This is achieved by utilizing the re library, which enables the use of regular expressions to identify and retrieve data based on defined patterns. This process can be demonstrated through an example of extracting data related to a person purchasing an iPhone at an Apple store.

The code in the following Python feature box uses regular expressions (regex) to match and extract specific data from a string. The string is a paragraph containing information about a person purchasing a new phone from the Apple store. The objective is to extract the product name, model, and price of the phone. First, the code starts by importing the necessary library for using regular expressions. Then, the string data is defined as a variable. Next, regex is used to search for specific patterns in the string. The first pattern searches for the words "product: " and captures anything that comes after it until it reaches a comma. The result is then stored in a variable named "product". Similarly, the second pattern looks for the words "model: " and captures anything that comes after it until it reaches a comma. The result is saved in a variable named "model". Finally, the third pattern searches for the words "price: " and captures any sequence of numbers or symbols that follows it until the end of the string. The result is saved in a variable named "price". After all the data is extracted, it is printed out to the screen, using concatenation to add appropriate labels before each variable.

This application of Python code demonstrates the effective use of regex and the re library to parse and extract specific data from a given data. By using this method, the desired information can be easily located and retrieved for further analysis or use.

    ## import necessary library
    import re
    # Define data to be parsed
    data = "Samantha went to the Apple store to purchase a new phone. She was specifically looking for the latest and most expensive model available. As she looked at the different options, she came across the product: iPhone 12, the product name caught her attention, as it was the newest version on the market. She then noticed the model: A2172, which confirmed that this was indeed the latest and most expensive model she was looking for. The price made her hesitate for a moment, but she decided that it was worth it price: $799. She purchased the iPhone 12 and was excited to show off her new phone to her friends."
    
    # Use regex to match and extract data based on specific pattern
    product = re.search(r"product: (.+?),", data).group(1)
    model = re.search(r"model: (.+?),", data).group(1)
    price = re.search(r"price: (.+?.+?.+?.+?)", data).group(1)
    
    # Print the extracted data
    print("product: " + product)
    print("model: " + model)
    print("price: " + price)
  

product: iPhone 12 model: A2172 price: $799

Processing and Storing Data

Once the data is collected, it should be processed and stored in a suitable format for further analysis. This is where data processing and storage come into play. Data processing manages the raw data first by cleaning it through the removal of irrelevant information and then by transforming it into a structured format. The cleaning process includes identifying and correcting any errors, inconsistencies, and missing values in a dataset and is essential for ensuring that the data is accurate, reliable, and usable for analysis or other purposes. Python is often utilized for data processing due to its flexibility and ease of use, and it offers a wide range of tools and libraries specifically designed for data processing. Once the data is processed, it needs to be stored for future use. (We will cover data storage in Data Cleaning and Preprocessing.) Python has several libraries that allow for efficient storage and manipulation of data in the form of DataFrames.

One method of storing data using Python is using the pandas library to create a DataFrame and then using the to_csv() function to save the DataFrame as a CSV (comma-separated values) file. This file can then be easily opened and accessed for future analysis or visualization. For example, the code in the following Python sidebar is a Python script that creates a dictionary with data about the presidents of the United States, including their ordered number and state of birth. A data dictionary is a data structure that stores data in key-value pairs, allowing for efficient retrieval of data using its key. It then uses the built-in CSV library to create a CSV file and write the data to it. This code is used to store the US presidents' data in a structured format for future use, analysis, or display.

    import csv
    
    # Create a dictionary to store the data
    presidents = {
        "1": ["George Washington", "Virginia"],
        "2": ["John Adams", "Massachusetts"],
        "3": ["Thomas Jefferson", "Virginia"],
        "4": ["James Madison", "Virginia"],
        "5": ["James Monroe", "Virginia"],
        "6": ["John Quincy Adams", "Massachusetts"],
        "7": ["Andrew Jackson", "South Carolina"],
        "8": ["Martin Van Buren", "New York"],
        "9": ["William Henry Harrison", "Virginia"],
        "10": ["John Tyler", "Virginia"],
        "11": ["James K. Polk", "North Carolina"],
        "12": ["Zachary Taylor", "Virginia"],
        "13": ["Millard Fillmore", "New York"],
        "14": ["Franklin Pierce", "New Hampshire"],
        "15": ["James Buchanan", "Pennsylvania"],
        "16": ["Abraham Lincoln", "Kentucky"],
        "17": ["Andrew Johnson", "North Carolina"],
        "18": ["Ulysses S. Grant", "Ohio"],
        "19": ["Rutherford B. Hayes", "Ohio"],
        "20": ["James A. Garfield", "Ohio"],
        "21": ["Chester A. Arthur", "Vermont"],
        "22": ["Grover Cleveland", "New Jersey"],
        "23": ["Benjamin Harrison", "Ohio"],
        "24": ["Grover Cleveland", "New Jersey"],
        "25": ["William McKinley", "Ohio"],
        "26": ["Theodore Roosevelt", "New York"],
        "27": ["William Howard Taft", "Ohio"],
        "28": ["Woodrow Wilson", "Virginia"],
        "29": ["Warren G. Harding", "Ohio"],
        "30": ["Calvin Coolidge", "Vermont"],
        "31": ["Herbert Hoover", "Iowa"],
        "32": ["Franklin D. Roosevelt", "New York"],
        "33": ["Harry S. Truman", "Missouri"],
        "34": ["Dwight D. Eisenhower", "Texas"],
        "35": ["John F. Kennedy", "Massachusetts"],
        "36": ["Lyndon B. Johnson", "Texas"],
        "37": ["Richard Nixon", "California"],
        "38": ["Gerald Ford", "Nebraska"],
        "39": ["Jimmy Carter", "Georgia"],
        "40": ["Ronald Reagan", "Illinois"],
        "41": ["George H. W. Bush", "Massachusetts"],
        "42": ["Bill Clinton", "Arkansas"],
        "43": ["George W. Bush", "Connecticut"],
        "44": ["Barack Obama", "Hawaii"],
        "45": ["Donald Trump", "New York"],
        "46": ["Joe Biden", "Pennsylvania"]
    }
    
    # Open a new CSV file in write mode
    with open("presidents.csv", "w", newline='') as csv_file:
        # Specify the fieldnames for the columns
        fieldnames = ["Number", "Name", "State of Birth"]
        # Create a writer object
        writer = csv.DictWriter(csv_file, fieldnames=fieldnames)
        # Write the header row
        writer.writeheader()
        # Loop through the presidents dictionary
        for key, value in presidents.items():
            # Write the data for each president to the CSV file
            writer.writerow({
                "Number": key,
                "Name": value[0],
                "State of Birth": value[1]
                })
    # Print a success message
    print("Data successfully stored in CSV file.")
  

Data successfully stored in CSV file.