Sign in to continue learning
Data I/O and Preprocessing with Python and SQL
24 hours 38 mins
Previously, you used the request module to get a web page, and what you got back was an HTML document. In this video, you'll explore how HTML is used to structure web pages, and how that structure affects web scraping. Nearly all web pages use HTML, Hypertext Markup Language. You can think of HTML as text enhanced with tags that provide additional information. Tags typically open and close, wrapping around some content. For example, the SeaSky website you saw in the previous video included the page title, Astronomy Calendar of Celestial Events, for calendar year 2030. This text is wrapped in an H1 tag. H1 means Level 1 Heading, the largest heading type. So the browser displaying this web page knows to format it as a large heading. Notice that the closing tag includes a forward slash. This entire block, including the tags and content, is called an element, so this title is an H1 element. You'll commonly see elements like H1, H2, all the way to H6 for headings, P for paragraph, A, which stands for anchor, for links to other pages, and so on. While most elements have both opening and closing tags like this one, HTML also includes a few self-closing tags that do not need a separate closing tag. The most common self-closing tag is IMG for images. HTML tags can also have attributes, additional information about an element, like its size, color, position, or content source. These attributes go inside the angle brackets of the opening tag. One of the most common attributes is href in the A tag, which creates a URL link. The href attribute, which stands for hypertext reference, specifies where the URL link should take you when the element is clicked. For example, this A tag from the CSky website contains the content US Naval Observatory and links to their website at usno.navy.mil.usno. Without the href attribute, this A tag wouldn't link anywhere. Take a look at this image tag from the CSky website. It contains four attributes, src, alt, width, and height. Src stands for source and specifies the URL or file path of the image to display. This image is displaying the calendar-legend.jpg image from the CSky website. Alt stands for alternative text, what shows up if the image fails to load. Width and height tell the browser how large the displayed image should be. Each HTML element, not just images, takes up some space on the page defined by its height and width. This element explicitly tells the browser how large the image should be. HTML classes are a special type of attribute that allow you to group elements together and apply consistent styling or functionality to them. A class is defined using the class attribute inside an HTML tag and is commonly assigned to multiple elements. For example, notice that each date in the CSky website is formatted in bold and green. Rather than setting the color and style for each individual element, the CSky website groups these elements in the same class, date text. Using classes to group elements makes some web scraping easier. In this case, you can use one class attribute, date text, to find every date in this website, assuming it's properly formatted. While HTML tags describe the purpose and appearance of individual elements, the HTML document as a whole structures individual elements in a hierarchy. Many elements can contain other elements. For example, a paragraph element might include several anchor elements for links. The structure of HTML is like a tree, with elements organized inside other elements. At the root of the tree, you'll find the HTML tag, which contains all the HTML content for a particular web page. This organization matters because if you're looking for a particular element, like this paragraph containing a celestial event, you'll need to know where it's located in the HTML tree. Take a look at the organization of the CSky website. It's organized into a header at the top for navigation, a sidebar on the left with different astronomy pages, and the main content on the right. Within this main content area, you have the title of the page, links to different years, an overall description, and then a legend for the different types of events, and then you get into the individual events themselves. HTML includes several container elements that are intended to help organize content logically by grouping other elements. The most common container you'll see is div, short for division. It's a flexible container that acts as an empty box for storing multiple elements. A division creates a block that starts on a new line and takes up the full available width on the page. You'll also see unordered list elements, essentially a bulleted list, and ordered list elements, a numbered list. These each contain multiple list item elements. You'll also encounter spans, which are often used to organize small pieces of text. They act similarly to divisions, like an empty box for storing multiple elements, but they don't start on a new line or take up the full available width. In the previous lesson, you used pd.readhtml to scrape tables from a website. This function only works for properly organized table elements, which are a type of container. Here's a snippet of what an HTML table with similar content might look like. Notice it's quite complex, with rows specified by tr and data specified by td. See, pd.readhtml throws an error with the C-Sky website, because even though the information is well organized, there aren't any HTML table elements. Don't feel the need to memorize each of the elements you saw in this video. The more you work with them, the better you'll understand them. You can always chat with your LLM to help you remember the details. In this video, you've seen how HTML forms the foundation of a webpage. Follow me to the next video to learn how to parse HTML with BeautifulSoup.