Sign in to continue learning
Natural Language Processing in TensorFlow
23h32m
So far this week, you've been looking at text and how to tokenize the text and then turn sentences into sequences using the tools available in TensorFlow. You did that using some very simple hard-coded sentences, but of course, when it comes to doing real-world problems, you'll be using a lot more data than just these simple sentences. So in this lesson, we'll take a look at some public datasets and how you can process them to get them ready to train a neural network. We'll start with this one, published by Rishabh Misra, with details on Kaggle at this link. It's a really fun CC0 public domain dataset at all around sarcasm detection. Really? Yeah, really. This dataset is very straightforward and simple, not to mention very easy to work with. It has three elements in it. The first, isSarcastic, is our label. It's a one if the record is considered sarcastic, otherwise it's a zero. The second is a headline, which is just plain text, and the third is the link to the article that the headline describes. Parsing the contents of HTML, stripping out scripts and styles, etc., is a little bit beyond the scope of this course, so we're just going to focus on the headlines. If you download the data from that Kaggle site, you'll see something like this. As you can see, it is a set of list entries with name-value pairs where the name is article-link, headline, and isSarcastic, and the values are as shown. To make it much easier to load this data into Python, I made a little tweak to the data to look like this, which you can feel free to do, or you can download my amended dataset from the link in the Colab for this part of the course. Once you have the data like this, it's then really easy to load it into Python. Let's take a look at the code. So first, you need to import JSON. This allows you to load data in JSON format and automatically create a Python data structure from it. To do that, you simply open the file and pass it to JSON.load, and you'll get a list containing lists of the three types of data, headlines, URLs, and isSarcastic labels. Because I want the sentences as a list of their own to pass to the tokenizer, I can then create a list of sentences. And later, if I want the labels for creating a neural network, I can create a list of them too. While I'm at it, I may as well do URLs, even though I'm not going to use them here, but you might want to. And now I can iterate through the list that was created with a forItemInDataStore loop, and for each item, I can then copy the headline to my sentences, the isSarcastic to my labels, and the article link to my URLs. And now I have something I can work with. So let's look at that next.