Bake, Craft, Code, Life
Text summarization, as the name implies, is a technique to summarize large pieces of text. Our goal is to generate summaries that make sense – remember high school and summarizing papers and books? Yes, we have to get the machine to do that. This far from an easy task; it is actually an area of…
Read More
Unpopular opinion: LDA models can be problematic. tl:dr; LDA models are often overused and misused. Latent Dirichlet Allocation (LDA) is a very common model used in natural language processing with the intent of clustering some amount of text data into topics. LDAs have become the staple natural language processing model with hundreds of tutorials…
Read More
If you have been in the field for a while, you’ll have realized by now that every single job you’ve had as a “data scientist” consists of different responsibilities and the expectations they have of you are also vastly different. Here I’ve listed some of the things I have learned along the way that have…
Read More
A lot of times I get asked about the applications of Natural Language Processing. If you look at most tutorials online, a lot of NLP is based on books and chat bots, but NLP can be applied to many other areas. A big are where NLP can be applied successfully is consumerism and merchandising. It…
Read More
Often we will be dealing with text; short or long, and often we only want to find out the important keywords of said text. As with most things, there are several algorithms out there that you could use for this. A common one that you’ve likely even used before is TF-IDF, but there are others…
Read More
Everyone knows South Park. I’ll admit I haven’t watched an episode in at least 5 years, but once I saw this data set I had to dig in. It’s a prime sample to do some quick and dirty linguistic pattern search. It also provide us with a way to understand how to “measure” language. The…
Read More
NOTE: THIS WAS DONE USING PYTHON 2.7. PLEASE, REFER TO UPDATED POST HERE. A lot of times we need to dig through hundreds of lines of text. Maybe we are digging through research journals or we need to go through large amounts of text. A great technique to speed up this process is by using…
Read More