AI News, Sarcasm Detection with Machine Learning in Spark
- On Monday, June 4, 2018
- By Read More
Sarcasm Detection with Machine Learning in Spark
This post is inspired by a site I found whilst searching for a way to detect sarcasm within sentences.
This search led me to the above link site where the author Mathieu Cliche cleverly came up with the idea of using tweets as the training set.
However searching for tweets that contain the hastag #sarcasm or #sarcastic would provide me with a vast amount of training data (providing a good percentage of those tweets are actually sarcstic).
Using that approach as the basis, I developed a Spark application using the MlLib api that would use the Naive Bayes classifier to detect sarcasm in sentences - This post will cover the basics and I will be expanding on this next time to utilise sarcastic tweets to train my model!
The file should have 2 “columns”, the first for the label(I used 1 for a sarcastic row and 0 for a non sarcastic row) and the second for the sentence.
Create two data frames: each with 2 columns “label” and “text” - one data frame for the training data, the other for the test data.
This data can then be used by the algorithm to build a model allowing it to predict/guess whether a similar vector is also sarcastic (or not).
This will now build a model that can be used to classify new sentences - that the model has never seen before - as sarcastic or not sarcastic by seeing if the new sentence (when converted to a vector) is more similar to sarcastic vectors, or non sarcastic vectors.
We want to create a tuple containing the predicted value, and the original label that we gave the data so we can see how accurate it performed.
When you (or anyone else) wishes to then predict the level of sarcasm within a sentence, they can simply write a Spark application that loads your model and can then use it as shown previously.
- On Tuesday, January 22, 2019
funny communication skills
how to communicate with others
How to Do Sentiment Analysis - Intro to Deep Learning #3
In this video, we'll use machine learning to help classify emotions! The example we'll use is classifying a movie review as either positive or negative via TF Learn ...
Sentiment Analysis in 4 Minutes
Link to the full Kaggle tutorial w/ code: Sentiment Analysis in 5 lines of ..
Garmin 430 Tricks and Secrets Revealed
If you have a Garmin GNS430, you probably know the basics, but in this video, IFR magazine's Jeff Van West reveals some ninja-level operating tricks that are ...
How David Fincher Hijacks Your Eyes
Get 10% off any purchase here: PATREON: T-SHIRT: .
Adding Machine Learning to your applications
Google provides infrastructure, services, and APIs for you to create your own machine learning models. They also have pretrained machine learning APIs that ...
Learning English Live - 22nd April 2018 - Prize Meanings - Ladybird Facts - Grammar - Chat
Learning English Live - Sunday 22nd April 2018. Improve your listening skills with the live chat. Today we will look at uses of the word 'set'. Things you might not ...
SV Seeker Live in HD - Apr 28, 2018 - The Diesel Jet Boat is ready for testing!
This is a higher resolution version of the Live video from earlier today. Use the code SPRING15 and get 15% off anything in our Overpriced Junk Store!
Life is easy. Why do we make it so hard? | Jon Jandai | TEDxDoiSuthep
Never miss a talk! SUBSCRIBE to the TEDx channel: Jon is a farmer from northeastern Thailand. He founded the Pun Pun Center for ..