AI News, Can Machine Learning predict Poverty?
- On Sunday, June 3, 2018
- By Read More
Can Machine Learning predict Poverty?
World Bank hosted its poverty prediction competition on the competition hosting website drivendata.org.
Some information we can derive right by looking at the dataset is: One way to dive deeper into data (quickly) is to use the new package Pandas-Profiling (which can be downloaded from GitHub here). This package does a lot of primary analysis and saves them as pretty HTML files one can view on their browser.
Some more conclusions we can draw are: If one looks at the datatypes of the objects, they can see that the data is a mix of categorical (attributes which can take one out of a constant number of enumerable values) and numerical values (both floats and integers).
Another important property of dataset is the imbalance between +ve and -ve classes (non-poor people vastly outnumber poor people).
To train models on such skewed data, we tried different approaches using an imbalanced-learn library in Python: The dataset was preprocessed as follows: We now talk about multiple approaches that we tried.
However, unlike text, this dataset has no concept of sequence, so we decided to use a Neural Network common in text classification, but doesn’t take order into account.
Country B where the highest accuracy we ever received (even better than our best performing model) was using Self Normalized Deep Neural Network, the results don’t translate on the leaderboard where we keep getting low scores (high logloss).
We wrote a data pipeline for trying out different tricks we have mentioned (successful/unsuccessful) at the start of the segment and a pipeline to Grid Search over different hyperparameters and try a 5-fold Cross-Validation.
The tricks which worked above combined with Grid Search gave massive boosts to our scores and we could beat 0.2 logloss and then 0.9 logloss score too.
- On Thursday, January 17, 2019
Predicting the Winning Team with Machine Learning
Can we predict the outcome of a football game given a dataset of past games? That's the question that we'll answer in this episode by using the scikit-learn ...
How to load a custom dataset with tf.data [Tensorflow]
We look into how to create TFRecords to and handle images from a custom dataset. Later we load these records into a model and do some predictions. Github ...
How to do the Titanic Kaggle competition in R - Part 1
As part of submitting to Data Science Dojo's Kaggle competition you need to create a model out of the titanic data set. We will show you how to do this using ...
How to Predict Stock Prices Easily - Intro to Deep Learning #7
We're going to predict the closing price of the S&P 500 using a special type of recurrent neural network called an LSTM network. I'll explain why we use ...
How to Make an Image Classifier - Intro to Deep Learning #6
We're going to make our own Image Classifier for cats & dogs in 40 lines of Python! First we'll go over the history of image classification, then we'll dive into the ...
Build a TensorFlow Image Classifier in 5 Min
In this episode we're going to train our own image classifier to detect Darth Vader images. The code for this repository is here: ...
Exploring StackOverflow data
When you're stuck while programming - who you gonna call? StackOverflow! It's an invaluable source of daily help to many. Interestingly, you can also download ...
Intro to Azure ML: Splitting & Categorical Casting
Before we can feed this dataset into a machine learning model there are two things we have to take care of. First we have to make sure all the categorical ...
Data Structure - Creating a Chatbot with Deep Learning, Python, and TensorFlow p.2
What's going on everyone and welcome to the 2nd part of the chatbot with Python and TensorFlow tutorial series. By now, I am assuming you have the data ...
Building image classification using the Microsoft AI platform - BRK3334
Come see the latest additions to the Cognitive Toolkit, which offer a Python API, as well as a GUI to have a non‐disruptive experience from data load through ...