This dataset contains product reviews and metadata from Amazon, including 142.8 million reviews spanning May 1996 - July 2014 for various product categories. I have analyzed dataset of kindle reviews here.
import json
from textblob import TextBlob
import pandas as pd
import gzip
Data Extraction
For this exercise I am using Amazon Kindle reviews dataset from http://jmcauley.ucsd.edu/data/amazon/. I have already downloaded & extracted json file in current directory
#Load review data from json file
reviews = []
for line in open('reviews_Kindle_Store_5.json', 'r'):
reviews.append(json.loads(line))
#Display first record
reviews[:1]
[{'asin': 'B000F83SZQ',
'helpful': [0, 0],
'overall': 5.0,
'reviewText': "I enjoy vintage books and movies so I enjoyed reading this book. The plot was unusual. Don't think killing someone in self-defense but leaving the scene and the body without notifying the police or hitting someone in the jaw to knock them out would wash today.Still it was a good read for me.",
'reviewTime': '05 5, 2014',
'reviewerID': 'A1F6404F1VG29J',
'reviewerName': 'Avidreader',
'summary': 'Nice vintage story',
'unixReviewTime': 1399248000}]
#Lets extract all reviews from this dataset
all_reviews = []
for i in range(len(reviews)):
all_reviews.append(reviews[i]['reviewText'])
#Display first 5 extracted reviews
all_reviews[:5]
["I enjoy vintage books and movies so I enjoyed reading this book. The plot was unusual. Don't think killing someone in self-defense but leaving the scene and the body without notifying the police or hitting someone in the jaw to knock them out would wash today.Still it was a good read for me.",
"This book is a reissue of an old one; the author was born in 1910. It's of the era of, say, Nero Wolfe. The introduction was quite interesting, explaining who the author was and why he's been forgotten; I'd never heard of him.The language is a little dated at times, like calling a gun a "heater." I also made good use of my Fire's dictionary to look up words like "deshabille" and "Canarsie." Still, it was well worth a look-see.",
"This was a fairly interesting read. It had old- style terminology.I was glad to get to read a story that doesn't have coarse, crasslanguage. I read for fun and relaxation......I like the free ebooksbecause I can check out a writer and decide if they are intriguing,innovative, and have enough of the command of Englishthat they can convey the story without crude language.",
"I'd never read any of the Amy Brewster mysteries until this one.. So I am really hooked on them now.",
'If you like period pieces - clothing, lingo, you will enjoy this mystery. Author had me guessing at least 2/3 of the way through.']
#Lets count total number of reviews
len(all_reviews)
#We have nearly a million reviews on kindle
982619
Analyzing Review Sentiments using TextBlob
#Lets create a textblob object for a review text
blob = TextBlob(all_reviews[1])
#Lets check sentiment of this review
blob.sentiment
#This returns 2 numbers, but what are they?
Sentiment(polarity=0.2825, subjectivity=0.38)
Whats Polarity & Subjectivity?
Polarity: Polarity score is a float within the range [-1.0, 1.0], also known as orientation polarity is the emotion expressed in the sentence. It can be positive, negative or neutral.
Subjectivity: It measures subjectivity of sentence or to what extent it represents someones personal feelings, views, or beliefs compared to objective truth or facts. subjectivity is a float within the range [0.0, 1.0] where 0.0 is very objective and 1.0 is very subjective.
#Lets find review sentiment for each review similarly
polarity_list,subjectivity_list=[],[]
for i in range(len(all_reviews)):
blob = TextBlob(all_reviews[i])
polarity_list.append(blob.sentiment.polarity)
subjectivity_list.append(blob.sentiment.subjectivity)
#Building above lists takes a lot of time, lets save them as csv to save time in future
polarity_df = pd.DataFrame(polarity_list, columns=["column"])
polarity_df.to_csv('polarity_list.csv', index=False)
subjectivity_df = pd.DataFrame(subjectivity_list, columns=["column"])
subjectivity_df.to_csv('subjectivity_list.csv', index=False)
#Checking few values from created lists
print(polarity_list[:5])
print(subjectivity_list[:5])
[0.45, 0.2825, 0.13750000000000004, 0.2, 0.05000000000000002]
[0.7000000000000001, 0.38, 0.5875, 0.2, 0.45]
#Lets plot distribution of polarity
import seaborn as sns
sns.set(color_codes=True)
from pylab import *
#Polarity Distribution
sns.distplot(polarity_list)
show()
#Subjectivity Distribution
sns.distplot(subjectivity_list)
show()
#Getting statistical summary of polarity & subjectivity
from scipy import stats
print(stats.describe(polarity_list))
print(stats.describe(subjectivity_list))
DescribeResult(nobs=982619, minmax=(-1.0, 1.0), mean=0.2554416996022165, variance=0.03613085062383566, skewness=-0.0034846642680489612, kurtosis=1.966151107339833)
DescribeResult(nobs=982619, minmax=(0.0, 1.0), mean=0.5485420019964269, variance=0.017758890024147884, skewness=-0.22705026370838458, kurtosis=1.817626935291675)
#Lets check few highly positive reviews with very high Polarity(>.9)
c = 0
for i in range(len(polarity_list)):
if(c>10): break
if(polarity_list[i]>.9 and c<10) :
print(all_reviews[i],"\n")
c+=1
What can I say Stormtroopers. A story with troopers is as awesome as a story with any Sith lord. AWESOME!!!!!!
At 1st, I didn't realize who the author was. This is a wonderful finish to "20,000 Leagues". Kept my interest.
This is a very good story about the lives of orphans and the problems they encounter. I recommend it and have read it a couple of times. Well written.
Excellent work from a great author. Well thought out and written series of books. And what a wonderful way to finish it.
She did a wonderful job at turning the character of someone that I could not stand into someone I began to champion.
Was a very good story I had started with the 4th one and then found the bundle, read one after the other,and reread the 4th one. Very good!
I started reading Eve Vaughn books on Thursday. Since then I have four of books. Let me tell u they will grip you until the end. Awesome read.
Great addition to the BDB series! I think J.R. Ward should do a follow-up book like this for Rhage as well. GREAT read!
Zsadist being who he is and what has happened to him and then finding Bella and how she helps transform him into what we all know him to be.....loving. Their story was was of the best in the BDB books.
As with the red/pink one of these i Have, works great to protect my Kindle, when not in use! Would buy again!
#Lets check few highly negative reviews with very high Polarity(<-.9)
c = 0
for i in range(len(polarity_list)):
if(c>10): break
if(polarity_list[i]<-.9 and c<10) :
print(all_reviews[i],"\n")
c+=1
It had never sunk in to me that these stories were so grim and horrid. I thought I was getting a book of children's fairy tales that I could read to my grandchildren! Be forewarned!
This Is The Worse Book I Have Ever Read!!!! Oh my goodness, who edited this garbage? Ms. McCray please don't write anymore books. What an awful story! What an awful end! BLECH!
i dont like it.its a waste of my time and money. i should of gotten something else. dont buy this it's terrible.boooooooooooooooooooooooooo!!!!!!!!!!!!!!!!!!!
It was a truly horrible read, and I'm not talking about the story, but the language of the book. It sounded strained and unnatural. I don't recommend this story.
God I hate books like this! Hate them! If I were reading this on paperback it would have been in shreds! This guy treats her like s#*t and yet whenever he touches her she just can't control herself. She's like a freakin' ally cat in heat. She has no pride! And he's an arrogant prick. Hate, hate, hate the characters! If I could have, I would have given this book a zero.
BORING BORING BORING! I didn't even bother to finish it, and have deleted it from my Kindle. Not recommended at all!
This is another book about the evil in this world.It also is a understanding of what goes through a child's mind and the despair they have.Highly recommend this if you are reading like to read about the unjust in this world.
It was a awful book, it was really boring. I didn't like it at all. It didn't have a story line
so hated this book not my cup of tea amistake to get very nasty i do not talk alot nasty
again, I have not read it, not even started it, so assume it is as boring as the rest of the books in the series
#Lets check few subjective reviews
c = 0
for i in range(len(subjectivity_list)):
if(c>10): break
if(subjectivity_list[i]>.9 and c<10) :
print(all_reviews[i],"\n")
c+=1
What can I say Stormtroopers. A story with troopers is as awesome as a story with any Sith lord. AWESOME!!!!!!
I have been a fan of Mark Twain ever since I found we share a birthday. I am glad I found this story.
Though these be the tales of the brothers, the tales are incorrect. These stories are happy tales while the tales written by the brothers were horrid and without a happy ending.
It had never sunk in to me that these stories were so grim and horrid. I thought I was getting a book of children's fairy tales that I could read to my grandchildren! Be forewarned!
am sorry ---- i accidentally deleted it from the kindle before i had time to check it out. i apologize.
At 1st, I didn't realize who the author was. This is a wonderful finish to "20,000 Leagues". Kept my interest.
A group of men escape by balloon a Confederate prison and land on a deserted island where all their needs are met by a mysterious benefactor.
Finally a Texas newspaper in Kindle format.Way to go Amazon !!
Excellent work from a great author. Well thought out and written series of books. And what a wonderful way to finish it.
I've never read McBain before and was disappointed that I found it boring. I only read 27% when I gave up and switched to another author.
#Now lets check some objective reviews
c = 0
for i in range(len(subjectivity_list)):
if(c>10): break
if(subjectivity_list[i]<.1 and c<10) :
print(all_reviews[i],"\n")
c+=1
He doesn't need my recommendation but since you asked, unless you are into thorough reprobates, give it a miss. As the wag once put it, Mark Twain shows promise.
I grew up considering Jules Verne and his contamparies as people with a scientific vision. But having read his work, including this one, I feel betrayed. These are racist people, believing in white man's superiority, which looks like a recurrent theme with bit of pseudoscience thrown in. Do not waste time.
the is a story about man, life,working together for the betterment of all.the betterment of this group of man.the respect life,all things intended .
I downloaded the Baltimore Catechism to do some research for a book I'm writing and for the memories of St. Mary's in Rockford, IL that I associate with it. It fully met both of those expectations.I will add that I never recieved five stars from any of the nuns for any reason.
lose the war,lose your boat lose your girl(maybe).kind of an insight into the mentality of "upper class germans" & their take on things
It was not like the in-print edition. It was printed like a word document. Not what I wanted. I didn't want to pay for that.
i would prefer to have images and ads. but for the price difference it is a no brainer. going to cancel print subscription. save trees. give the nation difference as a donation.
Lost mines of King Solomon, found them, what next for Allan Quartermain.Quatermain and his friends return to Africa for their next adventure.
i had the paperback of this series it keeps you reading i reread this book all the time it rocks
I bought this book because I read it once before and I thought I'd like to read it again I like fantasy and I recommend this book for anyone that likes fantasy also
#Lets check some neutral reviews
c = 0
for i in range(len(polarity_list)):
if(c>10): break
if(polarity_list[i]==0 and c<10) :
print(all_reviews[i],"\n")
c+=1
He doesn't need my recommendation but since you asked, unless you are into thorough reprobates, give it a miss. As the wag once put it, Mark Twain shows promise.
Not at all what I had expected. I was thinking it would be like the movie but was very pleasantly surprised.
the is a story about man, life,working together for the betterment of all.the betterment of this group of man.the respect life,all things intended .
A group of men escape by balloon a Confederate prison and land on a deserted island where all their needs are met by a mysterious benefactor.
I downloaded the Baltimore Catechism to do some research for a book I'm writing and for the memories of St. Mary's in Rockford, IL that I associate with it. It fully met both of those expectations.I will add that I never recieved five stars from any of the nuns for any reason.
just knowing the mythological background of our native inhabitants helps in understanding-not only those peoples, but ourselves and those around us.Thank you Amazon for providing this link to a part of history that should be known by all of us.120 words is not really enough to describe the wonderfulness of this book-well, I could go on and on
lose the war,lose your boat lose your girl(maybe).kind of an insight into the mentality of "upper class germans" & their take on things
Finally a Texas newspaper in Kindle format.Way to go Amazon !!
It was not like the in-print edition. It was printed like a word document. Not what I wanted. I didn't want to pay for that.
I cannot imagine life without IHT. Sorry? 14 more words needed? How about a repeat? I cannot imagine life without IHT.
Thumbnail Image by Claudio Toledo via https://www.flickr.com/photos/catr/2443946244