distinguishing gender and filtering emails using hdfs/nltk/python.
$10-30 USD
Ακυρώθηκε
Αναρτήθηκε πάνω από 7 χρόνια πριν
$10-30 USD
Πληρωμή κατά την παράδοση
Problem Statement 1:
We are thinking to take the live streaming data such as twitter tweets and analyze the tweet, differentiate the male and female tweets and represent them graphically. Generally, names ending with a, e and i are most likely the female names. So, we'd use this concept to distinguish male and female names.
Instructions:
Extract data from twitter. (using flume)
Store it in text file (store the extracted files in hdfs)
Do sentiment Analysis using nltk (use the files in hdfs to do sentiment analysis using nltk)
Distinguish the gender by the twitter names
Create graphical report using visualization
Store visualization report in hdfs using sandbox
Note: steps 3,4,5 should be written in python
Problem Statement 2:
Generally, people get hesitated when they are likely to see a spam email in their mailbox as they are expecting an important email. Emails consisting of words such as "won", "rewards", "lottery", "lucky" are mostly spam emails. So we'd like to use this concept and filter emails. We are planning to use Gmail data for streaming.
Instructions:
Extract data from a selected gmail account
Store it in hdfs using sandbox
Extract the text data from sandbox and analyze the data using nltk
Here filteration should be done using keywords such as won, rewards, lottery, jackpot, rebate, mailin rebate, lucky, winner and distinguish the email as spam or not
Use the python and nltk to visualize the filtered data
Using sandbox save data in hdfs.
Conditions:
If the mail contains the words won lottery, won jackpot, lucky winner, winner of the day are possibly 100% spam
If the mail contains keywords lucky, rebate is possibly 90% spam
If the mail contains keywords mailin rebate, rewards are 80% spam
Implementation:
We will
1. Collect the live streaming data
2. Write into a text file
3. Give the text file as input to python mapper and reducer
4. Use nltk, python for visualization of data.
Note:
Project should be completed in 3 days.
Execution should be shown using teamviewer.
Project ID: 12250178
Skills Required
Python, Natural Language, Hadoop