Read data from JSON and normalize then write into excel
read data from JSON and write in CSV
Install Pandas
bash
pip install pandasUbuntu Jupyter notebooks
bash
pip install jupyterlabjupyter labjupyter server listRead data
python
import pandas as pddf = pd.read_json('data.json')Show length of data
text
len(df.index)Give information about data frame
text
df.info()Get first element
Get first element and access to the index
text
first_element = df.iloc[0]first_element['helpdesk_ticket']Classes in python
python
class Person: def __init__(self, name, age): self.name = name self.age = agep1 =
Person("John", 36)print(p1.name)print(p1.age)What is a Series?
A Pandas Series is like a column in a table.
It is a one-dimensional array holding data of any type.
python
import pandas as pda = [1, 7, 2]myvar = pd.Series(a)print(myvar)text
0 11 72 2dtype: int64Similarity
Python For Loops
python
fruits = ["apple", "banana", "cherry"]
for x in fruits: print(x)Text Normalization for Natural Language Processing in Python
Install package
bash
pip install nltk
pip install unidecode
pip install pycontractions
pip install word2numberif got error of java version
text
$ sudo apt install openjdk-8-jdk$ sudo update-alternatives --set java
/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java$ pip install language-check$ pip install
pycontractionsbash
pip install language-tool-pythonDownload package
python
python3
import nltknltk.download()punktTokenization
This package will tokenize sentences.
python
sentences =
sent_tokenize(text)
print(sentences)['“Everything we’re doing is about going forward,” Phoebe Philo told Vogue in 2009, shortly before showing her debut Resort collection for Céline.', 'Although the label had garnered headlines when it was revived by Michael Kors in the late ’90s, it was Philo who truly brought the till then somewhat somnambulant luxury house to the forefront.', 'Critics credited her with pushing fashion in a new direction, toward a more spare, stripped-down kind of sophistication.', 'What Céline now offered women was, as the magazine put it, “a grown-up and hip way to put themselves together.”']Removing stopwords
python
stop_words = set(stopwords.words('english'))
filtered = [word for word in
word_tokenize(test_sentence) if word not in stop_words]
print(filtered)['“', 'Everything', 'going',
'forward', '”', 'Phoebe', 'Philo', 'told', 'Vogue', '2009', 'shortly', 'showing', 'debut', 'Resort',
'collection', 'Céline', '.']python
>>> import nltk>>> nltk.download('stopwords')Whitespace
python
test_sentence = re.sub(' +',' ', test_sentence)
print(test_sentence)Converting to lowercase
python
test_sentence = test_sentence.lower()
print(test_sentence)Expanding contractions
python
pattern = r'we[\’\']re'
replacement = 'we are'
test_sentence =
re.sub(pattern,replacement,test_sentence)
print(test_sentence)Downloader API for gensim
python
import gensim.downloader as apiapi.info() # return dict with info about available
models/datasetsapi.info("text8") # return dict with info about "text8" datasetpython
import gensim.downloader as apimodel = api.load("glove-twitter-25") # load glove vectorsSolve problem install package
text
Check Java installation : for language-check, Java 8 is expected For MacOS: Follow these steps
: https://stackoverflow.com/a/24657630 Install language-checkpip install language-check,
won't directly work, you would have to: Install from a forked & fixed repo for language-check pip install git+https://github.com/MCFreddie777/language-check.git Install pyemdpip install pyemd again runs into installation issues in Python 3.7, therefore, follow the following steps: Ensure you have XCode installed - xcode-select --install Set export MACOSX_DEPLOYMENT_TARGET=10.9 pip install pyemd, should now work Now, pip install pycontractions should work!Simple normalization
python
@staticmethod def simple_normalize(text) : text = re.sub(' +',' ', text) text =
text.replace('\n', ' ') text = text.replace('\t', ' ') text = text.replace('\r', ' ')
text = text.replace('\"', ' ') text = text.replace('\'', ' ') text = text.lower()
pattern = r'we[\'\']re' replacement = 'we are' text = re.sub(pattern,replacement,text)
pattern = r'we[\'\']ll' replacement = 'we will' text =
re.sub(pattern,replacement,text) return text