Read data from JSON and normalize then write into excel

read data from JSON and write in CSV

Install Pandas

bash
pip install pandas

Ubuntu Jupyter notebooks

bash
pip install jupyterlabjupyter labjupyter server list

Read data

python
import pandas as pddf = pd.read_json('data.json')

Show length of data

text
len(df.index)

Give information about data frame

text
df.info()

Get first element

Get first element and access to the index

text
first_element = df.iloc[0]first_element['helpdesk_ticket']

Classes in python

python
class Person:  def __init__(self, name, age):    self.name = name    self.age = agep1 =
Person("John", 36)print(p1.name)print(p1.age)

What is a Series?

A Pandas Series is like a column in a table.

It is a one-dimensional array holding data of any type.

python
import pandas as pda = [1, 7, 2]myvar = pd.Series(a)print(myvar)
text
0    11    72    2dtype: int64

Similarity

Python For Loops

python
fruits = ["apple", "banana", "cherry"]
for x in fruits:  print(x)

Text Normalization for Natural Language Processing in Python

Install package

bash
pip install nltk
pip install unidecode
pip install pycontractions
pip install word2number

if got error of java version

text
$ sudo apt install openjdk-8-jdk$ sudo update-alternatives --set java
/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java$ pip install language-check$ pip install
pycontractions
bash
pip install language-tool-python

Download package

python
python3
import nltknltk.download()punkt

Tokenization

This package will tokenize sentences.

python
sentences =
sent_tokenize(text)
print(sentences)['“Everything we’re doing is about going forward,” Phoebe Philo told Vogue in 2009, shortly before showing her debut Resort collection for Céline.', 'Although the label had garnered headlines when it was revived by Michael Kors in the late ’90s, it was Philo who truly brought the till then somewhat somnambulant luxury house to the forefront.', 'Critics credited her with pushing fashion in a new direction, toward a more spare, stripped-down kind of sophistication.', 'What Céline now offered women was, as the magazine put it, “a grown-up and hip way to put themselves together.”']

Removing stopwords

python
stop_words = set(stopwords.words('english'))
filtered = [word for word in
word_tokenize(test_sentence) if word not in stop_words]
print(filtered)['“', 'Everything', 'going',
'forward', '”', 'Phoebe', 'Philo', 'told', 'Vogue', '2009', 'shortly', 'showing', 'debut', 'Resort',
'collection', 'Céline', '.']
python
>>> import nltk>>> nltk.download('stopwords')

Whitespace

python
test_sentence = re.sub(' +',' ', test_sentence)
print(test_sentence)

Converting to lowercase

python
test_sentence = test_sentence.lower()
print(test_sentence)

Expanding contractions

python
pattern = r'we[\’\']re'
replacement = 'we are'
test_sentence =
re.sub(pattern,replacement,test_sentence)
print(test_sentence)

Downloader API for gensim

python
import gensim.downloader as apiapi.info()  # return dict with info about available
models/datasetsapi.info("text8")  # return dict with info about "text8" dataset
python
import gensim.downloader as apimodel = api.load("glove-twitter-25")  # load glove vectors

Solve problem install package

text
Check Java installation :    for language-check, Java 8 is expected    For MacOS: Follow these steps
: https://stackoverflow.com/a/24657630    Install language-checkpip install language-check,
won't directly work, you would have to:    Install from a forked & fixed repo for language-check    pip install git+https://github.com/MCFreddie777/language-check.git    Install pyemdpip install pyemd again runs into installation issues in Python 3.7, therefore, follow the following steps:    Ensure you have XCode installed - xcode-select --install    Set export MACOSX_DEPLOYMENT_TARGET=10.9    pip install pyemd, should now work    Now, pip install pycontractions should work!

Simple normalization

python
@staticmethod    def simple_normalize(text) :        text = re.sub(' +',' ', text)        text =
text.replace('\n', ' ')        text = text.replace('\t', ' ')        text = text.replace('\r', ' ')
text = text.replace('\"', ' ')        text = text.replace('\'', ' ')        text = text.lower()
pattern = r'we[\'\']re'        replacement = 'we are'        text = re.sub(pattern,replacement,text)
pattern = r'we[\'\']ll'        replacement = 'we will'        text =
re.sub(pattern,replacement,text)         return text