import string
documents = ["this isn't a sample.",
'this is another example.' ,
'this" also appears in the second example.'
'Is this an example?']
documents
table = str.maketrans('', '', string.punctuation)
doc_removed_punctuation = [w.translate(table) for w in documents]
doc_removed_punctuation
This post aims to introduce Bag of words which can be used as features for each document or images.
Simply, bag of words are frequency of word with associated index for each word.
import pandas as pd
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfTransformer
documents = ['this is a sample.',
'this is another example. "this" also appears in the second example.']
documents
count_vect = CountVectorizer()
word_counts = count_vect.fit_transform(documents)
df_word_counts = pd.DataFrame(word_counts.todense(), columns=count_vect.get_feature_names())
df_word_counts
tf_transformer = TfidfTransformer(use_idf=False).fit(df_word_counts)
word_freq = tf_transformer.transform(df_word_counts)
df_word_freq = pd.DataFrame(word_freq.todense(), columns=count_vect.get_feature_names())
df_word_freq
This post aims to introduce term frequency-inverse document frequency as known as TF-IDF, which indicates the importance of the words in a document considering the frequency of them across multiple documents and used for feature creation.
Term Frequency (TF)
Term Frequency can be computed as the number of occurrence of the word $n_{word}$ divided by the total number of words in a document $N_{word}$.
\begin{equation*} TF = \frac{n_{word}}{N_{word}} \end{equation*}Document Frequency (DF)
Document Frequency can be computed as the number of documents containing the word $n_{doc}$ divided by the number of documents $N_{doc}$.
\begin{equation*} DF = \frac{n_{doc}}{N_{doc}} \end{equation*}Inverse Document Frequency (IDF)
The inverse document frequency is the inverse of DF.
\begin{equation*} IDF = \frac{N_{doc}}{n_{doc}} \end{equation*}Practically, to avoid the explosion of IDF and dividing by zero, IDF can be computed by log format with adding 1 to denominator as below.
\begin{equation*} IDF = log(\frac{N_{doc}}{n_{doc}+1}) \end{equation*}Reference
import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer
documents = ['this is a sample.',
'this is another example. "this" also appears in the second example.']
documents
Now applying TF-IDF to each sentence, we will obtain the feature vector for each document accordingly.
vectorizer = TfidfVectorizer()
X = vectorizer.fit_transform(documents)
df_tfidf = pd.DataFrame(X.todense(), columns=vectorizer.get_feature_names())
df_tfidf
# The highest TF-IDF for each document
df_tfidf.idxmax(axis=1)
# TF-IDF is zero if the word does not appear in a document
df_tfidf==0
This post aims to introduce how to do survival analysis using lifelines. In this post, I use fellowship information in 200 Words a day to see what the survival curve looks like, which might be useful for users retention.
200 Words a day is the platform where those who wants to build a writing habit make a post with more than 200 words. There is a function to show how many consecutive days each user has been making a post as X-day streak.
Reference
from bs4 import BeautifulSoup
import requests
html_simple = '<h1>This is Title<h1>'
html_simple
soup = BeautifulSoup(html_simple)
soup.text
This post aims to show how to calculate the trace of a matrix using numpy i.e., $tr(A)$
$tr(A)$ is defined as
$$ tr(A) = \sum_{i=1}^{n} a_{ii} = a_{11} + a_{22} + \cdots + a_{nn} $$Reference:
import numpy as np
arr = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
arr
arr.trace()
sum([arr[i, i] for i in range(len(arr))])
This post aims to introduce how to conduct dimensionality reduction with Principal Component Analysis (PCA).
Dimensionality reduction with PCA can be used as a part of preprocessing to improve the accuracy of prediction when we have a lot of features that has correlation mutually.
The figure below visually explains what PCA does. The blue dots are original data points in 2D. The red dots are projected data onto 1D rotating line. The red dotted line from blue points to red points are the trace of the projection. When the moving line overlaps with the pink line, the projected dot on the line is most widely distributed. If we apply PCA to this 2D data, 1D data can be obtained on this 1D line.
Reference
This post aims to introduce k-means clustering using artificial data.
This post aims to describe an array using pandas. As an example, Boston Housing Data is used in this post.
Reference
import pandas as pd
from sklearn.datasets import load_boston
%matplotlib inline
boston = load_boston()
df_boston = pd.DataFrame(boston['data'], columns=boston['feature_names'])
df_boston.head()
pandas DataFrame has a method, called describe, which shows basic statistics based on the data types for each columns
df_boston.describe()