Category Archives: Statistics

Cryptographic roots in Blockchain technology

The Blockchain is expected to be the revolutionary technology to take the centre stage in our society where the traditional ledger system once dominates, from bitcoin that emerged in the finance sector to fields where transactions are dependent on authenticity, be it a paper document from bank, an import / export data exchange, or even documents in judicial systems, it is important to understand the principles of its fundamental roots in cryptography.

For example, to ensure the rightful spending of currency in bitcoins, there are a lot of technology being in place on the virtual money market. One being the Elliptic Curve Cryptography that is based on mathematics to ensure the identity of parties involved in any bitcoin transactions.



Plotting gamma distribution in TI Nspire

The gamma function is not a built-in function in the TI Nspire. Nevertheless this function can easily be defined and used for visualising the gamma probability distribution function.



Poisson prediction for soccer match odds

It is the World Cup season. There are many prediction models, and one of the most widely used statistical technique is the Poisson distribution:


The historic match results are available in the public domain, for example, at The data are analysed to obtain an index referred to attack strength or goal expectancy. This can further elaborates into more complex data like home team and away team expectancy.


By using a matrix of score scales, usually from 0 to 9, all possible outcomes under 10 goals per team are defined with the respective probability. An example matrix with scores from 0 to 2 will look like one below:


This is the basics of predicting soccer in a quantitative way using the Poisson distribution. There are of course many other prediction methods and models, including organic method like asking the famous Paul the octopus 🙂

51% Blockchain attack

One of the predicted, and proved feasible, attack to the blockchain technology is the 51% attack. There are several forms of this attack including weakness in the blockchain algorithm itself that allowed easier than usual forking of chain to dishonestly win the popular vote mechanism. In other cases, the perpetrator has to amass more hashing power than all other nodes in the blockchain combined to corrupt the distributed ledger, and hence the name of “51%”.

An interesting mathematical study on the effectiveness of this attack is based on the regularized incomplete beta function to calculate the probability of success of a perpetrator.


Looking for insights from Fitbit data with R

With a month’s of Fitbit data, it’s about time to harvest for some insights from this technology packed wristband.


fitbitdata = read.csv('fitbit.csv')
fitbitdata % mutate(dow = wday(Date))
fitbitdata$dowlabel <- factor(fitbitdata$dow,levels=1:7,labels=c("Mon","Tue","Wed","Thu","Fri","Sat","Sun"),ordered=TRUE)
fitbitdata$scyl <- as.factor(as.integer(fitbitdata$distance)/max(fitbitdata$distance))
c1 <- rainbow(7)
c2 <- rainbow(7, alpha=0.4)
c3 <- rainbow(7, v=0.8)
boxplot(fitbitdata$steps~fitbitdata$dowlabel, col=c2, medcol=c3, whiskcol=c1, staplecol=c3, boxcol=c3, outcol=c3, pch=23, cex=2, alpha=fitbitdata$scyl)


Number of steps and distance traveled data per day is collected from the Fitbit’s phone app, converted into CSV format, and then uploaded to R for data analysis. With a few lines of R code to draw a box-plot for day of week analysis, this data set with a third part statistical package will fill the gap until the Fitbit App offers something more sophisticated .



Logistic Regression – from Nspire to R to Theano

Logistic regression is a very powerful tool for classification and prediction. It works very well with linearly separable problem. This installment will attempt to recap on its practical implementation, from traditional perspective by maximum likelihood, to more machine learning approach by neural network, as well as from handheld calculator to GPU cores.

The heart of the logistic regression model is the logistic function. It takes in any real value and return value in the range from 0 to 1. This is ideal for binary classifier system. The following is a graph of this function.

TI Nspire

In the TI Nspire calculator, logistic regression is provided as a built-in function but is limited to single variable. For multi-valued problems, custom programming is required to apply optimization techniques to determine the coefficients of the regression model. One such application as shown below is the Nelder-Mead method in TI Nspire calculator.

Suppose in a data set from university admission records, there are four attributes (independent variables: SAT score, GPA, Interview score, Aptitude score) and one outcome (“Admission“) as the dependent variable.

Through the use of a Nelder-Mead program, the logistic function is first defined as l. It takes all regression coefficients (a1, a2, a3, a4, b), dependent variable (s), independent variables (x1, x2, x3, x4), and then simply return the logistic probability. Next, the function to optimize in the Nelder-Mead program is defined as nmfunc. This is the likelihood function on the logistic function. Since Nelder-Mead is a minimization algorithm the negative of this function is taken. On completion of the program run, the regression coefficients in the result matrix are available for prediction, as in the following case of a sample data with [GPA=1500, SAT=3, Interview=8, Aptitude=60].



In R, as a sophisticated statistical package, the calculation is much simpler. Consider the sample case above, it is just a few lines of commands to invoke its built-in logistic model.



Apart from the traditional methods, modern advances in computing paradigms made possible neural network coupled with specialized hardware, for example GPU, for solving these problem in a manner much more efficiently, especially on huge volume of data. The Python library Theano is a complex library supporting and enriching these calculations through optimization and symbolic expression evaluation. It also features compiler capabilities for CUDA and integrates Computer Algebra System into Python.

One of the examples come with the Theano documentation depicted the application of logistic regression to showcase various Theano features. It first initializes a random set of data as the sample input and outcome using numpy.random. And then the regression model is created by defining expressions required for the logistic model, including the logistic function and likelihood function. Lastly by using the theano.function method, the symbolic expression graph coded for the regression model is finally compiled into callable objects for the training of neural network and subsequent prediction application.


A nice feature from Theano is the pretty printing of the expression model in a tree like text format. This is such a feel-like-home reminiscence of my days reading SQL query plans for tuning database queries.