Perl Script for Feature Selection

Applied Math & Computer Science Lab

Data Analysis, Optimization & Mathematical Modeling, Artificial Intelligence, Neural Net For Everyday Life Applications

AI/Data Mining Links

Online Free Courses

Feature Subset Selection with Perl

Introduction

Feature selection is useful for learning relevant features from data. Many different algorithms exist for this technique. We will consider numerical example and programming in perl for feature subset selection using sequential forward algorithm. This algorithm search through feature space for good features. Thus it is filter type algorithm. The name of algorithm indicates how the search is implemented: it's start from empty set and add features one by one. There are also different ways how to measure goodness of selected feature subset. We will use correlation measure.

Numerical Example

The following artificial dataset is created for this example: The numbers in the first 4 columns are choosen at random, the last column is the class label, it's calculated based on the first and third column by following formula:
y = 2 IF x₀ + x₂ > 5
y = 1 IF x₀ + x₂ <= 5

The index here starts from 0 as the program also counts from 0. The index 0 means first column from the left
This dataset is saved in the txt file , space is separator between columns and the program will open this file and load data to memory for processing.
Let say we don't know how is obtained last column and want to detect features that are relevant to this column.

Perl Script for Feature Subset Selection

The script first does several passes through dataset and calculates correlation coefficients for all pairs of features including last column. Then it starts looking for features. In the beginning the subset of features is empty. In the first step we are looking for one feature that make max our goodness criteria. We add this feature to our subset of features. In the program we use string of column indexes separated by "_" to track selected features.
After first pass we are doing second step. In this pass we again choose one by one all features except the one we aready choosed in the previous step and calculate the goodness of 2 features sets , and keep the record of best feature.In the end of this pass we select the best feature and update our record for feature subset.
In the similar way we are doing next steps. For example in the next step we add one by one any remaining feature to our selected 2 feature subset. This gives us 3 feature subset and we calculate the goodness and again select the best set in the end.

Program Output

The program also output some temp. result. So if we run for our artificial example we will get the following output.

A:e>perl fe3.cgi
r[0][0] =  1
r[0][1] =  -0.149071198499986
r[0][2] =  0.396508095136578
r[0][3] =  -0.43019774330758
r[0][4] =  0.588119693312978
r[1][0] =  -0.149071198499986
r[1][1] =  1
r[1][2] =  0.329911695183279
r[1][3] =  -0.283811825938196
r[1][4] =  0.288675134594813
r[2][0] =  0.396508095136578
r[2][1] =  0.329911695183279
r[2][2] =  1
r[2][3] =  -0.378875334004775
r[2][4] =  0.83622997766139
r[3][0] =  -0.43019774330758
r[3][1] =  -0.283811825938196
r[3][2] =  -0.378875334004775
r[3][3] =  1
r[3][4] =  -0.338240712601273
r[4][0] =  0.588119693312978
r[4][1] =  0.288675134594813
r[4][2] =  0.83622997766139
r[4][3] =  -0.338240712601273
r[4][4] =  1

+++++++++++++++++++++++++++++++++++++++++++++++++
 z=0 v_string =  _2_  T_max=  0.83622997766139
+++++++++++++++++++++++++++++++++++++++++++++++++

+++++++++++++++++++++++++++++++++++++++++++++++++
 z=1 v_string =  _2_0_  T_max=  3.59223352169834
+++++++++++++++++++++++++++++++++++++++++++++++++

+++++++++++++++++++++++++++++++++++++++++++++++++
 z=2 v_string =  _2_0_1_  T_max=  1.95664470273798
+++++++++++++++++++++++++++++++++++++++++++++++++

+++++++++++++++++++++++++++++++++++++++++++++++++
 z=3 v_string =  _2_0_1_3_  T_max=  0.698435750257981
+++++++++++++++++++++++++++++++++++++++++++++++++

References

1. Feature subset selection-perl script (Source code)
2. Feature selection-perl script (Single variable ranking)
3. Feature selection - online demo

External Links
Feature Selection for High-Dimensional Data: A Fast Correlation-Based Filter Solution, Lei Yu, Huan Liu