Entropy, used in machine learning, for layman

The measure/value of information in a dataset is information gain or entropy (Shannon entropy). This concept is widely used in building decision trees (used in machine learning).

What is this entropy?
As the name suggests, its the disturbance in the dataset. Lets say we have a dataset which has few attributes and one of these attributes is "clicksource". It can have one of either "mobile" or "desktop" values (in reality it can have more than 2 but for simplicity sake we will consider it can have one of these 2 values).

Now lets assume we have 150 records. We will take below examples to explain about the entropy and then will see the mathematical formula for this.

Example 1 
There are 75 records which have "mobile" value for clicksource attribute.

Rest 75 records have "desktop" values for same attribute.

Question -> Is this uniform data w.r.t. clicksource? Can this further be classified/split based upon clicksource?

Example 2

There are 100 records which have "mobile" value for clicksource attribute.

Rest 50 records have "desktop" values for same attribute.

Question -> Does this dataset has better uniformity (for the lack of better term) than in example 1? If yes then why?

Example 3

There are 140 records which have "mobile" value for clicksource attribute.

Rest 10 records have "desktop" values for same attribute.

Question -> Does this dataset has better uniformity than in example 1 and 2? If yes then why?

When you think about these, you realize that uniformity is more in dataset in example 3 than in previous examples. Reason, most (app. 93%) records are from same class (or category) of "mobile" for "clicksource" attribute. If there is more uniformity than it has less distrubance, in other words it has less entropy.

Again, In other words, we can say that if we can NOT further classify the dataset then we have zero entropy. But if we can then entropy is greater than zero. If we have entropy as 1 then the dataset is random. Are these fair statements to make?

Mathematical representation
===========================
Now lets see how can we represent entropy mathematically , the fun part.

As per ID3 algorithm (and try to correlate this with examples above)

Collection = S (150 in above examples)
p(C) = Proportion of S belonging to class C (in above examples class is "mobile" or "desktop")

Entropy of S = Sum of ( - p(C) log2 p(C) )

Why do we have log2 (log of base 2), we will discuss later.

Example 1 entropy
===================
p(C) for "mobile" = 75/150 = 0.5
p(C) for "desktop" = 75/150 = 0.5

(hint = log of base 2 of 0.5 is equal to -1)

Entropy = -0.5 * ( log2 0.5) - 0.5 (log2 0.5)
= -0.5 * -1 - 0.5 * -1
= 0.5 + 0.5 = 1

Which is right as the dataset is random having equally all (in this case 2 , mobile and desktop) values of attributes.

Example 2 entropy
===================
p(C) for "mobile" = 100/150 = 0.67
p(C) for "desktop" = 50/150 = 0.33

(hint = log of base 2 of 0.67 is equal to -0.58 , log of base 2 of 0.33 is -1.6)

Entropy = -0.67 * ( log2 0.67) - 0.33 (log2 0.33)
= -0.67 * -0.58 - 0.33 * -1.6
= 0.39 + 0.53 = ~0.9

Has less entropy than first but still ...

Example 3 entropy
===================
p(C) for "mobile" = 140/150 = 0.93
p(C) for "desktop" = 10/150 = 0.07

(hint = log of base 2 of 0.93 is equal to -0.105 , log of base 2 of 0.07 is -3.8)

Entropy = -0.93 * ( log2 0.93) - 0.07 (log2 0.07)
= -0.93 * -0.105 - 0.07 * -3.8
= 0.098 + 0.26 = ~0.37

Much less entropy.

Lets say all records are having "mobile" value for clicksource attribute. The entropy will be? Make a guess.
Here it is, Log of base 2 for 1 (i.e. 150/150) is 0 . Hence entropy is zero. Fully uniformed records i.e. belonging to just 1 class.

Now why base 2
Last line explained it, log of base 2 for 1 is zero, so log of base 2 is right factor for this measurement :) .

Cloudera VM setup on VMPlayer, to develop hadoop applications

Setup for Cloudera VM
======================

a) download VM player from VMWare site
1. https://my.vmware.com/web/vmware/downloads
2. select "Download" opposite to "VMware Player". It is almost at the end of the page.
3. Now select the VMPlayer for your operating system.
b) cloudera standalone VM for VMPlayer
1. download it from -> http://www.cloudera.com/content/support/en/downloads/quickstart_vms/cdh-5-1-x1.html
1.1 Select version CDH 5.1x, if not already selected.
1.2 Download for VMWare.
2. open it using winzip or 7zip or any other unzip utility you might have.
2.1 remember the location where it is unzipped
3. Open VMWare player (Downloaded in a) above)
4. From VM player window, click on "Open virtual machine" from right hand pane.
5. Select the location in 2.1 above.
6. Now VM Player has opened the VM
c) VMPlayer playing the VM
d) change VM memory or storage setting by right clicking on the VM name on VMPlayer and by selecting "settings"
d.1) select "hard disk" from "hardware" tab and expand to allocate more space.
d.1.1) if you have space then give more than 100GB to this, if you are trying to process huge data.
d.2) select "memory" from "hardware" and can change the  memory settings
d.2.1) Give 2GB or more.
d) For this VM, user id and password both are cloudera
e) run some hdfs commands
- hadoop fs -ls  /
- user directory in hdfs is  /user/cloudera
- fs -ls /user/cloudera
- . (dot) in hdfs command for a directory will translate to this directory (/user/cloudera)
- upload a file to hdfs
- on linux console type this -> touch test
- issue this command (without quotes) - "hadoop fs -put test . "
- above and  "hadoop fs -put test /user/cloudera/" mean the same thing


f) hadoop installation is at -> /etc/hadoop/conf
g) type "hadoop version" to verify the version of hadoop
h) Download netbeans (make sure you have allocated equal or more than 2GB memory to this VM)
- Open firefix browser from within the VM
- if you are not able to connect to internet then a) make sure your net connection is woking and
b) right clict at top rigt corner of the VM and change "network adapter" settings from "hardware" tab
Change for the right network settings. Mostly "Bridged" network connection is needed.
- Go to neatbeans.org
- click on downloads
- on the next page, click on "Download" under Java SE
- save the file
- once download finishes, go to this directory /home/cloudera/Downloads
- cd /home/cloudera/Downloads
- to execute the script, run the below command
- chmod +x netbeans-8.0.1-javase-linux.sh
- start installing netbeans, run the below command
- ./netbeans-8.0.1-javase-linux.sh
- accept all the defaults and keep clicking "next" and at the end "install"

Now you are ready to develop big data processing programs.

virtual machine in VMPlayer not connecting to internet

 use bridged connection then click on network adapter, uncheck the network adapters which are not needed. Select only needed network adapters..

Hadoop 2 setup on local environment

Short notes

- Follow http://raseshmori.wordpress.com/2012/10/14/install-hadoop-nextgen-yarn-multi-node-cluster/
- enable ssh to localhost
- install open ssh client etc
- change mapredude.shuffle to mapreduce_shuffle in the xml file in abive blog.
   - http://www.thecloudavenue.com/2012/01/getting-started-with-nextgen-mapreduce.html
- change all master: hosts in all *.xml files etc to locahost
 e.g

    yarn.resourcemanager.resource-tracker.address
    localhost:8031
 

 
    yarn.resourcemanager.scheduler.address
    localhost:8030
 
 
    yarn.resourcemanager.address
    localhost:8040
 

gstreamer java enable debuggig

Many people find it difficult to debug gstreamer pipeline written in gstreamer java. Only thing you need to do it to set GST_DEBUG environment variable e.g. on linux boxes it can be set by export GST_DEBUG=3 (r whatever debug level you want).

After this is set the just run the java application pipeline. You will see all debug messages printed. HTH.

Audio over RTP(real time protocol) using Gstreamer java

Just a test code. If it does something bad..don blame me :) .

Port used is 5000

package com.xxxxxxxxxxxxxx;

import java.util.logging.Logger;

import org.gstreamer.Bus;
import org.gstreamer.Element;
import org.gstreamer.Element.PAD_ADDED;
import org.gstreamer.ElementFactory;
import org.gstreamer.Gst;
import org.gstreamer.GstObject;
import org.gstreamer.Pad;
import org.gstreamer.Pipeline;
import org.gstreamer.State;
import org.gstreamer.elements.good.RTPBin;
import org.gstreamer.lowlevel.MainLoop;

/**
 * Sends audio over rtp
 *
 * use this command to test on remote host
 *      gst-launch-0.10 -v udpsrc port=5000 ! "application/x-rtp,media=(string)audio, clock-rate=(int)44100, width=16, height=16, encoding-name=(string)L16, encoding-params=(string)1, channels=(int)1, channel-positions=(int)1, payload=(int)96" ! rtpL16depay ! audioconvert ! autoaudiosink
 *
 * @author skhurana
 *
 */
public class HWRtp {

    private static Logger logger = Logger.getLogger("HW") ;
    private static final String REMOTE_HOST = "hostIPAddressWhereStreamingWillHappen" ;
   
    public static void main(String[] args) {
        final MainLoop loop = new MainLoop();
        args = Gst.init("RtpUtil", args);
        logger.info("Gstreamer version is " + Gst.getVersionString());
        Element source, demuxer,  conv, sink ;
        final Element decoder ;
        Bus bus;
        Pipeline pipeline  ;
       
        pipeline = new Pipeline("audio-player") ;
        source = ElementFactory.make("filesrc", "file-source") ;
        demuxer = ElementFactory.make("oggdemux", "ogg-demuxer") ;
        decoder = ElementFactory.make("vorbisdec", "vorbis-decoder") ;
        // audioconvert - Convert audio to different formats.
        conv = ElementFactory.make("audioconvert", "converter") ;
       
       
        // setup the pipeline
        source.set("location", "/home/skhurana/rtx/mediacontent/test.ogg") ;
        // add message handler
        //TODO - Do something with loop here
        bus = pipeline.getBus() ;
        bus.connect(new Bus.EOS() {
            public void endOfStream(GstObject source) {
                logger.info(" End of stream reached, source is...." + source.getName()) ;
                loop.quit() ;
                loop.dispose();
            }
        }) ;
       
        bus.connect(new Bus.ERROR() {
            public void errorMessage(GstObject source, int code, String message) {
                // TODO Auto-generated method stub
                logger.info(" Error occurred, code is " + code + ". Message is " + message) ;
            }
        }) ;
       
        // do rtp stuff now
        // ************* rtp stuff starts
        // create rtpbin
        RTPBin rtpbin = new RTPBin("audio-rtp") ;
        // create udpsink - send data over network via udp
        Element udpSink_rtpout = ElementFactory.make("udpsink", "udpsink0") ;
        udpSink_rtpout.set("host", REMOTE_HOST) ;
        udpSink_rtpout.set("port", 5000) ;
       
        // rtpL16pay - payload encode raw audio over rtp packets
        Element rtpL16pay = ElementFactory.make("rtpL16pay", "rtpPay") ;
        // ************** rtp stuff ends
       
        // add all elements into a pipeline
        pipeline.addMany(source, demuxer, decoder, conv, rtpL16pay, rtpbin, udpSink_rtpout) ;
       
       
        // link the elements together
        Element.linkMany(source, demuxer) ;
        Element.linkMany(decoder, conv, rtpL16pay) ;
        // link to rtp
        logger.info("Linking pads ....") ;
        Element.linkPads(rtpL16pay, "src", rtpbin, "send_rtp_sink_0") ;
        Element.linkPads(rtpbin, "send_rtp_src_0", udpSink_rtpout, "sink") ;
       
       
        demuxer.connect(new PAD_ADDED() {
            public void padAdded(Element element, Pad pad) {
                logger.info("Dynamic pad added, linking demuxer/decoder") ;
                Pad sinkPad = decoder.getStaticPad("sink") ;
                Element.linkPads(element, pad.getName(), decoder, sinkPad.getName()) ;
            }
        }) ;
       
       
        // note that demuxer will be linked to decoder dynamically. The reason is that
        // Ogg may contain various streams e.g. audio and video.
        // The source pad(s) will be created at run time, by the demuxer when it detects
        // the amount and nature of streams.
       
        // set the pipeline to playing state
        logger.info(" pipeline now playing....and ") ;
        pipeline.setState(State.PLAYING) ;
        logger.info(" Pipeline state is " + pipeline.getState()) ;
       
       
        loop.run() ;
        //TODO - What do I do here ?
        logger.info(" returned ..stopping playback") ;
        pipeline.setState(State.NULL) ;
       
       
       
    }
}

Command Pattern

Why this pattern has name as 'Command' ? Because someone gives a command and not care how..who will do it...