Free domain name

http://www.dot.tk/en/index.html?lang=en

Linux Email server

http://www.iredmail.org/

Git timeout settings of 10 mins

Many folks would have faced this problem that git command times out (after 10 mins) . In linux (centos) this timeout setting can be changed by editing the file

• " sudo vim /etc/init.d/jenkins" 
• Add -Dorg.jenkinsci.plugins.gitclient.Git.timeOut=60 to JAVA_CMD 
• JAVA_CMD might look like

 JAVA_CMD="$JENKINS_JAVA_CMD $JENKINS_JAVA_OPTIONS -Dorg.jenkinsci.plugins.gitclient.Git.timeOut=60 -DJENKINS_HOME=$JENKINS_HOME -jar $JENKINS_WAR


Start jenkins service and it will not time out till 60 mins.  The time 60 is in minutes.

Entropy, used in machine learning, for layman

The measure/value of information in a dataset is information gain or entropy (Shannon entropy). This concept is widely used in building decision trees (used in machine learning).

What is this entropy?
As the name suggests, its the disturbance in the dataset. Lets say we have a dataset which has few attributes and one of these attributes is "clicksource". It can have one of either "mobile" or "desktop" values (in reality it can have more than 2 but for simplicity sake we will consider it can have one of these 2 values).

Now lets assume we have 150 records. We will take below examples to explain about the entropy and then will see the mathematical formula for this.

Example 1 
There are 75 records which have "mobile" value for clicksource attribute.

Rest 75 records have "desktop" values for same attribute.

Question -> Is this uniform data w.r.t. clicksource? Can this further be classified/split based upon clicksource?

Example 2

There are 100 records which have "mobile" value for clicksource attribute.

Rest 50 records have "desktop" values for same attribute.

Question -> Does this dataset has better uniformity (for the lack of better term) than in example 1? If yes then why?

Example 3

There are 140 records which have "mobile" value for clicksource attribute.

Rest 10 records have "desktop" values for same attribute.

Question -> Does this dataset has better uniformity than in example 1 and 2? If yes then why?

When you think about these, you realize that uniformity is more in dataset in example 3 than in previous examples. Reason, most (app. 93%) records are from same class (or category) of "mobile" for "clicksource" attribute. If there is more uniformity than it has less distrubance, in other words it has less entropy.

Again, In other words, we can say that if we can NOT further classify the dataset then we have zero entropy. But if we can then entropy is greater than zero. If we have entropy as 1 then the dataset is random. Are these fair statements to make?

Mathematical representation
===========================
Now lets see how can we represent entropy mathematically , the fun part.

As per ID3 algorithm (and try to correlate this with examples above)

Collection = S (150 in above examples)
p(C) = Proportion of S belonging to class C (in above examples class is "mobile" or "desktop")

Entropy of S = Sum of ( - p(C) log2 p(C) )

Why do we have log2 (log of base 2), we will discuss later.

Example 1 entropy
===================
p(C) for "mobile" = 75/150 = 0.5
p(C) for "desktop" = 75/150 = 0.5

(hint = log of base 2 of 0.5 is equal to -1)

Entropy = -0.5 * ( log2 0.5) - 0.5 (log2 0.5)
= -0.5 * -1 - 0.5 * -1
= 0.5 + 0.5 = 1

Which is right as the dataset is random having equally all (in this case 2 , mobile and desktop) values of attributes.

Example 2 entropy
===================
p(C) for "mobile" = 100/150 = 0.67
p(C) for "desktop" = 50/150 = 0.33

(hint = log of base 2 of 0.67 is equal to -0.58 , log of base 2 of 0.33 is -1.6)

Entropy = -0.67 * ( log2 0.67) - 0.33 (log2 0.33)
= -0.67 * -0.58 - 0.33 * -1.6
= 0.39 + 0.53 = ~0.9

Has less entropy than first but still ...

Example 3 entropy
===================
p(C) for "mobile" = 140/150 = 0.93
p(C) for "desktop" = 10/150 = 0.07

(hint = log of base 2 of 0.93 is equal to -0.105 , log of base 2 of 0.07 is -3.8)

Entropy = -0.93 * ( log2 0.93) - 0.07 (log2 0.07)
= -0.93 * -0.105 - 0.07 * -3.8
= 0.098 + 0.26 = ~0.37

Much less entropy.

Lets say all records are having "mobile" value for clicksource attribute. The entropy will be? Make a guess.
Here it is, Log of base 2 for 1 (i.e. 150/150) is 0 . Hence entropy is zero. Fully uniformed records i.e. belonging to just 1 class.

Now why base 2
Last line explained it, log of base 2 for 1 is zero, so log of base 2 is right factor for this measurement :) .

Cloudera VM setup on VMPlayer, to develop hadoop applications

Setup for Cloudera VM
======================

a) download VM player from VMWare site
1. https://my.vmware.com/web/vmware/downloads
2. select "Download" opposite to "VMware Player". It is almost at the end of the page.
3. Now select the VMPlayer for your operating system.
b) cloudera standalone VM for VMPlayer
1. download it from -> http://www.cloudera.com/content/support/en/downloads/quickstart_vms/cdh-5-1-x1.html
1.1 Select version CDH 5.1x, if not already selected.
1.2 Download for VMWare.
2. open it using winzip or 7zip or any other unzip utility you might have.
2.1 remember the location where it is unzipped
3. Open VMWare player (Downloaded in a) above)
4. From VM player window, click on "Open virtual machine" from right hand pane.
5. Select the location in 2.1 above.
6. Now VM Player has opened the VM
c) VMPlayer playing the VM
d) change VM memory or storage setting by right clicking on the VM name on VMPlayer and by selecting "settings"
d.1) select "hard disk" from "hardware" tab and expand to allocate more space.
d.1.1) if you have space then give more than 100GB to this, if you are trying to process huge data.
d.2) select "memory" from "hardware" and can change the  memory settings
d.2.1) Give 2GB or more.
d) For this VM, user id and password both are cloudera
e) run some hdfs commands
- hadoop fs -ls  /
- user directory in hdfs is  /user/cloudera
- fs -ls /user/cloudera
- . (dot) in hdfs command for a directory will translate to this directory (/user/cloudera)
- upload a file to hdfs
- on linux console type this -> touch test
- issue this command (without quotes) - "hadoop fs -put test . "
- above and  "hadoop fs -put test /user/cloudera/" mean the same thing


f) hadoop installation is at -> /etc/hadoop/conf
g) type "hadoop version" to verify the version of hadoop
h) Download netbeans (make sure you have allocated equal or more than 2GB memory to this VM)
- Open firefix browser from within the VM
- if you are not able to connect to internet then a) make sure your net connection is woking and
b) right clict at top rigt corner of the VM and change "network adapter" settings from "hardware" tab
Change for the right network settings. Mostly "Bridged" network connection is needed.
- Go to neatbeans.org
- click on downloads
- on the next page, click on "Download" under Java SE
- save the file
- once download finishes, go to this directory /home/cloudera/Downloads
- cd /home/cloudera/Downloads
- to execute the script, run the below command
- chmod +x netbeans-8.0.1-javase-linux.sh
- start installing netbeans, run the below command
- ./netbeans-8.0.1-javase-linux.sh
- accept all the defaults and keep clicking "next" and at the end "install"

Now you are ready to develop big data processing programs.

virtual machine in VMPlayer not connecting to internet

 use bridged connection then click on network adapter, uncheck the network adapters which are not needed. Select only needed network adapters..

Hadoop 2 setup on local environment

Short notes

- Follow http://raseshmori.wordpress.com/2012/10/14/install-hadoop-nextgen-yarn-multi-node-cluster/
- enable ssh to localhost
- install open ssh client etc
- change mapredude.shuffle to mapreduce_shuffle in the xml file in abive blog.
   - http://www.thecloudavenue.com/2012/01/getting-started-with-nextgen-mapreduce.html
- change all master: hosts in all *.xml files etc to locahost
 e.g

    yarn.resourcemanager.resource-tracker.address
    localhost:8031
 

 
    yarn.resourcemanager.scheduler.address
    localhost:8030
 
 
    yarn.resourcemanager.address
    localhost:8040
 

gstreamer java enable debuggig

Many people find it difficult to debug gstreamer pipeline written in gstreamer java. Only thing you need to do it to set GST_DEBUG environment variable e.g. on linux boxes it can be set by export GST_DEBUG=3 (r whatever debug level you want).

After this is set the just run the java application pipeline. You will see all debug messages printed. HTH.

Audio over RTP(real time protocol) using Gstreamer java

Just a test code. If it does something bad..don blame me :) .

Port used is 5000

package com.xxxxxxxxxxxxxx;

import java.util.logging.Logger;

import org.gstreamer.Bus;
import org.gstreamer.Element;
import org.gstreamer.Element.PAD_ADDED;
import org.gstreamer.ElementFactory;
import org.gstreamer.Gst;
import org.gstreamer.GstObject;
import org.gstreamer.Pad;
import org.gstreamer.Pipeline;
import org.gstreamer.State;
import org.gstreamer.elements.good.RTPBin;
import org.gstreamer.lowlevel.MainLoop;

/**
 * Sends audio over rtp
 *
 * use this command to test on remote host
 *      gst-launch-0.10 -v udpsrc port=5000 ! "application/x-rtp,media=(string)audio, clock-rate=(int)44100, width=16, height=16, encoding-name=(string)L16, encoding-params=(string)1, channels=(int)1, channel-positions=(int)1, payload=(int)96" ! rtpL16depay ! audioconvert ! autoaudiosink
 *
 * @author skhurana
 *
 */
public class HWRtp {

    private static Logger logger = Logger.getLogger("HW") ;
    private static final String REMOTE_HOST = "hostIPAddressWhereStreamingWillHappen" ;
   
    public static void main(String[] args) {
        final MainLoop loop = new MainLoop();
        args = Gst.init("RtpUtil", args);
        logger.info("Gstreamer version is " + Gst.getVersionString());
        Element source, demuxer,  conv, sink ;
        final Element decoder ;
        Bus bus;
        Pipeline pipeline  ;
       
        pipeline = new Pipeline("audio-player") ;
        source = ElementFactory.make("filesrc", "file-source") ;
        demuxer = ElementFactory.make("oggdemux", "ogg-demuxer") ;
        decoder = ElementFactory.make("vorbisdec", "vorbis-decoder") ;
        // audioconvert - Convert audio to different formats.
        conv = ElementFactory.make("audioconvert", "converter") ;
       
       
        // setup the pipeline
        source.set("location", "/home/skhurana/rtx/mediacontent/test.ogg") ;
        // add message handler
        //TODO - Do something with loop here
        bus = pipeline.getBus() ;
        bus.connect(new Bus.EOS() {
            public void endOfStream(GstObject source) {
                logger.info(" End of stream reached, source is...." + source.getName()) ;
                loop.quit() ;
                loop.dispose();
            }
        }) ;
       
        bus.connect(new Bus.ERROR() {
            public void errorMessage(GstObject source, int code, String message) {
                // TODO Auto-generated method stub
                logger.info(" Error occurred, code is " + code + ". Message is " + message) ;
            }
        }) ;
       
        // do rtp stuff now
        // ************* rtp stuff starts
        // create rtpbin
        RTPBin rtpbin = new RTPBin("audio-rtp") ;
        // create udpsink - send data over network via udp
        Element udpSink_rtpout = ElementFactory.make("udpsink", "udpsink0") ;
        udpSink_rtpout.set("host", REMOTE_HOST) ;
        udpSink_rtpout.set("port", 5000) ;
       
        // rtpL16pay - payload encode raw audio over rtp packets
        Element rtpL16pay = ElementFactory.make("rtpL16pay", "rtpPay") ;
        // ************** rtp stuff ends
       
        // add all elements into a pipeline
        pipeline.addMany(source, demuxer, decoder, conv, rtpL16pay, rtpbin, udpSink_rtpout) ;
       
       
        // link the elements together
        Element.linkMany(source, demuxer) ;
        Element.linkMany(decoder, conv, rtpL16pay) ;
        // link to rtp
        logger.info("Linking pads ....") ;
        Element.linkPads(rtpL16pay, "src", rtpbin, "send_rtp_sink_0") ;
        Element.linkPads(rtpbin, "send_rtp_src_0", udpSink_rtpout, "sink") ;
       
       
        demuxer.connect(new PAD_ADDED() {
            public void padAdded(Element element, Pad pad) {
                logger.info("Dynamic pad added, linking demuxer/decoder") ;
                Pad sinkPad = decoder.getStaticPad("sink") ;
                Element.linkPads(element, pad.getName(), decoder, sinkPad.getName()) ;
            }
        }) ;
       
       
        // note that demuxer will be linked to decoder dynamically. The reason is that
        // Ogg may contain various streams e.g. audio and video.
        // The source pad(s) will be created at run time, by the demuxer when it detects
        // the amount and nature of streams.
       
        // set the pipeline to playing state
        logger.info(" pipeline now playing....and ") ;
        pipeline.setState(State.PLAYING) ;
        logger.info(" Pipeline state is " + pipeline.getState()) ;
       
       
        loop.run() ;
        //TODO - What do I do here ?
        logger.info(" returned ..stopping playback") ;
        pipeline.setState(State.NULL) ;
       
       
       
    }
}

Command Pattern

Why this pattern has name as 'Command' ? Because someone gives a command and not care how..who will do it...

How To Become A Hacker

How To Become A Hacker

How To Become A Hacker

How To Become A Hacker

Visitor Pattern

Why - As per wikipedia it is "is a way of separating an algorithm from an object structure it operates on".

Lets discuss with an example

- object structure - e.g. getters and setters of a class with attributes interest, principle.
(i.e. getInterest(), getPrinciple()). Lets say the class name having these getters and setters is 'Amount'.

- algorithm to calculate the interest in a class InterestCalculator. While getting the interest from 'Amount', InterestCalculator.calculate needs to be called.

- Amount is Visitable (Visitable is interface which has 'getInterest' method)

class Amount implements Visitable {

getInterest(Visitor visitor) {
visitor.visit(this) ;
}

}

- InterestCalculator is 'Visitor' (Visitor is an interface which have visit method)

class InterestCalculator implements Visitor {

visit(Amount amount) {
// calculation goes here <<<<<<<------------
}
}

- The client (e.g. main method) has to know which visitor (i.e. intetestcalculator) to inject into the accept method.

It invokes it like
Amount a = new Amount() ;
// set values in 'a' object e.g. Principal
a.getInterest(new InterestCalculator)

helios emf


Helios In Action: Modeling from Lynn Gayowski on Vimeo.

XQuery

Sometimes when u have a big XML document, you want to iterate through some elements of it.  One way of doing it is to parse it and process the needed elements. Another way might be to use XQuery/XPath. You can directly access the needed elements/attributes and their values using XQuery. 

I present an example which uses Saxon s9api. Include saxon and saxon's s9api jar in the classpath. Consider this xml string  -> 

" Abiteboul "

Suppose if we want to know the years of the book (imagine there are 100s of 1000s books in the xml, though this xml shows just 1 book). We can use the follwing java code.

Processor processor = new Processor(false);

ItemTypeFactory itf = new ItemTypeFactory(processor);

ItemType integerType = itf.getAtomicType(new QName("http://www.w3.org/2001/XMLSchema", "integer"));

DocumentBuilder builder = processor.newDocumentBuilder();

        builder.setLineNumbering(true);

        builder.setWhitespaceStrippingPolicy(WhitespaceStrippingPolicy.ALL);

        String str = " Abiteboul ";

        StringValue strVal = new StringValue(str.subSequence(0, str.length()));

        InputStream bais = new ByteArrayInputStream(str.getBytes());

        StreamSource ss = new StreamSource(bais);

XdmNode fileNode = builder.build(ss);

XPathCompiler xpc = processor.newXPathCompiler();

XPathExecutable xqe = xpc.compile("/BOOKS/BOOK/@YEAR");

XPathSelector selector = xqe.load();

selector.setContextItem(fileNode);

XdmValue val = selector.evaluate();

XdmSequenceIterator itr = val.iterator();

while(itr.hasNext()) {

XdmItem xi = itr.next();

XdmNode xn = (XdmNode) xi;

System.out.println(" Node Name " + xn.getNodeName().getLocalName());

System.out.println(" Node Value " + xn.getStringValue());

}

Note - We can read an xml saved in a file too in addition to xml in string variable as in above example.

JWS

New spec of JWS uses JAXB 2.0 for JAVA/XML binding. The binding info can be provided in form of annotations on java classes or in external binding file. If annotations have this info then it will be used during deployment and run time. But if the binding information is in external file then it will be used only during deployment time and not during execution. 

One good feature of this JWS spec is that if some info is not provided on java classes for exposing them as web service then it will assume defaults. 

JPA One to Many query

Assume there is a parent entity and has one to many relationship with a child entity. We want to write a query where in a record of parent entity is comapred against each row of its children, we can IN operator.For example 

SELECT e.name, d.name from Department d, IN(d.employees) e where e.salary > 100000

Size in JPA

JPA has a very nice feature wherein we can get the size of rows in a child table.  The EJB QL will be something like this

Select r from Department d where size(d.employees)  > 100

 

Developer..

Copied from http://zuru.sufx.net/dev/developers.jpg

Choosing a technology for enterprise level application development

At the start of the many projects, the teams and business people face the problem of selecting the best technology for the project. There are various factors which need to be considered while selecting any technology. I had been part of many such teams.
Couple of teams decided to go for technology which I would never recommend .

In one such case, Oracle form was selected to develop the project. The only reason it was selected because it provides quickest way to develop UI. But that not the way to develop an enterprise level project. One has to see the maintenance cost, durability, support for the tech from the vendor, longevity of the project etc.

In another project HTML DB was selected. Everything works fine and application goes into production. But the nightmares start when customers start asking for changes, enhancements etc. Since in such tech. the code (mainly SQL and procedural code) is scattered all over the places the changes become headaches. One example is that the team needed to refactor(or normalize) the tables but this idea has to be dropped as the tables were used at many places in SQLs and SQLs were scattered all over the places. Even after changing the testing would have been almost impossible.

The solution is to develop the systems in OO languages and the one which provides total ecosystem (from ui, server code etc till deployment phase) for enterprise development.

J2EE (now JEE) perfectly fits the bill. Many advantages go in favor of JEE tech stack trace. First, it is based upon open standards. If you feel like improving something in it, you can be part of JCP. There is a huge open source community behind it along with well established vendors like IBM, Oracle, SUN etc. It is not dependent upon the whims of one big corporation which will do anything to kill the competition and then later will force its customer to pay heavy license fees and/or buy upgraded expensive versions. J2EE tech provides many frameworks to make the system stable, robust, easy to change,easy to monitor etc (junit, logging, continuous integration etc, mock objects etc). All this adds up to good ROI and feel good and confident factor for the customer.