Keeping Forward

RabbIT

2011-12-12T13:39:00.001-05:00

Enable filter rabbit.filter.ReverseProxy, by adding it to the front of httpinfilters value.

Configure values for sector rabbit.filter.ReverseProxy.

Run RabbIT:

java -jar jars/rabbit4.jar -f conf/rabbit.conf

Resource: http://www.khelekore.org/rabbit/

Memory allocation settings in Hadoop

2011-12-02T17:31:00.001-05:00

Edit file conf/mapred-site.xml to change amount of memory allocated to sorting:

<property> 
    <name>io.sort.mb</name>  
    <value>300</value>  
</property>

Edit file conf/mapred-site.xml to change amount of memory allocated to each map/reduce task:

<property>
    <name>mapred.child.java.opts</name>
    <value>-Xmx800m</value>  
</property>

Edit file conf/hadoop-env.sh to change amount of memory allocated to Hadoop daemons:

export HADOOP_HEAPSIZE=1000

Change ports used by Hadoop

2011-12-02T17:18:00.001-05:00

Edit file conf/hdfs-site.xml to change ports used by HDFS

    <property>
        <name>dfs.secondary.http.address</name>
        <value>0.0.0.0:51090</value>
    </property>
    <property>
        <name>dfs.datanode.address</name>
        <value>0.0.0.0:51010</value>
    </property>
    <property>
        <name>dfs.datanode.http.address</name>
        <value>0.0.0.0:51075</value>
    </property>
    <property>
        <name>dfs.datanode.https.address</name>
        <value>0.0.0.0:51475</value>
    </property>
    <property>
        <name>dfs.datanode.ipc.address</name>
        <value>0.0.0.0:51020</value>
    </property>
    <property>
        <name>dfs.http.address</name>
        <value>0.0.0.0:51070</value>
    </property>
    <property>
        <name>dfs.https.address</name>
        <value>0.0.0.0:51470</value>
    </property>

Edit file conf/mapred-site.xml o change ports used by MapReduce

    <property>
        <name>mapred.job.tracker.http.address</name>
        <value>0.0.0.0:51030</value>
    </property>

    <property>
        <name>mapred.task.tracker.http.address</name>
        <value>0.0.0.0:51060</value>
    </property>

exclude directories when using GNU tar

2011-12-02T16:53:00.001-05:00

tar zvcf name.tar.gz --exclude path/to/dir1 --exclude path/to/dir2 path/to/tar

Note:

Do not include a trailing '/' in the path of excluded directories. Otherwise, it won't work.
Put --exclude before the directory/file to be tarred.

Install RPM packages with non-root account

2011-11-18T23:46:00.001-05:00

Recently, I need to install RPM to RedHat Linux, but I don't have root access. I found this post: http://ajaya.name/?p=6353. However, some commands in the command are not correct or need more clarification. So I wrote down my experience below.

create file ~/.rpmmacros and add following line
%_rpmlock_path lib/rpm/__db.000
Initialize the database by running command:

rpm --initdb \

    --root /home/<user_name>/rpm-local/ \

    --dbpath /home/<user_name>/rpm-local/lib/rpm

Check the dependence

rpm --root /home/<user_name>/rpm-local/ \
    --dbpath /home/<user_name>/rpm-local/lib/rpm \
    -ivh package.rpm

Install package

rpm --root /home/<user_name>/rpm-local \
    --dbpath /home/<user_name>/rpm-local/lib/rpm \ 
    --relocate /usr=/home/<user_name>/rpm-local \
    --nodeps \
    -ivh package.rpm

Resources

Add "Edit with Vim" item to context-menu in Windows for Vim

2011-10-27T15:18:00.001-04:00

This post shows how to add "Edit with Vim" and "Tab Edit with Vim" items to context menu (pop out when you right click a file) in Windows.

Run regedit.exe, go to HKEY_LOCAL_MACHINE/SOFTWARE/Classes/*/shell/

Create new key "Tab Edit with &Vim" (Right click parent entry -> New -> Key)
Create new key "Edit with Vim"
Create new key "command" under "Tab Edit with &Vim"
Edit the entry with name "(Default)", change its data to
<vim_dir>\gvim.exe" -p --remote-tab-silent "%1" "%*"
Create new key "command" under "Edit with Vim".
Edit the entry with name "(Default)", change its data to
<vim_dir>\gvim.exe "%1"

It seems that latest versions of vim automatically create the registry entry:

HKEY_LOCAL_MACHINE\SOFTWARE\Vim\Gvim and the data of entry path points to the vim executable.
HKEY_CLASSES_ROOT\*\shellex\ContextmenuHandlers\gvim. Data of the entry "(Default)" is the CLSID (51EEE242-AD87-11d3-9C1E-0090278BBD99).
You can delete this entry because we have already added menu item "Edit with Vim".
HKEY_CLASSES_ROOT\Applications\gvim.exe\shell\edit\command. Data of the entry "(Default)" is "<vim_dir>\gvim.exe" "%1"
HKEY_CLASSES_ROOT\CLSID\{51EEE242-AD87-11d3-9C1E-0090278BBD99}
It has a key named InProcServer32. The data of "(Default)" is <vim_dir>\gvimext.dll
Some other keys

Logging in Hadoop

2011-10-17T18:03:00.000-04:00

Hadoop uses log4j via Apache common logging. The config file is conf/log4j.properties.

Some important variables are set in the command line. Following is a snippet cut from the whole command line used to launch HDFS name node.

-Dhadoop.root.logger=INFO,DRFA
-Dhadoop.log.dir=/N/u/hdfs/programs/hadoop-0.21.0/bin/../logs
-Dhadoop.log.file=hadoop-hdfs-namenode-b009.log
-Dhadoop.home.dir=/N/u/hdfs/programs/hadoop-0.21.0/bin/..
-Dhadoop.id.str=hdfs

You can see that log dir, log file, log level, logger are set. DRFA is defined in conf/log4j.properties.

pdf to eps conversion

2011-10-09T11:57:00.001-04:00

Currently, I need to convert pdf files to eps so that they can be included in latex files.

Use Acrobat Pro open the pdf file.
Click File -> export -> PostScript -> Encapsulated PostScript or use "Save As" and change "Save As Type"
However, the bounding box is NOT correctly calculated.
You can use gsview to correct it. Use gsview open the eps file, click "File -> PS to EPS", select "Automatically calculate Bounding Box" and save the output file.
Use ghostscript. Execute following command:
gswin32 -sDEVICE=epswrite -sOutputFile=<filename>.eps <filename>.pdf
This works well and bounding box is correctly calculated.
Use Xpdf (http://www.foolabs.com/xpdf/download.html)
pdftops -eps <filename>.pdf <filename>.eps
However, the bounding box is NOT correctly calculated.

Brief Latex notes for equations

2011-10-09T11:43:00.001-04:00

In-line: $…$

Single line, without equation number: \[ … \] or \begin{equation*} … \end{equation*}

Single line, with equation number: \begin{equation} … \end{equation}

Multi-line, without equation number: \begin{align*} ... \end{align*}

Multi-line, with equation number: \begin{align} ... \end{align}

"a double backslash (\\) is used to separate the lines, and an ampersand symbol (&) is used to indicate the place at which the formulas should be aligned."

For align, \label must be put in front of each equation. For equation, it does not matter you put it in front or in the end.

Make vim-latex to generate output in a specified directory

2011-10-08T12:02:00.001-04:00

Recently I started to use latex to write papers. I want to edit latex in my favorite editor - vim. I found the project vim-latex: http://vim-latex.sourceforge.net/. It is powerful and convenient to use.

However, one feature I want is to generate output files (.div, .ps, .log, etc) into a separate directory rather than the same directory as tex files. It turns out that vim-latex does not support it natively. So I hacked into the source code to make it work on Windows.

Edit file ~/vimfiles/ftplugin/tex.vim

Add following config:
```
  set iskeyword+=: 
  let g:Tex_Outdir='out' 
  let g:Tex_ViewRule_pdf='"Foxit Reader.exe" ' 
  let g:Tex_CompileRule_dvi='mkdir '.g:Tex_Outdir.' & latex -output-directory='.g:Tex_Outdir.' -src-specials --interaction=nonstopmode $*' 
  let g:Tex_CompileRule_pdf=g:Tex_CompileRule_dvi.' & cd '.g:Tex_Outdir.' & dvipdfm $*.dvi'
```
Basically above config specifies
1. output directory of latex compilation
2. How to view PDF files (if you don't give full path, the command needs to be in env variable PATH)
3. How to compile dvi: create the output directory and put output there
4. How to compile pdf: first compile tex to dvi, and then call dvipdfm to generate pdf
Change ~/vimfiles/ftplugin/latex-suite/compiler.vim

Change line 252 to (this line adds the full path of output directory):
let execString = 'start '.s:viewer.' "'.expand('%:p:h').'/'.g:Tex_Outdir.'/$*.'.s:target.'"'

Change line 405 to (this line adds the full path of output directory):
let execString = 'silent! !'.viewer.' "'.expand('%:p:h').'/'.g:Tex_Outdir.'/'.mainfnameRoot.'.'.s:target.'" '.line('.').' "'.expand('%').'"'
Note: the text in blue is what I added (diff against the original code)

Use command 'TTarget' to switch amont dvi, ps, pdf, etc.

How to publish artifacts to ivy local repository and use it in another project

2011-09-19T18:11:00.001-04:00

Build artifacts that are required by another project

ivy.xml: Attributes organisation, module and revision of element ivy-module/info.

1) Add a new resolver to ivy by adding following snippet to ivysettings.xml:

<filesystem name="gerald-local-ivy" m2compatible="false" force="false" local="true">
    <artifact pattern="${ivy.default.ivy.user.dir}/local/[organisation]/[module]/[type]s/[artifact]-[revision](-[classifier]).[ext]"/>
</filesystem>

This will create a directory local under <user.home>/.ivy2/.

2) Add following snippet to your build.xml:

<target name="publish" depends="jar" description="Publish">
    <!-- following property defines the version to publish -->
    <property name="ivy.deliver.revision" value="${version}"/>
    <ivy:publish resolver="gerald-local-ivy" forcedeliver="true"
    	 settingsRef="${ant.project.name}.ivy.settings" overwrite="true">
    	<artifacts pattern="${build.dir}/[artifact]-[revision](-[classifier]).[ext]" />
    </ivy:publish>
</target>

The resolver attribute of ivy:publish must match name attribute specified in step 1). Change pattern attribute for artifacts element to match where you put the artifact.

3) Use command ant publish to publish your jar.

Build main project that depends on the artifacts built above.

1) Add the same resolver to ivy by adding following snippet to ivysettings.xml:

<filesystem name="gerald-local-ivy" m2compatible="false" force="false" local="true">
    <artifact pattern="${ivy.default.ivy.user.dir}/local/[organisation]/[module]/[type]s/[artifact]-[revision](-[classifier]).[ext]"/>
</filesystem>

2) Add it to your effective resolver chain.

3) Add dependency declaration to ivy.xml

<dependency org="<organization>" name="<module>"

            rev="<version>" conf="<common->master>">

The attributes org, name, rev must match the values specified when you built the dependency jar.

4) Manually remove the artifacts that exist in local cache (<user.home>/.ivy2/cache).

5) Build your project

Resources

http://ant.apache.org/ivy/history/2.2.0/use/publish.html
http://ant.apache.org/ivy/history/latest-milestone/resolver/chain.html
http://stackoverflow.com/questions/353336/how-does-ivypublish-work
http://mail-archives.apache.org/mod_mbox/ant-ivy-user/201002.mbox/%3C27714488.post@talk.nabble.com%3E

Configure ivy to use local Maven repository

2011-09-18T18:58:00.001-04:00

This post shows how to configure ivy to use specific local Maven repository.

1) Add following config as a child element of tag "resolvers" in you ivy settings file:

    <filesystem name="local-maven-2" m2compatible="true" force="false" local="true">
       <artifact pattern="${gerald.repo.dir}/[organisation]/[module]/[revision]/[module]-[revision].[ext]"/>
       <ivy pattern="${gerald.repo.dir}/[organisation]/[module]/[revision]/[module]-[revision].pom"/>
    </filesystem>

2) Then add it to your chain resolver config. Example:

    <chain name="internal" dual="true">
      <resolver ref="local-maven-2"/>
      <resolver ref="apache-snapshot"/> 
      <resolver ref="maven2"/>
    </chain>

If you have multiple resolver chain, make sure that the correct one, which is effective for you build, is changed.

3) After ivy caches the artifacts in its own local repo (the first time the dependency is resolved), it will not pick changes you made to the artifacts in the original Maven repository. In other words, if you use "mvn clean install" to re-publish the artifact, the new version will NOT propagate to ivy.
You can change the default behavior by tweaking parameters:
checkmodified, changingPattern, changingMatcher, alwaysCheckExactRevision
Read this article for details: http://ant.apache.org/ivy/history/trunk/settings/resolvers.html
One example:

    <chain name="default" dual="true" 
	     checkmodified="true" changingPattern=".*SNAPSHOT">
     ......
    </chain>

Resources:

http://mail-archives.apache.org/mod_mbox/ant-ivy-user/200807.mbox/raw/%3C94bda3fa0807202250y446a818eodb527dba96c8ac93@mail.gmail.com%3E/

Vdbench notes

2011-09-13T00:11:00.001-04:00

Two types of supported testings. Following table shows the configuration groups for these testings.

Storage device testing	File system testing
General - optional	General - optional
Host Definition(HD) - optional	Host Definition (HD) - optional
Storage Definition(SD)	File System Definition (FSD)
Workload Definition(WD)	File System Workload Definition(FSWD)
Run Definition (RD)	Run Definition (RD)

Different configuration groups may be specified in one single file or multiple files (use multiple -f options to specify all of them).

Comments: a line starting with '/', '#', or '*' is a comment.
Continuation: end a line with a comma plus a whitespace.

Output file
-o dirname+ # if dirname has existed, use dirname001.
-o dirname.tod # the directory name will be dirname.yymmdd.hhmmss

General parameters

Host definition

Storage definition

Workload definition

Run definition

File System Definition

File System Workload Definition

Run definition for file system testing

Resources:
Official doc: http://downloads.sourceforge.net/project/vdbench/vdbench502.pdf?r=http%3A%2F%2Fsourceforge.net%2Fprojects%2Fvdbench%2Ffiles%2F&ts=1315925901&use_mirror=superb-sea2

R notes

2011-09-09T19:19:00.001-04:00

Construct a list of lists

a = c(1,2,3)
b = c(4,5,6)
c = list(a, b)
d = list(c, list(a)) # different from list(c, a)
e = list(d, name="gerald")

Append to a list:

mylist[[length(mylist) + 1]] = obj

Append to a vector:

append(vec, val)

Type information

typeof(var), class(var)

Resources

http://www.statmethods.net/input/datatypes.html

How to get network information in Linux

2011-09-05T16:35:00.001-04:00

Following files/dirs can be directly read:

/proc/net/dev
/sys/class/net/<if_name>/
/sys/class/net/<if_name>/statistics

Tool netstat is your friend to get network-related information:

netstat -I # display interface info
netstat -s # display statistics info
netstat -r # display routing info
netstat -tlnp # display TCP listening info

Other tools: sar, ifconfig, iftop

How to see network card and disk speed

2011-07-08T01:56:00.001-04:00

Sometimes, you may want to know the hardware speed limit for network interface cards and disks.

Network Interface Card

I use commands

dmesg | grep -i ethernet
dmesg | grep -i infiniband

The output looks like

Intel(R) Gigabit Ethernet Network Driver - version 2.1.0-k2-1
igb 0000:0b:00.0: Intel(R) Gigabit Ethernet Network Connection
igb 0000:0b:00.1: Intel(R) Gigabit Ethernet Network Connection

and

mlx4_ib: Mellanox ConnectX InfiniBand driver v1.0 (April 4, 2008)
Registered RDS/infiniband transport

I also use command

/sbin/lspci

The output has following useful information

0b:00.0 Ethernet controller: Intel Corporation 82575EB Gigabit Network Connection (rev 02)
0b:00.1 Ethernet controller: Intel Corporation 82575EB Gigabit Network Connection (rev 02)
10:00.0 InfiniBand: Mellanox Technologies MT26418 [ConnectX VPI PCIe 2.0 5GT/s - IB DDR / 10GigE] (rev a0)

Disk

/sbin/lspci | grep -i ata

Other useful tools

mii-tools
hdparm

They need root privilege usually.

Hadoop datanode version control

2011-05-12T10:12:00.000-04:00

Sometimes, when you upgrade your Hadoop, you may get following error in your namenode log:

Incompatible build versions: namenode BV = Unknown; datanode BV =

./common-0.21.0/src/saveVersion.sh generates package-info.java which includes version information. The content looks like

@HadoopVersionAnnotation(version="0.21.1-SNAPSHOT", revision="1", branch="",
                         user="username", date="Mon Nov 15 12:28:49 EST 2010",
                         url="your_domain/path",
                         srcChecksum="a1aeb15b4854808d152989ba76f90fac")
saveVersion.sh is executed when you build Hadoop using ant. It is specified in build.xml (target "init").

In Java code, class org.apache.hadoop.util.VersionInfo manages version. It gets version information from package-info.java generated by saveVersion.sh.

In class org.apache.hadoop.hdfs.server.datanode.DataNode, method handshake checks whether build version is equal. How build version is calculated is shown below.

public static String getBuildVersion(){
return VersionInfo.getVersion() +
" from " + VersionInfo.getRevision() +
" by " + VersionInfo.getUser() +
" source checksum " + VersionInfo.getSrcChecksum();
}
So, the quick solution is that you upgrade all installations of Hadoop on different nodes.

How to install user-provided jars to Hadoop

2011-05-12T04:16:00.000-04:00

If you write a MapReduce program and compile it to a jar, you usually run it with following command:

./bin/hadoop jar your_jar_name

If you want to your jar loaded when Hadoop is started (e.g. you add a new service which should be initiated and started by Hadoop), you can follow steps shown below.

In file bin/hadoop-config.sh, you can find following snippet:

for f in $HADOOP_COMMON_HOME/hadoop-*.jar; do
  CLASSPATH=${CLASSPATH}:$f;
done

So only jars whose names starting with "hadoop-" are loaded by default.

Drop your jar to the directory where Hadoop is used, and change file bin/hadoop-config.sh to add

CLASSPATH=${CLASSPATH}:$HADOOP_COMMON_HOME/your_jar_name

Debug/profile heap/gc in Java

2011-04-15T01:24:00.001-04:00

HPROF

Profiler agent.
Examples:
java -agentlib:hprof=help
java -agentlib:hprof=heap=sites
java -agentlib:hprof=heap=dump
java -agentlib:hprof=cpu=samples

" By default, heap profiling information (sites and dump) is written out to java.hprof.txt (in ASCII) in the current working directory.

The output is normally generated when the VM exits, although this can be disabled by setting the “dump on exit” option to “n” ( doe=n). In addition, a profile is generated when Ctrl-\ or Ctrl-Break (depending on platform) is pressed. On Solaris OS and Linux a profile is also generated when a QUIT signal is received ( kill -QUIT pid). If Ctrl-\ or Ctrl-Break is pressed multiple times, multiple profiles are generated to the one file. "

jmap

The jmap command-line utility prints memory related statistics for a running VM or core file.

Commands:

jmap -histo <pid> #show histogram of objects
jmap -dump:format=b,file=<file> #dump heap in HPROF format (can be processed by jhat)

jstat

"The jstat utility uses the built-in instrumentation in the HotSpot VM to provide information on performance and resource consumption of running applications. "

show garbage collection info, class loading info, compilation info, etc.

visualgc

GUI to show results of jstat.

Java VisualVM

http://download.oracle.com/javase/6/docs/technotes/guides/visualvm/index.html
command: jvisualvm

"Java VisualVM is a tool that provides a visual interface for viewing detailed information about Java applications while they are running on a Java Virtual Machine (JVM), and for troubleshooting and profiling these applications."

JConsole

"This tool is compliant with Java Management Extensions (JMX). The tool uses the built-in JMX instrumentation in the Java Virtual Machine to provide information on the performance and resource consumption of running applications."

jhat (java heap analysis tool)

"The jhat tool provides a convenient means to browse the object topology in a heap snapshot. This tool was introduced in the Java SE 6 release to replace the Heap Analysis Tool (HAT). "

Command:

jhat <hprof_file_name>

Eclipse MAT

jdb

Misc.

"As of Java SE 5.0 update 7, the -XX:+HeapDumpOnOutOfMemoryError command-line option
tells the HotSpot VM to generate a heap dump when an OutOfMemoryError occurs (see
section 1.9).
As of Java SE 5.0 update 14, the -XX:+HeapDumpOnCtrlBreak command-line option tells the
HotSpot VM to generate a heap dump when a Ctrl-Break or SIGQUIT signal is received (see
section 1.10). "

Resources

http://www.oracle.com/technetwork/java/javase/index-137495.html

Eucalyptus notes

2011-04-14T19:05:00.001-04:00

https://portal.futuregrid.org/tutorials/eucalyptus

Bundle images

http://open.eucalyptus.com/participate/wiki/modifying-prepackaged-image

http://open.eucalyptus.com/wiki/EucalyptusImageManagement_v1.6

Bundle running instance

http://megam.info/2010/04/28/euca-bundle-vol-solved/
http://open.eucalyptus.com/forum/how-create-snapshot-running-vm-instance-euca2ools-or-eucalyptus-web-interface

Basic commands

euca-bundle-vol  \
    --kernel <kernel_id> \
    --ramdisk <randisk_i> \
    -d <dir_to_put_bundled_img> \
    -p <prefix> \
    -r <arch>

euca-upload-bundle -b <bucket_name> -m <path_to_manifest_file>

euca-register <manifest_file>

Euca2tools and its dependency download

http://open.eucalyptus.com/downloads

http://eucalyptussoftware.com/downloads/releases/euca2ools-1.3.1-src-deps.tar.gz

http://eucalyptussoftware.com/downloads/releases/euca2ools-1.3.1.tar.gz

How to decommission nodes/blacklist nodes

2011-04-08T11:43:00.001-04:00

HDFS

Put following config in conf/hdfs-site.xml:

<property>
  <name>dfs.hosts.exclude</name>
  <value>/full/path/of/host/exclude/file</value>
</property>

Use following command to ask HDFS to re-read host exclude file and decommission nodes accordingly.

./bin/hadoop dfsadmin -refreshNodes

MapReduce

Put following config in conf/mapred-site.xml

<property>
  <name>mapred.hosts.exclude</name>
  <value>/full/path/of/host/exclude/file</value>
</property>

Use following command to ask Hadoop MapReduce to reconfigure nodes.

./bin/hadoop mradmin -refreshNodes

Whitelist/Recommission

Also you can "whitelist" nodes. In other words, you can specify which nodes are allowed to connect to namenode/jobtracker.

HDFS

Put following config in conf/hdfs-site.xml:

<property>
  <name>dfs.hosts</name>
  <value>/full/path/to/whitelisted/node/file</value>
</property>

Use following command to ask Hadoop to refresh node status to based on configuration.

./bin/hadoop dfsadmin -refreshNodes

MapReduce

Put following config in conf/mapred-site.xml

<property>
  <name>mapred.hosts</name>
  <value>>/full/path/to/whitelisted/node/file</value>
</property>

Use following command to ask Hadoop MapReduce to reconfigure nodes.

./bin/hadoop mradmin -refreshNodes

Support of -mradmin was added in 0.21.0. See JIRA issue https://issues.apache.org/jira/browse/HADOOP-5643 for details.

Japan earthquake GPS data visualization gadget

2011-03-19T01:19:00.001-04:00

I made a gadget version of QuakeSim Japan earthquake data visualization portal. It shows data (longitude, latitude and height) collected by GPS stations during Japan earthquake.

You can click http://www.google.com/ig/adde?synd=open&source=ggyp&moduleurl=hosting.gmodules.com%2Fig%2Fgadgets%2Ffile%2F105322631994749779353%2Fquakesim-japan.xml to add it to your iGoogle. After it is added, maximize it by clicking the icon near top right corner of the gadget

Link for the gadget is

http://www.google.com/ig/directory?url=hosting.gmodules.com%2Fig%2Fgadgets%2Ffile%2F105322631994749779353%2Fquakesim-japan.xml

Thank Xiaoming Gao for providing service pages.

Install ns2 (ns-2.33) on Ubuntu Maverick

2011-03-02T17:59:00.000-05:00

Install prerequisite:

sudo apt-get install \  
    tcl tcl-dev \ 
    libotcl1 libotcl1-dev  \ 
    tclcl-dev tclcl \ 
    tk tk-dev

./configure failed complaining that some tcl, tk related files cannot be found. It turns out those packages have been installed but file locations are different than what the configure script expects. Following is a fix.

You need to change two variables in file configure : TCL_TCL_PLACES and TK_TCL_PLACES.

Add
/usr/share/tcltk/tcl$TCL_VERS \
/usr/share/tcltk/tcl$TCL_HI_VERS
to variable TCL_TCL_PLACES.

Add
/usr/share/tcltk/tk$TK_HI_VERS \
/usr/share/tcltk/tk$TK_VERS"
to variable TK_TCL_PLACES.

execute ./configure

Official page: http://www.isi.edu/nsnam/ns/ns-build.html

Hadoop tips

2010-12-31T23:57:00.000-05:00

Change logging level

For each daemon, there is a service at http://daemon_address:port/logLevel through which you can get and set logging level.
Use command line
hadoop daemonlog -getLevel daemon_address:port fullQualifiedClassName
hadoop daemonlog -setLevel daemon_address:port fullQualifiedClassName logLevel
Permanent change
Change file log4j.properties. Example
    log4j.logger.org.apache.hadoop.mapred.JobTracker=DEBUG
    log4j.logger.org.apache.hadoop.mapred.TaskTracker=DEBUG
    log4j.logger.org.apache.hadoop.fs.FSNamesystem=DEBUG

Commission and decommission nodes
Following four config parameters are related:
dfs.hosts
dfs.hosts.exclude
mapreduce.jobtracker.hosts.filename (mapred.hosts for old version)
mapreduce.jobtracker.hosts.exclude.filename (mapred.hosts.exclude for old version)

For HDFS, execute "hadoop dfsadmin -refreshNodes" after you change the include file or exclude file.
From the mailing list, I know "mradmin -refreshNodes was added in 0.21". So for MapReduce, you can use "hadoop mradmin -refreshNodes" after you change the include file or exclude file to commission or decommission a node respectively.
To permanently add or remove a node, you also need to change slave file conf/slaves.
Block scanner report
http://datanode_address:50075/blockScannerReport
If you want to check blocks and block locations of a specific file, use following command:
hadoop fsck file_to_check -files -blocks -locations -racks
Note: you should execute it on master node.
Use "hadoop fsck /" to check health of the whole file system.

Extract some continuous lines from a file

2010-12-23T00:51:00.000-05:00

Sometimes, I want to extract some continuous lines from a file, e.g. line 10 to line 100. I was thinking whether there is any linux command to do that. Unfortunately, I did not find the command in vanilla Linux distros. Suddenly, I found that it can achieved by combining commands head and tail.

Let's say you want to extract line min to line max, both inclusive.
Calculate nlines=(max-min+1). Then use following command:

cat <filename>|head -n <max>|tail -n <nlines>

vim quickfix and location list

2010-12-17T01:00:00.001-05:00

Commands	Description
copen	open quickfix window
cclose	close quickfix window
cwindow	open quickfix window if its content is not empty.

cc [nr]	display error [nr]
cr	display the first error.
cfirst	display the first error.
clast	display the last error.
[count]cn	display [count] next error
[count]cp	display [count] previous error
[count]cnf	display first error in the [count] next file
[count]cpf	display first error in the [count] previous file

For commands related to location list, just replace first 'c' with 'l' in above commands.

Compile and Install vim on Linux

2010-12-16T11:43:00.001-05:00

Commands

wget ftp://ftp.vim.org/pub/vim/unix/vim-7.3.tar.bz2
tar jvxf vim-7.3.tar.bz2
cd vim73
./configure --enable-gui=no \
            --enable-multibyte \
            --enable-cscope \
            --disable-netbeans \
            --prefix=<your_desired_vim_home>
make
make test
make install

Change you environment variable PATH to add vim bin path. Following commands are for bash.

Temporary change
export PATH="<your_vim_home>/bin;$PATH"

Permanent change
echo -e '\nexport PATH="<your_vim_home>/bin;$PATH" ' >> ~/.bash_profile

Delete executables and object (.o) files

2010-12-15T13:53:00.001-05:00

Following two commands can delete all ELF executables and ELF object (.o) files.

find ./ -executable -type f|xargs -I{} file {} | grep ELF|cut -d ':' -f 1|xargs -I{} rm {}

find ./ -name "*.o" -type f|xargs -I{} file {} | grep ELF|cut -d ':' -f 1|xargs -I{} rm {}

Some executables may not be found because its "x" bit is not set. Use following command to find them

find ./ -type f|xargs -I{} file {} | grep ELF|cut -d ':' -f 1|xargs -I{} rm {}

Note:
-executable is not supported in old version of find.

How to change hostname in Ubuntu

2010-12-13T17:41:00.001-05:00

Temporary change

hostname <new_host_name>

Permanent change

Edit /etc/hostname to specify your new hostname
sudoedit /etc/hostname
sudo service hostname start

Ubuntu init scripts and upstart jobs

2010-12-13T17:35:00.001-05:00

Init script

Those scripts are located in directory /etc/init.d. Note: some of them have been converted to upstart jobs (see next section) and they should not be invoked directly. To check whether it's a upstart job, check directory whether file /etc/init/<job>.conf exists for a specific job.

upstart jobs

http://upstart.ubuntu.com/ Use "man 5 init" to see the syntax of the conf file.

Each upstart job has a conf file in directory /etc/init/<job>.conf. You should not directly invoke the init script to start/stop the job. You should use commands initctl to do that. E.g. initctl restart hostname.
initctl list will list upstart jobs that are running.

service

It can be used to interact with the init scripts, no matter they are upstart jobs or regular init script. For upstart jobs, it does not run /etc/init.d/<job> . Instead it runs "start <job>" directly.

E.g. service hostname status

invoke-rc.d

Another tool to start/stop init jobs. In my opinion you should use command service because invoke-rc.d does NOT detect whether the job is a upstart job or regular init job. Usually, this is not a big deal because upstart job shell script automatically calls initctrl related commands (start, stop, reload, etc) .

Example - network

I will give an example about how network interfaces are managed by init daemon.

As you may know, ifup and ifdown can be used to bring up or down network interfaces.

/etc/network/interfaces are used by ifup and ifdown to know how you want your system to connect to the network.

Sample file

# interfaces lo and eth0 should be started when ifup -a is invoked.
auto lo eth0
# eth1 is allowed to be brought up by subsystem hotplug.
allow-hotplug eth1
# For interface lo, it should use internet protocol and it is a loopback device.
iface lo inet loopback
# Interface eth1 uses internet protocol and dhcp for configuration
iface eth1 inet dhcp

For upstart job networking, its config file is /etc/init/networking.conf:

description "configure virtual network devices"

start on (local-filesystems
and stopped udevtrigger)

task

pre-start exec mkdir -p /var/run/network

exec ifup -a

Notice the last line? Yes, it invoke ifup to bring up those interfaces that are marked as "auto" in file /etc/network/interfaces.

Job network-interface is used when a network interface is added or removed. Its config file is /etc/init/network-interface.conf

description "configure network device"

start on net-device-added
stop on net-device-removed INTERFACE=$INTERFACE

instance $INTERFACE

pre-start script
    if [ "$INTERFACE" = lo ]; then
    # bring this up even if /etc/network/interfaces is broken
    ifconfig lo 127.0.0.1 up || true
    initctl emit -n net-device-up \
        IFACE=lo LOGICAL=lo ADDRFAM=inet METHOD=loopback || true
    fi
    mkdir -p /var/run/network
    exec ifup --allow auto $INTERFACE
end script

post-stop exec ifdown --allow auto $INTERFACE

Line "exec ifup --allow auto $INTERFACE" bring up the newly added interface if it is set to be brought up automatically. The trigger event is "net-device-added" or "net-device-removed" which is sent by upstart-udev-bridge. It basically forwards events received from udev to init daemon. When your network interface (e.g. eth0) is detected by udev, finally a net-device-added event is sent to network-interface upstart job which runs ifup to bring it up.

upstart job hostname. (/etc/init/hostname.conf). It includes following line:
exec hostname -b -F /etc/hostname
Now you should know how to change hostname.
Edit file /etc/hostname, run command "sudo service start hostname", "sudo start hostname", or "sudo initctl start hostname".

Recover corrupted partition table

2010-12-13T13:16:00.000-05:00

Partition table of my linux drive was corrupted recently. I could not start up my Ubuntu.

I burned a Ubuntu CD. But when I tried to boot into the liveCD, it always gave me errors. It seems to be a CD burning/CD drive problem. Then I made a live USB drive which worked great. Following two tools can be used to "guess" the partition table.

gpart
This program is kind of old and is not maintained any longer. It can just recognizes some file systems (ext3, ext4, etc are not recognized correctly)
testdisk
This is a great tool which a text UI. You can find information here. You just follow the instructions. Check the "guessed" partition table match your real partition table(if you have backup, you are lucky.).

Run fsck to check integrity of your file system.

Afterthoughts:

Back up your partition table!
USB drive is more stable than CD in this case.

printf cheatsheet

2010-11-28T16:33:00.001-05:00

Format string

% [flag]* [minimum_field_width] [precision] [length_modifier] <conversion_specifier>

Flags

Flag	Description
#	The value should be converted to an "alternate form".
0	zero padded. By default, blank padded
-	left adjusted
' '	(A single space) A blank should be left before a positive number (or empty string) produced by a signed conversion.
+	A sign is always placed before a number produced by a signed conversion.

Minimum Field Width

"a decimal digit string (with non-zero first digit). If the value has fewer characters, it will be padded. In no case does a nonexistent or small field width cause truncation of a field."

Precision

a period ('.') followed by an optional decimal digit string. It has different meanings for different conversions.

Precision	Description
d, i, o, u, x, and X	minimum number of digits to appear	printf("%.2d", 1) ==> "01"
a, A, e, E, f, and F	number of digits to appear after the radix character	printf("%.2f",0.1) ==> "0.10"
g and G	maximum number of significant digits
s and S	maximum number of characters to be printed from a string	printf("%.2s","hello") ==> "he"

Length Modifier

For each conversion specifier, there is expected argument type. For example, for conversion d, and i, type of arguments should be int. Length Modifiers can be used to specify argument types rather than expected type by default.

Modifier	Conversion	Argument types
hh	d, i, o, u, x, or X	signed char or unsigned char
hh	n	signed char*
h	d, i, o, u, x, or X	short int or unsigned short int
h	b	short int*
l	d, i, o, u, x, or X	long int, or unsigned long int
	n	long int*
	c	wint_t
	s	wchar_t
ll	d, i, o, u, x, or X	long long int, or unsigned long long int
ll	n	long long int*
L	a, A, e, E, f, F, g, or G	long double
j	d, i, o, u, x, or X	intmax_t, uintmax_t
z	d, i, o, u, x, or X	size_t, ssize_t
t	d, i, o, u, x, or X	ptrdiff_t

Conversion specifier

conversion	arguments	notation
d, i	int argument	signed decimal notation
o, u, x, X	unsigned int	unsigned octal, unsigned decimal, unsigned hexdec notation	abcedf are used for x. ABCDEF are used for X.
e, E	double	rounded and converted in the style [-]d.ddde±dd	precision is 6 by default.
f, F	double	rounded and converted to decimal notation in the style [-]ddd.ddd	precision is 6 by default.
g, G	double	converted in style f or e (or F or E for G conversions). Style e is used if the exponent from its conversion is less than -4 or greater than or equal to the precision.
a, A	double	converted to hexadecimal notation. for a, (using the letters abcdef) in the style [-]0xh.hhhhp±d;	C99; not in SUSv2
c	int	converted to an unsigned char, and the resulting character is written
s	const char*	Characters from the array are written.
p	void *	printed in hexadecimal (as if by %#x or %#lx)
n	int *	The number of characters written so far is stored into the integer indicated by the int * (or variant) pointer argument.

Some Hard Drive and File System benchmark tools

2010-11-28T01:37:00.000-05:00

IOMeter

iozone

http://www.iozone.org

Document: http://www.iozone.org/docs/IOzone_msword_98.pdf
Manual: http://linux.die.net/man/1/iozone

Read, write, re-read, re-write, read backwards, read strided, fread, fwrite, random read/write, pread/pwrite variants

iozone -a | tee result.txt

iozone supports bunch of command line options. I summarized them in following table

Category	Options	Note
Auto mode	-a: record size 4k - 16M, file size 64k - 512M. -z: Used in conjunction with -a to test all possible record sizes. (Normally Iozone omits testing of small record sizes for very large files when used in full automatic mode. ) -A: more coverage
Test file	-f filename: the name for temporary file under test. -F fn1 fn2: # of files should be equal to # of processors/threads
Output	-b filename: output of an Excel compatible file -R: Generate Excel report.
Record size	-r #: record size -y #: minimum record size for auto mode -q #: maximum record size for auto mode
File size	-s #: size of the file to test -g #: maximum file size for auto mode -n #: minimum file size for auto mode
tests	-i #: specifies which test to run 0=write/rewrite, 1=read/re-read, 2=random-read/write, 3=Read-backwards, 4=Re-write-record, 5=stride-read, 6=fwrite/re-fwrite, 7=fread/Re-fread, 8=mixed workload, 9=pwrite/Re-pwrite, 10=pread/Re-pread, 11=pwritev/Re-pwritev, 12=preadv/Re-preadv	One will always need to specify 0 so that any of the following tests will have a file to measure. This means -I 0 creates files used by following tests. -i # -i # -i # is also supported so that one may select more than one test.
	-+p percent_reads: the percentage of threads/processes that will perform read testing in the mixed workload test case
	-+B: sequential mixed workload testing
throughput tests	-t #: Run Iozone in a throughput mode. -T: Use POSIX pthreads for throughput tests	This option allows the user to specify how many threads or processes to have active during the measurement.
processes/threads	-l #: lower limit on number of processes to run -u #: upper limit on number of processes to run
Timing	-c: include close() in timing calculation -e: include fflush(), fsync() in timing calculation
Other control	-H #: Use POSIX async I/O with # async operations -k #: Use POSIX async I/O (no bcopy) with # async operations. -I: use direct I/O if possible -m: use multiple buffers internally -o: Writes are synchronously written to disk -p: purges the processor cache before each file operation. -W Lock file when reading or writing. -K: Inject some random accesses in the testing.

Examples

It's important to use -I option to turn on DIRECT I/O. Otherwise, linux's page caches (buffer cache) may give you ridiculous fast read speed.

Auto Mode

iozone -a -n 512m -g 1g

Single Test

Sequential write, Sequential reads
iozone -r 64k -s 1g -b excel.xls -R -i 0 -i 1 -I
Use two processes to test sequential writes, sequential reads
iozone -r 64k -s 1g -b excel.xls -R -i 0 -i 1 -I -l 2 -u 2
Use one and two processes to tests (two runs. In first run, one process is created. In second run, two processes are created)
iozone -r 64k -s 1g -b excel.xls -R -i 0 -i 1 -I -l 1 -u 2
Random writes, Random reads
iozone -r 64k -s 1g -b excel.xls-R -i 0 -i 1 -K -I

Throughput Test

iozone -t 2
roughly equivalent to "iozone -l 2 -u 2"

Mixed Workload Test

iozone -r 64k -s 1g -b excel.xls -R -i 0 -i 8 -+p 50

Result visualization

./Generate_Graphs result.txt

It will a directory for each operation (e.g. read, write, fread, fwrite). In each directory, there are two files generated - iozone_gen_out.gnuplot and <operation>.ps (this file is generated after you view the corresponding result using gnuplot). Under the hood, it uses gnu3d.dem to render the data using Gnuplot after those data files are generated for all tested operations. You can call it directly without regenerating separate data files.

gnuplot gnu3d.dem

There are two more scripts that can be used for visualization - report.pl and iozone_visualization.pl.
When I tried to run them in Linux (using command ./report.pl result.txt), I had following error:

-bash: ./iozone_visualizer.pl: /usr/bin/perl^M: bad interpreter: No such file or directory

You can use command perl report.pl result.txt to run it successfully.
The solution is to change those two files from dos type to unix type (mainly the new line character conversion).

sudo aptitude install tofrodos
fromdos report.pl
fromdos iozone_visualizer.pl

Now, you should be able to report.pl and iozone_visualization.pl directly. When they are run, a directory named 'report_result' is created. In the directory are Gnuplot scripts (*.do files) and PNG images for all tested operations. Those PNG images are generated by running those Gnuplot scripts. For example, read.png is generated by running "gnuplot read.do". A HTML page (index.html) is generated by iozone_visualization.pl which contains all of those PNG images in the same page.

Bonnie

http://www.textuality.com/bonnie/

Bonnie64

http://code.google.com/p/bonnie-64
Bonnie++
After you download the source, just execute following commands to build and run.
./configure --prefix=installation_dir
./make
./make install
cd <installation_dir>/sbin/
./bonnie++
http://www.coker.com.au/bonnie++/
Manual: http://linux.die.net/man/8/bonnie++

Resources

Permission of authorized_keys and private key file

2010-11-25T16:57:00.001-05:00

~/.ssh/authorized_keys: should be 600

~/.ssh/id_dsa, ~/.ssh/id_rsa: must be 600

If permissions are not set correctly, the login process may fail without giving any useful information!

awk/gawk notes

2010-11-25T11:24:00.000-05:00


RS	Record Separator	single character	That character separates the records.
		regular expression	Text in input matches reg exp separates records.
		null string	Records are separated by blank lines. The newline character always acts as a field separator, in addition to whatever value FS may have.
NR	number of records seen so far
FNR	The input record number in the current input file
ORS	output record separator
FS	Field separator	single character	Fields are separated by that character
		single space	fields are separated by runs of spaces and/or tabs and/or newlines.
		Null string	each individual character becomes a separate field.
		regular expression
OFS	output field separator
FIELDWIDTHS		a space separated list of numbers	each field is expected to have fixed width. The value of FS is ignored. Assigning a new value to FS overrides the use of FIELDWIDTHS, and restores the default behavior.
NF	number of fields		Decrementing NF causes the values of fields past the new value to be lost, and the value of $0 to be recomputed
IGNORECASE	Controls the case-sensitivity of all regular expression and string operations.	non-zero zero	non-zero: ignore case

Field Reference	How to reference a field	$1, $2, … $NF	Access a field. Assigning a value to an existing field causes the whole record to be rebuilt when $0 is referenced.
		$-1, $-2	fatal error
		non-existent fields	For read, produce null-string. For write, 1)increase NF 2) create intervening fields with null string 3) $0 is recomputed
		$0	whole record. assigning a value to $0 causes the record to be resplit, creating new values for the fields.
CONVFMT			A number is converted to a string by using the value of CONVFMT as a format string for sprintf(3), with the numeric value of the variable as the argument. However, even though all numbers in AWK are floating-point, integral values are always converted as integers.
OFMT

All arrays in AWK are associative, i.e. indexed by string values.

i = "A"; j = "B"; k = "C"
x[i, j, k] = "hello, world\n"

key is "A\034B\034C" and value is "hello, world\n". Key test: val in array. for(val in array)…

XLink

2010-11-14T17:59:00.003-05:00

http://www.xml.com/lpt/a/1038

Can be transformed to RDF?

One RDF use include to include other RDFs.

XLink and HLink: http://www.xml.com/lpt/a/1038

Enhancements to html link

When to actuate the link
In current html link impl, the link is actuated when it is clicked. In XLink and HLink, a link can be actuated when the page containing the link is loaded
More effects when a link is actuated.
embed, new, replace, etc.

HLink supports creation of arbitrary link element.

<hlink namespace="http://www.example.com/markup"
       element="home"
       locator="/"
       effect="replace"
       actuate="onRequest"/>
<hlink namespace="http://www.example.com/markup"
       element="home"
       locator="/icons/home.png"
       effect="embed"
       actuate="onLoad"/>

<home/>

XLink supports creation of links among more than two resources.
Add more metadata to links
Links can be specified outside the linked resources.
In HTML, users can only specify links within the source resource.
When you write
```
 <a href=”destination.resource”>source</a>
```
this piece of code must be located in the source html. In other words, the user cannot specify links among external resources.
XLink adds this support.

METS, DIDL, ORE

http://www.oreillynet.com/xml/blog/2008/06/oaiore_compound_documents_draf.html

http://www.oreillynet.com/xml/blog/2008/05/bad_xml.html

http://www.dehora.net/journal/2008/06/18/dates-in-atom/

http://dret.net/netdret/docs/wilde-cacm2008-xml-fever.html

http://www.tbray.org/ongoing/When/200x/2006/01/09/On-XML-Language-Design

Google AppEngine mail test

2010-11-14T17:57:00.001-05:00

Mail

How to test: http://aralbalkan.com/1311

Two bugs:

http://code.google.com/p/googleappengine/issues/detail?id=626
For this bug, you can upgrade your python to new version (2.5.4 and up).

http://code.google.com/p/googleappengine/issues/detail?id=1061

http://groups.google.com/group/app-engine-patch/browse_thread/thread/1662f95d9cacee24

Deb package manipulation notes (deb make, view, install, etc)

2010-11-14T17:56:00.001-05:00

Directory /var/lib/dpkg/info/ contains package related files. For each package, its conffiles, md5sums, preinst, postinst, prerm, postrm, list of installed files, etc are kept there.

dpkg-dev

debian/files: "The list of generated files which are part of the upload being prepared."

.changes: upload control file

dpkg-buildpackage
build binary or source packages from sources

dpkg-architecture: set and determine the architecture for package building

dpkg-checkbuilddeps: check build dependencies and conflicts. By default, debian/control is read.

dpkg-distaddfile: adds an entry for a named file to debian/files.
dpkg-genchanges:
dpkg-gencontrol: generate Debian control files
dpkg-gensymbols:

dpkg-name
dpkg-scanpackages: create Packages index files
dpkg-scansources: create Sources index files
dpkg-shlibdeps
dpkg-source: packs and unpacks Debian source archives.
dpkg-vendor: query vendor information
dpkg-parsechangelog: get changelog information

Vendor

/etc/dpkg/origins/default

devscripts

debchange

debhelper

dh-make

This package is useful when you have a regular source package (not debian source package) and want to debianlize it.
dh_make must be invoked within a directory containing the source code, which must be named <packagename>-<version>. The <packagename> must be all lowercase, digits and dashes.
As I mentioned, there are two types of debian source packages – native and non-native.
For non-native package, obviously you need the original source tree. The reason is that the original source tree is needed to deb tools to generate diff. dh_make makes sure original source tarball(<packagename>_<version>.orig.tar.gz) exists.
Option –f can be used to specify location of the tarball.
If –f is not given, dh_make searches parent directory for file <packagename>_<version>.orig.tar.gz and directory <packagename>_<version>.orig. If either of them exists, it will be fine. If neither exists, dh_make will complain and exit.
If you want to create a original source tarball based on the code in current directory, use option "—createorig". Then current directory is copied to <packagename>_<version>.orig in parent directory.

key: public key
secret: private key

Trusted pub keys are stored in file /etc/apt/trusted.gpg (not /etc/apt/trustdb.gpg)

apt-key list

gpg --recv-keys --keyserver keyserver.ubuntu.com key_ID_here;
gpg --export --armor key_ID_here | sudo apt-key add -

http://wiki.debian.org/SecureApt
https://help.ubuntu.com/community/SecureApt

Downloaded deb packages are stored at /var/cache/apt/archives/ and /var/cache/apt/archives/partial/.

Low-level understanding

Deb binary package format

man deb
The manual describes debian binary package format
deb package is ar archive. So you can read content of a deb package using command:
ar tf pkg_name.deb
On my machine, the output is

    debian-binary 
    control.tar.gz 
    data.tar.gz

Extract content of a deb pkg using command:
ar xof pkg_name.deb

Deb control

control.tar.gz is a control file. Its format is deb-control.
"It is a gzipped tar archive containing the package control information, as a series of plain files, of which the file control is mandatory and contains the core control information."
Use command tar zvxf control.tar.gz to extract control files. The most important file is control. The format of the file is described in man deb-control.
conffiles: this file lists all configuration files used by this package.
control:
md5sums
postinst
postrm
preinst
prerm

Deb data

"It contains the filesystem as a tar archive, either not compressed".

High-level understanding

Ubuntu provides some tools to make it more convenient to manipulate deb package so that users don't need to use ar, tar, etc to extract files/information manually.

First, command dpkg-deb comes really handy

    dpkg-deb –I: provides information of a deb pkg. (Extracts info from file control) 
    dpkg-deb –c: list content of the package. (Extracts info from data.tar.gz)
    dpkg-deb –x: extract a deb archive 
    dpkg-deb –X: extract a deb archive and print list of extracted files. 
    dpkg-deb –e: extract control information to DEBIAN directory if not specified.
                 (Extract files from control.tar.gz)

Deb Source Package

Format of source package is described in section "SOURCE PACKAGE FORMATS" within manual "man dpkg-source".
Also read http://www.debian.org/doc/debian-policy/ch-source.html for more info.

There are two types of source packages: native and non-native.
Layout of native package

  .dsc: includes package info and md5 checksum for the package content.
  .tar.gz

Layout of non-native package:

  .dsc: debian source control 
  .orig.tar.gz: source code
  .diff.gz: 1)patches applied to the source code; 2) debian package (debain/ dir)

Download a source package instead of binary packge:

  apt-get source pkg_name     #Download and unpack
  apt-get source --download-only pkg_name    #only download

Then command dpkg-source comes handy to manipulate source package.

  dpkg-source –x pkg_name.dsc    # Extract a source package.

If you use command "apt-get source pkg_name", the package has been download and extracted. So you don't need to execute this command. If you use command "apt-get source --download pkg_name", you can use this command to extract the downloaded package and apply the patch.
If you don't want the patch to be applied, add option --skip-debianization.

If the directory where you execute command "dpkg-source –x" is different from the directory where downloaded source package is stored, option "-su, –sp, sn" can be used to specify where source tarball will be copied to current direcotory.

In all cases any existing original source tree will be removed! So be sure to backup your code if it is in current directory.

  dpkg-source –sn –x pkg_name.src    #original source tarball is not copied to current directory. But source tree is unpacked to current dir and patch is applied
  dpkg-source –sp –x pkg_name.src    #source tarball is copied to current directory, unpacked, and patch is applied
  dpkg-source –su –x pkg_name.src    #Copy source tarball to current directory, both original source tree and patched source tree are extracted.

If you want the original source is extracted also, use command "dpkg-source –su –x pkg_name.dsc".
When I extracted the source package, I got following warning:
gpgv: Can't check signature: public key not found
dpkg-source: warning: failed to verify signature on ./pkg_name.dsc

This means public key has not been found which is needed to verify signature of the package. The dpkg-source manaul can tell you more:

--no-check
  Do not check signatures and checksums before unpacking.
--require-valid-signature
  Refuse to unpack  the source package if it doesn't contain an OpenPGP signature that can be verified either with the user's trustedkeys.gpg keyring, one of the vendor-specific  keyrings,  or  one of the official Debian keyrings (/usr/share/keyrings/debiankeyring.gpg and /usr/share/keyrings/debianmaintainers.gpg).

debian/ direcotory

Version:
https://wiki.ubuntu.com/PackagingGuide/Complete#changelog

changelog

Default file is located at debian/changelog. Changelog contains a list of changes. Note: it has a specific format. Command debchange can be used to edit the file.

debchange –a        #append a changelog entry at current version
debchange –i         #increase release number for non-native packages (2.4-1ubuntu1 –> 2.4-1ubuntu2).
debchange –v        #create a changelog entry for a arbitrary new version.
debchange --create #create a new changelog file
debchange –c changelogfile #edit a specified changelog file instead of default one.

Read http://www.debian.org/doc/debian-policy/ch-source.html#s-dpkgchangelog for more info.

dpkg-source –b # build source package
man deb-version
Debian package version number format

Export:
gpg --export-secret-keys keyID
gpg --export keyID    #export public key
gpg --gen-key
gpg –k   #list pub keys
gpg –K   #list secret keys

copyright

Read this: https://wiki.ubuntu.com/PackagingGuide/Basic#Copyright%20Information

control

https://wiki.ubuntu.com/PackagingGuide/Complete#control

rules
It specifies how to compile, install the app and create the .deb package.
https://wiki.ubuntu.com/PackagingGuide/Complete#rules

DEBFULLNAME:
DEBEMAIL:

Package build

Binary package

dpkg-buildpackage
debuild: wrap dpkg-buildpackage and some other tools. Or you can set variable DEBSIGN_KEYID to the key id.
Use debuild –kKEYID to specify the key used to sign the package.
If you want to pass parameters to dpkg-buildpackage, set variable DEBUILD_DPKG_BUILDPACKAGE_OPTS.

debsign –kkeyID
debsign –m'LastName FirstName (Comment) <email_address>'

Source package: debuild –S

lintian
Debian package checker
lintian -Ivai *.dsc

sudo pbuilder build pkg_name.dsc

dpkg-query –s pkg_name    #conf files are listed

https://wiki.ubuntu.com/PackagingGuide/Complete
https://wiki.ubuntu.com/DebootstrapChroot
https://wiki.ubuntu.com/PackagingGuide/Basic
https://wiki.ubuntu.com/PbuilderHowto
https://help.ubuntu.com/community/GnuPrivacyGuardHowto

http://www.debian.org/doc/FAQ/ch-pkg_basics.en.html

http://www.debian.org/doc/manuals/maint-guide/index.en.html

http://www.debian.org/doc/debian-policy/

How to build Hadoop in Eclipse

2010-11-14T02:13:00.000-05:00

Subclipse

Install SVN
I installed "Slik-SVN" because it provides JavaHL lib for 64bit Windows.
Install Subclipse plugin to Eclipse.
Change eclipse.ini to add following parameters after -vmargs:
-Djava.library.path=/usr/share/jni/lib (For linux)
-Djava.library.path=<svn_install_dir>/bin (For Windows)
Start eclipse
Goto WIndow --> Preference --> Team --> SVN
In section "SVN Interface", it should say something like "JavaHL (JNI) … SlikSvn". If it says "JavaHL(JNI) not available", it means subclipse cannot find JavaHL library. Check step 3).

Code checkout

Example: svn co http://svn.apache.org/repos/asf/hadoop/common/tags/release-0.21.0/
You can also check out code within Eclipse using Subclipse plugin.

Install Ant and Ivy

Read http://ant.apache.org/ for how to install Ant.

Download ivy jar and put it into directory ANT_HOME/lib/. If ANT_HOME is not specified explicitly, it is the installation directory.

IvyDE

Hadoop is managed by ivy. You need IvyIDE Eclipse plugin. Read http://ant.apache.org/ivy/ivyde/ for more info. IvyDE includes ivy jar file itself. So it does not use the ivy jar you installed in last step. Also it seems that ANT_HOME variable is set to <eclipse_dir>\plugins\org.apache.ant_1.7.1.v20090120-1145 (version number may vary for you).

Shell and Unix commands on Windows

Hadoop build file invokes sh and some other linux commands such as tr, sed to build the project. Of course those commands don't exist on Windows.

Following two projects port linux tools to Windows:

I use the first one. You just download the tarball and decompress it to a directory. This directory must be passed to Ant. The usual way is you put it into environment variable "PATH". Ant will pick it up automatically. It's true for command line use of Ant. It does not work well for Ant within Eclipse. Following sections include instructions about how to pass PATH to Ant in Eclipse.
For command line use, try command "ant compile".

Create Eclipse Project for Hadoop

New --> Java Project
Select "Create project from existing source". Then select the directory where code is located.
Click "Next"
CHANGE OUTPUT DIRECTORY TO <workspace_name>/build.
The default directory bin is used by Hadoop for different purposes.
Click "Finish"
Add JDK's tools.jar to build path. It is not included in JRE.
Change source directories to tell Eclipse which directories include Java source code.
Right click project name --> Build Path --> Configure Build Path… --> Source
Make IvyDE to manage ivy dependencies.
Right click project name --> Build Path --> Configure Build Path… --> Libraries --> Add Library --> IvyDE Managed Dependencies --> Next --> (couple of IvyDE setting steps) --> OK.
It may take some time for IvyDE to resolve dependencies.

Create Run Configuration

Right click "build.xml" --> Run As --> Ant Build … (not "Ant Build")
A dialog should pop up

Switch to "Targets" tab: select corresponding target (e.g. compile) you want to execute.
Switch to "JRE" tab: select "separate JRE"
This step is for Windows users.
Switch to Environment Tab: set PATH. (to include where those linux tools are included on Window)
You can click "Select" and choose variable "Path". But in my case, its value does NOT include all of the content of the variable (use "path" command in command line). probably, Eclipse has some restriction about length of value of environment variable. If it's too long, it will be truncated.
Click "Run"

See Console for messages.

Customize project builder

It's more convenient to use "Builder" than right click "build.xml" --> Run As --> Ant Build … --> Run. Following steps tell you how to use ant as default builder. Then you can use "Project-->Build Project" to build your project (same as any regular native Eclipse Java project).

Right click project name --> Properties --> Builder --> New --> Ant Builder

Select the build file (usually "build.xml").
Switch to "Targets" tab. Specify which targets are executed when the project is built or cleaned.
This step is for Windows users.
Switch to "Environment" tab. Add PATH environment variable if needed. (to include where those linux tools are included on Window)

Deselect default java builder.

package installation log on Ubuntu (dpkg, apt-get, aptitude)

2010-11-10T18:13:00.001-05:00

Dpkg log

All deb package operations must go through deb system. So no matter you use apt-get install or deb -I, it will be logged in /var/log/dpkg.log.

lesspipe can show content of .gz files directly. But it cannot show normal text file .

Show install and upgrade history for dkg.log.#.gz files:

ls /var/log/dpkg.log*|sort -r|xargs -I{} lesspipe {}|egrep "^[0-9\-]+[[:space:]][0-9:]+[[:space:]](install|upgrade)[[:space:]]"

Show install and upgrade history for dkg.log and dpkg.log.# files:

ls /var/log/dpkg.log*|sort -r|grep -v ".gz"|xargs -I{} cat {}|egrep "^[0-9\-]+[[:space:]][0-9:]+[[:space:]](install|upgrade)[[:space:]]"

apt-get log

apt-get logges to /var/log/apt/term.log

aptitude

/var/log/aptitude

Resources

http://superuser.com/questions/6338/how-do-you-track-which-packages-were-installed-on-ubuntu-linux

As far as logs, apt-get notoriously doesn't have one;
dpkg does (at /var/log/dpkg.log) but it's famously hard to parse and can only be read with root privileges;
aptitude has one at /var/log/aptitude and you can page through it with regular user privileges.

paratrac on Ubuntu

2010-11-10T01:13:00.001-05:00

Compile ftrack

Dependencies

Depends on: fuse-dev, glib-dev, gthread-dev

sudo apt-get install libfuse-dev libglib2.0-dev

Use following commands to check whether they are installed successfully

pkg-config --libs --cflags glib-2.0
pkg-config --libs --cflags fuse

Build

cd fuse/ftrac/
./configure prefix=your_prefix
make
make install

Add ftrack to PATH:
export PATH=your_prefix/bin:$PATH
which ftrac

Use fusetrac.py

add parent directory of paratrac to PYTHONPATH

python fusetrac.py -t /tmp/fuse/

FUSE FS is mounted and a monitoring page is shown. From now on, when you access /tmp/fuse, data on monitoring page will be changed.

cd /tmp/fuse

Python in Ubuntu/Debian

2010-11-09T19:38:00.001-05:00

http://www.debian.org/doc/packaging-manuals/python-policy/ch-python.html

site module

http://docs.python.org/library/site.html

I assume sys.prefix and sys.exec_prefix are /usr. It may be different on your machine.

/usr/lib/pythonX.Y/site-packages

/usr/lib/site-python

"It sees if it refers to an existing directory, and if so, adds it to sys.path and also inspects the newly added path for configuration files."

Note: sub-directories are not added.

If .pth files exist in those directories, its contents are additional items (one per line) to be added to sys.path.

local admin

A special directory is dedicated to public Python modules installed by the local administrator, /usr/local/lib/pythonX.Y/dist-packages for python2.6 and later, and /usr/local/lib/pythonX.Y/site-packages for python2.5 and earlier. For a local installation by the administrator of python2.6 and later, a special directory is reserved to Python modules which should only be available to this Python, /usr/local/lib/pythonX.Y/site-packages. Unfortunately, for python2.5 and earlier this directory is also visible to the system Python. Additional information on appending site-specific paths to the module search path is available in the official documentation of the site module.

Central repository

It seems all (not all?) modules are installed into directory /usr/share/pyshared. It's a central module repository for python. Python modules in other system directories are symbolic references to files in this directory.

/usr/lib/pyshared/python2.6/: .so python extensions?

python-central

python-central is a tool for Python module management.

pycentral: register and build utility for Python packages. It manages python modules you installed.

pyversions prints information about installed, supported python runtimes,

py_compilefiles: compiles Python .py source files into .pyc or .pyo bytecode format.

It adds hooks for runtime change:

/usr/share/python/runtime.d/pycentral.rtinstall
/usr/share/python/runtime.d/pycentral.rtremove
/usr/share/python/runtime.d/pycentral.rtupdate

python-support

python-central is another tool for Python module management.

modules managed by python-support are installed in another directory which is added to the sys.path using the .pth mechanism. The .pth mechanism is documented in the Python documentation of the site module.

During installation, it adds a file /usr/lib/python2.6/dist-packages/python-support.pth which contains /usr/lib/pymodules/pythonX.Y/. This directory also contains the byte-compiled modules for version pythonX.Y.

update-python-modules can be used to rebuild those modules.

It also adds hooks for runtime change:

    /usr/share/python/runtime.d/python-support.rtinstall
    /usr/share/python/runtime.d/python-support.rtremove
    /usr/share/python/runtime.d/python-support.rtupdate

euca2tools

2010-11-06T18:37:00.001-04:00

Resources:

http://wiki.debian.org/euca2ools

http://open.eucalyptus.com/wiki/Euca2oolsUsing

If you choose to use REST APIs, following options are necessary:

-U: endpoint to which requests are sent

-a: access key ID

-s: secret key

Note: for some tools, access key ID is specified via option "-A" instead of "-a", "-S" instead of "-s".

Install a package from Debian repository for Ubuntu

2010-11-06T18:10:00.000-04:00

You can directly use Debian repository. But Debian packages may or may not be compatible with Ubuntu. So you take your own risks by doing so. Another way is to download source and build the package, which is described below.

1) Add a line to /etc/apt/sources.list

deb-src repo-url

2) Update package index

sudo apt-get update

3) Install dependencies: (These dependencies are downloaded from Ubuntu repository, not Debian repository)

sudo apt-get build-dep pkg_name

4) Download source and build package

apt-get -b source pkg_name

Now you should have a pkg_name.deb file generated in current directory.

5) Install the package

sudo dpkg -I pkg_name.deb

6) Revert /etc/apt/sources.list file by removing the line added in step 1).

7) Rebuild package index

sudo apt-get update

You are done!

"Twitter is over capacity"

2010-11-03T23:42:00.001-04:00

Arrays in Bash (Cheatsheet)

2010-10-30T11:08:00.001-04:00

Description	Indexed array	Associative array
Declare an array	declare -a array	declare -A array
Assignment	array=(value1 value2 … valuen)	array=([sub1]=value1 … [subN]=valueN) Note:You must declare it first you can use declare -A array=([sub1]=value1 … [subN]=valueN)
Expand to all values	${array[@]} or ${array[*]}	${array[@]} or ${array[*]}
Expand to all keys	${!array[@]} or ${!array[*]}	${!array[@]} or ${!array[*]}
length of an array	${#array[@]} or ${#array[*]}	${#array[@]} or ${#array[*]}
Remove entire array	unset array or unset array[@] or unset array[*]	unset array or unset array[@] or unset array[*]
Remove an element	unset array[index]	unset array[index]
Array join	array1+=(${array2[@]}) or array1+=(v1 v2 … vN)	array1+=([sub1]=v1 [sub2]=v2 … [subN]=vN)
Length of an element:	${#array[index]}	${#array[index]}

Resources

http://tldp.org/LDP/abs/html/arrays.html

Bash config file loading order

2010-10-29T16:17:00.001-04:00

interactive	login	files sourced	Related option
Y	Y	/etc/profile, {~/.bash_profile\| ~/.bash_login\| ~/.profile } Only one of the files in brackets will be executed.	--noprofile
N	Y		--noprofile
Y	N	/etc/bash.bashrc, ~/.bashrc	--norc

Kindle 3rd generation

2010-10-29T11:14:00.001-04:00

I got problems when I tried to read some books in Chinese.

I will basically investigate following questions

What encodings are supported by Kindle
What fonts are supported
If the fonts used in the PDF cannot be found, how Kindle handles it.
What formats are supported?

GNOME notes

2010-10-27T00:28:00.001-04:00

GDM

/etc/init.d/gdm, /etc/init/gdm.conf: these two files are used to start gdm (actually gdm-binary).

Default display manager is stored here: /etc/X11/default-display-manager

GDM configuration: http://library.gnome.org/admin/gdm/2.32/gdm.html#configuration

Tools: gdmsetup

/etc/gdm

/user/share/gdm/

After the user logins in, gnome-session is invoked.

http://library.gnome.org/admin/system-admin-guide/stable/sessions-1.html.en

man gnome-session

Default session is specified at /desktop/gnome/session/default_session.

Use gconf-editor to view value of /desktop/gnome/session/default_session. In my case the value is "gnome-settings-daemon".
/usr/lib/gnome-settings-daemon/gnome-settings-daemon --gconf-prefix=/apps/gdm/simple-greeter/settings-manager-plugins. The description of option "--gconf-prefix" is "GConf prefix from which to load plugin settings"

Some resources:

http://live.gnome.org/SessionManagement/GnomeSession

http://standards.freedesktop.org/desktop-entry-spec/latest/

Autostart-spec: http://standards.freedesktop.org/autostart-spec/autostart-spec-latest.html

http://lists.freedesktop.org/archives/xdg/2007-January/007436.html

Autostart apps are specified in directory /etc/xdg/autostart/, ~/.config/autostart/

Use /usr/bin/gnome-session-properties to set session properties (e.g. autostart programs). Those changes are specific to a user.

JAASRealm + SSL

2010-10-26T00:51:00.000-04:00

http://tomcat.apache.org/tomcat-6.0-doc/realm-howto.html#JAASRealm

This is a good article on the topic: http://blog.frankel.ch/custom-loginmodule-in-tomcat. Some of following code is borrowed from the article.

Following steps are listed on the official tomcat doc. I will elaborate each of them.

"Write your own LoginModule, User and Role classes based on JAAS (see the JAAS Authentication Tutorial and the JAAS Login Module Developer's Guide) to be managed by the JAAS Login Context (javax.security.auth.login.LoginContext) When developing your LoginModule, note that JAASRealm's built-in CallbackHandler only recognizes the NameCallback and PasswordCallback at present. "

package test;

import java.io.IOException;
import java.util.Map;

import javax.security.auth.Subject;
import javax.security.auth.callback.Callback;
import javax.security.auth.callback.CallbackHandler;
import javax.security.auth.callback.NameCallback;
import javax.security.auth.callback.PasswordCallback;
import javax.security.auth.callback.UnsupportedCallbackException;
import javax.security.auth.login.LoginException;
import javax.security.auth.spi.LoginModule;

/**
 * Login module that simply matches name and password to perform authentication.
 * If successful, set principal to name and credential to "AuthorizedUser".
 *
 * @author Nicolas Fränkel. Modified by Gerald Guo.
 * @since 2 avr. 2009
 */
public class PlainLoginModule implements LoginModule {

    /** Callback handler to store between initialization and authentication. */
    private CallbackHandler handler;

    /** Subject to store. */
    private Subject subject;

    /** Login name. */
    private String login;

    /**
     * This implementation always return false.
     *
     * @see javax.security.auth.spi.LoginModule#abort()
     */
    @Override
    public boolean abort() throws LoginException {

        return false;
    }

    /**
     * This is where, should the entire authentication process succeeds,
     * principal would be set.
     *
     * @see javax.security.auth.spi.LoginModule#commit()
     */
    @Override
    public boolean commit() throws LoginException {

        try {

            PlainUserPrincipal user = new PlainUserPrincipal(login);
            PlainRolePrincipal role = new PlainRolePrincipal("AuthorizedUser");

            subject.getPrincipals().add(user);
            subject.getPrincipals().add(role);

            return true;

        } catch (Exception e) {

            throw new LoginException(e.getMessage());
        }
    }

    /**
     * This implementation ignores both state and options.
     *
     * @see javax.security.auth.spi.LoginModule#initialize(javax.security.auth.Subject,
     *      javax.security.auth.callback.CallbackHandler, java.util.Map,
     *      java.util.Map)
     */
    @Override
    public void initialize(Subject aSubject, CallbackHandler aCallbackHandler, Map aSharedState, Map aOptions) {

        handler = aCallbackHandler;
        subject = aSubject;
    }

    /**
     * This method checks whether the name and the password are the same.
     *
     * @see javax.security.auth.spi.LoginModule#login()
     */
    @Override
    public boolean login() throws LoginException {

        Callback[] callbacks = new Callback[2];
        callbacks[0] = new NameCallback("login");
        callbacks[1] = new PasswordCallback("password", true);

        try {

            handler.handle(callbacks);

            String name = ((NameCallback) callbacks[0]).getName();
            String password = String.valueOf(((PasswordCallback) callbacks[1]).getPassword());

            if (!name.equals(password)) {

                throw new LoginException("Authentication failed");
            }

            login = name;

            return true;

        } catch (IOException e) {

            throw new LoginException(e.getMessage());

        } catch (UnsupportedCallbackException e) {

            throw new LoginException(e.getMessage());
        }
    }

    /**
     * Clears subject from principal and credentials.
     *
     * @see javax.security.auth.spi.LoginModule#logout()
     */
    @Override
    public boolean logout() throws LoginException {

        try {

            PlainUserPrincipal user = new PlainUserPrincipal(login);
            PlainRolePrincipal role = new PlainRolePrincipal("admin");

            subject.getPrincipals().remove(user);
            subject.getPrincipals().remove(role);

            return true;

        } catch (Exception e) {

            throw new LoginException(e.getMessage());
        }
    }
}

"Although not specified in JAAS, you should create seperate classes to distinguish between users and roles, extending javax.security.Principal, so that Tomcat can tell which Principals returned from your login module are users and which are roles (see org.apache.catalina.realm.JAASRealm). Regardless, the first Principal returned is always treated as the user Principal. "
Also read the API doc http://tomcat.apache.org/tomcat-5.5-doc/catalina/docs/api/org/apache/catalina/realm/JAASRealm.html. If authentication succeeds, your LoginModule must attach at least a user principal and a user role to subject.

package test;

import java.security.Principal;

public class PlainRolePrincipal implements Principal {

    String roleName;
    
    public PlainRolePrincipal(String name) {
        roleName = name;
    }
    public String getName() {
        return roleName;
    }
    
    public String toString() {
        return ("RolePrincipal: " + roleName);
    }   

    public boolean equals(Object obj) {
        if (this == obj) {
            return true;
        }   
        if (obj instanceof PlainRolePrincipal) {
            PlainRolePrincipal other = (PlainRolePrincipal) obj;
            return roleName.equals(other.roleName);
        }   
        return false;
    }   

    public int hashCode() {
        return roleName.hashCode();
    }   
}

You get the idea, you can implement class PlainUserPrincipal in a similar way.

"Place the compiled classes on Tomcat's classpath "
"Set up a login.config file for Java (see JAAS LoginConfig file) and tell Tomcat where to find it by specifying its location to the JVM, for instance by setting the environment variable: JAVA_OPTS=$JAVA_OPTS -Djava.security.auth.login.config==$CATALINA_BASE/conf/jaas.config "
Create a JAAS config file:
-----------------------------
```
CertBasedCustomLogin {
    test.CertBasedLoginModule
    sufficient;
};
```
-----------------------------
When you launch tomcat, use -Djava.security.auth.login.config= to specify where the config file is stored.

"Configure your security-constraints in your web.xml for the resources you want to protect"
The goal of the whole process is to protect some resources. This step specifies which resources should be protected.

<security-constraint>
    <web-resource-collection>
        <web-resource-name>Secure Content</web-resource-name>
        <url-pattern>/cert-protected-users/*</url-pattern>
    </web-resource-collection>
    <auth-constraint>
        <role-name>AuthorizedUser</role-name>
    </auth-constraint>
    <user-data-constraint>
        <transport-guarantee>NONE</transport-guarantee>
    </user-data-constraint>
</security-constraint>
<!-- ... -->
<login-config>
    <auth-method>CLIENT-CERT</auth-method>
    <realm-name>The Restricted Zone</realm-name>
</login-config>
<!-- ... -->
<security-role>
    <description>The role required to access restricted content </description>
    <role-name>AuthorizedUser</role-name>
</security-role>

Basically, it says only users with role "AuthorizedUser" can access the resources cert-protected-users/*.
Note:role-name must match the role attached to subject in step 1) ("AuthorizedUser" in our case) for successful access.

"Configure the JAASRealm module in your server.xml"
Actually, to put web app specific context config into server.xml is not recommended. Instead, I put a file named context.xml under directory META-INF.
```
<Context>
    <Realm className="org.apache.catalina.realm.JAASRealm" appName="CertBasedCustomLogin"
        userClassNames="test.PlainUserPrincipal"
        roleClassNames="test.PlainRolePrincipal">
    </Realm>
</Context>
```
The value of appName must match the name specified in step 4.
Add "-Dsun.security.ssl.allowUnsafeRenegotiation=true" for renegotiation support. (Read http://java.sun.com/javase/javaseforbusiness/docs/TLSReadme.html for more information)

Note

Some versions of Tomcat have problems to support JAASRealm + SSL mutual auth (https://issues.apache.org/bugzilla/show_bug.cgi?id=45576). I tried 6.0.18, 6.0.20 and 6.0.29. Only 6.0.20 works for me. 6.0.29 gave errors when I tried.

More resources:

http://tomcat.apache.org/tomcat-6.0-doc/realm-howto.html#JAASRealm

http://tomcat.apache.org/tomcat-5.5-doc/catalina/docs/api/org/apache/catalina/realm/JAASRealm.html

http://wiki.metawerx.net/wiki/Web.xml.AuthConstraint

https://issues.apache.org/bugzilla/show_bug.cgi?id=45576

http://java.sun.com/javase/javaseforbusiness/docs/TLSReadme.html

web.xml schema (simplified)

2010-10-25T14:59:00.001-04:00

  <xsd:complexType name="web-appType">

    <xsd:choice minOccurs="0" maxOccurs="unbounded">
      <xsd:group ref="j2ee:descriptionGroup"/>
      <xsd:element name="distributable" type="j2ee:emptyType"/>
      <xsd:element name="context-param" type="j2ee:param-valueType">
        <xsd:annotation>
          <xsd:documentation> 
            The context-param element contains the declaration
            of a web application's servlet context
            initialization parameters.  
          </xsd:documentation>
        </xsd:annotation>
      </xsd:element>

      <xsd:element name="filter" type="j2ee:filterType"/>
      <xsd:element name="filter-mapping" type="j2ee:filter-mappingType"/>
      <xsd:element name="listener" type="j2ee:listenerType"/>
      <xsd:element name="servlet" type="j2ee:servletType"/>
      <xsd:element name="servlet-mapping" type="j2ee:servlet-mappingType"/>
      <xsd:element name="session-config" type="j2ee:session-configType"/>
      <xsd:element name="mime-mapping" type="j2ee:mime-mappingType"/>
      <xsd:element name="welcome-file-list" type="j2ee:welcome-file-listType"/>
      <xsd:element name="error-page" type="j2ee:error-pageType"/>
      <xsd:element name="jsp-config" type="j2ee:jsp-configType"/>
      <xsd:element name="security-constraint" type="j2ee:security-constraintType"/>
      <xsd:element name="login-config" type="j2ee:login-configType"/>
      <xsd:element name="security-role" type="j2ee:security-roleType"/>
      <xsd:group ref="j2ee:jndiEnvironmentRefsGroup"/>
      <xsd:element name="message-destination" type="j2ee:message-destinationType"/>
      <xsd:element name="locale-encoding-mapping-list" type="j2ee:locale-encoding-mapping-listType"/>
    </xsd:choice>

    <xsd:attribute name="version" type="j2ee:web-app-versionType" use="required"/>
    <xsd:attribute name="id" type="xsd:ID"/>
  </xsd:complexType>

Apache SSL + SVN notes

2010-10-22T23:05:00.000-04:00

Recently I started to use Apache http server again. I am trying to build a SVN repository which can be accessed through HTTPS.

Environment

Ubuntu
Apache 2.2.17 source

Doc: http://httpd.apache.org/docs/2.2/

Build

./configure --prefix=/home/gerald/servers/httpd-2.2.17 --enable-ssl --enable-dav --enable-so
make
make install

add bin directory to your PATH

add man pages:

function addManPath() { 
    if (($# != 1)); then return 0; fi

    path="$1" 
    if [ "x$MANPATH" == "x" ]; then 
        export MANPATH="$(manpath):$path" 
    else 
        export MANPATH="${MANPATH}:$path" 
    fi  
}

addManPath "~/servers/httpd-2.2.17/bin/man"

start up apache server: apachectl start

benchmarking: ab -n 10000 -c 100 http://localhost:80/

Show modules:
httpd -M //show all loaded modules
httpd -S      // show parsed virtual host settings
httpd -l    //listed compiled in modules
httpd -L   //list available configuration directives
httpd -V //show compile settings (not settings for compiling the whole package, the settings for compiling the server - httpd).

Configure SSL

Prepare your certificate and private key.
Uncomment line "Include conf/extra/httpd-ssl.conf" in httpd.conf.
Change file "conf/extra/httpd-ssl.conf". The most important directives are SSLCertificateFile and SSLCertificateKeyFile.
Test whether you can access your website through HTTPS.

SSL + SVN

Get modules dav_svn and authz_svn

wget http://altruistic.lbl.gov/mirrors/ubuntu/pool/universe/s/subversion/libapache2-svn_1.6.5dfsg-1ubuntu1_i386.deb

dpkg-deb -x libapache2-svn_1.6.5dfsg-1ubuntu1_i386.deb

copy two module (.so files) to apache modules directory.

Configure modules

Edit file <Apache>/conf/extra/dav_svn.load

LoadModule dav_svn_module modules/mod_dav_svn.so
LoadModule authz_svn_module modules/mod_authz_svn.so

Edit file <Apache>/conf/extra/httpd.conf, add following two lines

Include conf/extra/dav_svn.load
Include conf/extra/dav_svn.conf

Edit file <Apache>/conf/extra/dav_svn.conf

<Location /svn/> <!-- trailing / is necessary!! -->
  DAV svn
  
  SSLRequireSSL  # enforce use of HTTPS
  #SVNPath /var/lib/svn
  SVNParentPath /home/svn/projects
  SVNListParentPath on 
  
  AuthType Basic
  AuthName "Subversion Repository"
  AuthUserFile Apache_Dir/conf/dav_svn.passwd

  # To enable authorization via mod_authz_svn
  AuthzSVNAccessFile Apache_Dir/conf/dav_svn.authz

  Require valid-user
</Location>

http://stackoverflow.com/questions/488778/how-do-i-list-all-repositories-with-the-svnparentpath-directive-on-apachesvn

Create authentication and authorization files

Create password file: htpassword -cm <Apache>/conf/dav_svn.passwd gerald

Edit file <Apache>/conf/dav_svn.authz

[groups]
admin=gerald
guests=guest

[/]
@admin=rw

[repository_name:/directory]
@admin=rw

Test

Restart Apache httpd server.
Go to https://your_ip/svn/ (note: the trailing / is necessary!)

Permission Problem

If you see following error when you try to commit some code:

svn: Commit failed (details follow):
svn: Can't open file '/path/to/your/repo/db/txn-current-lock': Permission denied

follow these steps:

Execute command: ps -wwf $(pgrep httpd)
You should say one of the processes is run as root. All other processes are run as daemon (in my case).
To make httpd able to access(read/write) your svn repository, you should set the file permissions of svn repository correctly.
chown -R gerald:daemon /path/to/svn/repo
chmod -R 770 /path/to/svn/repo

Replace token using ant in Maven 2

2010-10-15T14:31:00.000-04:00

<plugin>
    <artifactid>maven-antrun-plugin</artifactid>
    <executions>
      <execution>
        <id>Copy and filter af file</id>
        <goals><goal>run</goal></goals>
        <phase>prepare-package</phase>
        <configuration>
          <tasks> 
            <copy file="source_file" filtering="true" failonerror="true" overwrite="true" tofile="dest_file">
              <filterset>
                <filter value="${variable_name}" token="token_to_be_replaced" />
                <filter value="value" token="token_to_be_replaced" />
               </filterset>
            </copy>
          </tasks>
        </configuration>
      </execution>
    </executions>
</plugin>

XAuth!

2010-09-15T13:16:00.001-04:00

Meebo and some other supporters just released XAuth. The video on page http://xauth.org/info/ is really informative. XAuth provides front-end solution for registration of various web service sessions.

If you are authenticated to a service, the service puts a registry entry into XAuth local storage.
Other mashup apps/publisher websites can ask XAuth for a list of web services that the user has been authenticated to. Then the app can adjust UI according to the retrieved data.

In current reference implementation, it requires HTML5 features – postMessage and local storage.

Javascript code: http://github.com/xauth/xauth
Official web site: http://xauth.org/

Numbers Everyone Should Know

2010-09-12T21:42:00.001-04:00

LADIS2009 keynote slides: http://www.cs.cornell.edu/projects/ladis2009/talks/dean-keynote-ladis2009.pdf

Following numbers are excerpted from the above slides.

L1 cache reference	0.5 ns
Branch mispredict	5 ns
L2 cache reference	7 ns
Mutex lock/unlock	25 ns
Main memory reference	100 ns
Compress 1K bytes with Zippy	3,000 ns
Send 2K bytes over 1 Gbps network	20,000 ns
Read 1 MB sequentially from memory	250,000 ns
Round trip within same datacenter	500,000 ns
Disk seek	10,000,000 ns
Read 1 MB sequentially from disk	20,000,000 ns

Clear memory cache in Linux

2010-06-26T01:20:00.001-04:00

Read this article

http://linux-mm.org/Drop_Caches

Basically, run commands
sync
sudo echo 3 > /proc/sys/vm/drop_caches

If you want to control "swappiness" (to what extent apps are swapped), read following posts:
http://kerneltrap.org/node/3000
http://lwn.net/Articles/92892/

Basic command: sudo echo 0-100 > /proc/sys/vm/swappiness

Steve Huffman talks about lessons learned at Reddit

2010-06-06T16:24:00.001-04:00

Very interesting talk. Helpful to those who want to build scalable distributed systems.

Video: http://vimeo.com/10506751
Transcript is here: http://carsonified.com/blog/dev/steve-huffman-on-lessons-learned-at-reddit/

One interesting lesson is lesson 3: open schema.
They combined relational data model and key-value store. Relational data model is powerful (in some sense) and can naturally represent real-world data models. However, as data set size grows, it seems that relational databases cannot scale up easily. As a result, many large companies develop their own storage systems - Amazon's Dynamo, Google's BigTable, Cassandra, etc. Some use key-value model which seems to be able to scale better than relational model. To change schema does NOT incur much overhead. Different rows/entities can have different number of columns/attributes.

Another post on caching in web app: http://www.mysqlperformanceblog.com/2010/05/19/beyond-great-cache-hit-ratio/

Create a file of a given size on linux

2010-03-27T00:56:00.000-04:00

Let's say you want to create a file whose size is 1K. You can achieve that by using either of following two commands:

    dd if=/dev/zero of=file bs=1024 count=1

    dd of=file bs=1024 count=0 seek=1

In my test, the generated files are different. The second command generates a file with holes. You can try command stat file. The output in my machine is:

  File: `bigfile'
  Size: 1024            Blocks: 0          IO Block: 4096   regular file

The number of allocated blocks is 0 although size of the file is 1024.
Note: if sparse file is not supported, it may write 1024K zeros to the file.

As a user, you don't need to worry about whether the file is sparse or not. It will grow as needed.

Ubuntu package downgrade

2010-03-25T03:42:00.000-04:00

I tried to use some Karmic sources for my Intrepid. Of course, this is bad. The reason I had to do it was the package of new version I needed only exist in Karmic repository.
After installing a package, suddenly I got the following error when I tried to use gvim
"gvim: error while loading shared library: libgailutil.so.18: cannot open shared object file: No such file or directory"

Obviously, the old libgail18 was removed which is needed by Intrepid. Because I used unmatched sources, the apt-get did not detect any problem.

I tried to install libgail18 using command
sudo apt-get install libgail18
It did not work and the error message is

Package libgail18 is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
However the following packages replace it:
  libgtk2.0-0
E: Package libgail18 has no installation candidate

Finally, I figured out the cause is package libgtk2.0-0, libgtk2.0-0-common, etc. These packages are too new for Intrepid. They are for Karmic. So I tried to remove these packages using command
    sudo apt-get remove libgtk2.0-0 libgtk2.0-0-common etc
However, apt-get always gave error messages which told me those packages were needed by lots of other packages. It is obviously true. Also I tried command
    sudo apt-get install --reinstall libgtk2.0-0
It also did not work. The error message is the package cannot be found. Again the cause I think was the those installed packages were too new and did not match version of ubuntu.

It turned out that I need to use dpkg command
    dpkg --remove –-depends libgtk2.0-0 libgtk2.0-0-common etc

Then use following command to fix the broken dependencies:
sudo apt-get install –f
libgtk2.0-0, libgtk2.0-0-common of correct versions are downloaded and installed.

Probably following commands are needed to clean up:
sudo dpkg --configure –a
dpkg-reconfigure

One big difficult I encountered was it was hard to collect detailed log about which files are created/updated/removed by which package during installation. It makes much easier to locate those packages which remove needed files.

SSH, XAuth and X11 Forward after user switch at remote site

2010-03-25T03:12:00.001-04:00

Problem

User A connects to server S using ssh. X11 forward is enabled using –Y option. Then X11 should work smoothly.
User A connects to server S using ssh. X11 forward is enabled. Then user A switches to another User B (using command su B or ssh localhost –l B). After that, X11 forward won't work. The error should look like
"SSH gateway: X11 authentication failed. Error: Can't open display:" or
"Error: Can't open display:".
Readers may ask why user A does not directly connect to server S as user B given user A knows password of user B. The reason is that sometimes user B is a restricted user account so that he cannot log in remotely.

Solution

Run command
echo ${DISPLAY}
Sample result:
localhost:11.0

Command: xauth list
The output should be like:

your_host_name/unix:11  MIT-MAGIC-COOKIE-1  d1e63de6fd7bc3800d868c3b64ca4531
your_host_name/unix:0  MIT-MAGIC-COOKIE-1  e044d47b672dcade1362cd632236f919
your_host_name/unix:10  MIT-MAGIC-COOKIE-1  2aa3bc47d1c209fd06577f4b45f83383

Pick the entry with the same display number as the output in step 1)
In this example, display number is 11, so the entry we pick is

your_host_name/unix:11  MIT-MAGIC-COOKIE-1  d1e63de6fd7bc3800d868c3b64ca4531

switch to another user using either of the following ways
1) su user_name
or su – user_name
read "man su" for difference between these two commands.
2) ssh localhost –l user_name
In step 3), if you ran command "su – user_name" or "ssh localhost –l user_name", you should run command
export DISPLAY=localhost:11.0
Value of DISPLAY should be the same as the output in step 1).

add the entry obtained in step 2) to the .Xauthority file. You can either add it manually to the file or use tool xauth to do it. The way to use xauth to add an entry:
xauth add :11 . d1e63de6fd7bc3800d868c3b64ca4531
The cookie string (long string) must match the one in step 2). The display number (:11) must match the result in step 1)
Try command
xclock

Or you can combine step 1), 2), 3) and 4) into one long command:

    (tmpfile=/tmp/xauth_tmp_entry; \
    xauth extract ${tmpfile} :$(echo $DISPLAY|cut -d : -f 2 ); \
    chmod a+r ${tmpfile}; \
    su user_name -c "xauth merge ${tmpfile}"; \
    rm ${tmpfile} )

Note: replace user_name with the real target user name.

Disadvantage

Each time the user reconnects the remote machine using ssh, the whole process described above must be redone :-( The reason is that sshd may choose another display number and cookie value.

How SSH X authorization works?

From ssh manual:

"ssh will also automatically set up Xauthority data on the server machine. For this purpose, it
will generate a random authorization cookie, store it in Xauthority on the server, and verify
that any forwarded connections carry this cookie and replace it by the real cookie when the
connection is opened. The real authentication cookie is never sent to the server machine (and
no cookies are sent in the plain)."

http://blogs.gnome.org/markmc/2005/02/25/ssh-x-forwarding-and-xauth/

Every time a user connects to a remote server using ssh, a proxy X server is created. And that X server is used by the sshd process (a new process is forked each time a new connection comes in). The process is like:

  user ---> server ---> fork a new process, 
                        create a proxy X server
                        create pseudo terminal, etc.
                                       |

                        a program that needs X is used

  local display <--- verify <--- the X output is forwarded by sshd to client

It seems that after a user connects to a remote server using ssh, another proxy X server is not created when the user ssh to localhost or 127.0.0.1.

Android Development Tips

2010-02-21T20:05:00.001-05:00

Logging

http://groups.google.com/group/android-ndk/browse_thread/thread/d6c5e8f25f1ee543

#include <android/log.h> 
__android_log_print(ANDROID_LOG_INFO, "your_tag", "fmt string", parameter1, parameter2);

Then you will see it in logcat output.

Read file build/platforms/android-1.5/common/include/android/log.h in ndk for more info.

Syntax Highlighter syntax conversion

2010-02-21T19:57:00.000-05:00

http://alexgorbatchev.com/wiki/SyntaxHighlighter

I found that usage pattern was changed in recent releases.

Original sample usage:

<pre class="java:nocontrols" name="code">
    your code
</pre>

New sample usage:

<pre class="brush:java">
    your code
</pre>

Change your posts to use new version of Syntax Highlighter:

Invoke following command in vim:
%s/<pre $.*$class="$\w\+$*$:\w\+$*" $.*$>/<pre class="brush:\2" \1 \4>/g

Google Blogger/Blogspot line break conversion

2010-02-21T19:55:00.000-05:00

Google Blogger/Blogspot new line conversion

Google blogger/blogspot provides an option to let users decide whether hard-returns in the post are converted to <br/>

Set it by clicking:
Your blog admin page –> Settings -> Formatting -> Convert line breaks

Originally, I set it to "yes". Latter I decided to manually insert <br/> instead of using automatical hard-return conversion. So I set the option to "no". As a result, my previous posts were messed up because different lines were concatenated into a single line.

I used vim to convert those posts to new format.

Append <br/> to each line
%s/$/<br\/>/g

Sometimes, you don't want to append new lines to block-level elements such as div, ol.
Use following vim commands:

    %s/\(<\/div>\s*\)\@<!$/<br\/>/g 
    %s/\(<\/ul>\s*\)\@<!$/<br\/>/g 
    %s/\(<\/ol>\s*\)\@<!$/<br\/>/g 
    %s/\(<\/dl>\s*\)\@<!$/<br\/>/g
    ......

Vim regular expression: http://vimdoc.sourceforge.net/htmldoc/pattern.html#pattern-overview

Another option is to write a script to download posts, adjust returns/newlines, and publish them.

Eucalyptus Code Reading

2010-01-11T02:29:00.002-05:00

Some notes on Eucalyptus source

Architecture
Clound Controller (CLC)
Cluster Controller (CC)
1. client
2. server
Storage Controller
Node Controller (NC)
1. Server
2. Client
Misc.

Architecture

See: http://open.eucalyptus.com/wiki/EucalyptusInstall_v1.6

Clound Controller (CLC)

The most complex component in terms of implementation and nuber of lines of code. Implemented in Java and code is located under directory *clc*. It seems it supports both ant and maven build.

Packages used:

mule

Modules:

authentication
bootstrap
Built to generate an executable named eucalyptus-cloud which can be used to start up the cloud.
cloud
cluster-manager
/edu/ucsb/eucalyptus/cloud/cluster
Allocator.java
ClusterAllocator.java
ClusterEndpoint.java
ClusterEnvelope.java

QueuedEvent.java
QueuedLogEvent.java

Reservation.java
Reservations.java
VmInstance.java
VmInstances.java

VmAllocationTransaction.java
VmTypeAvailability.java
VmTypeVerify.java
VmTypes.java

Callbacks
UnassignAddressCallback.java
AssignAddressCallback.java
ConfigureNetworkCallback.java
ConsoleOutputCallback.java
MultiClusterCallback.java
QueuedEventCallback.java
RebootCallback.java
StartNetworkCallback.java
StopNetworkCallback.java
TerminateCallback.java
VmRunCallback.java
VolumeAttachCallback.java
VolumeDetachCallback.java

/edu/ucsb/eucalyptus/cloud/cluster
AddressManager.java
CreateVmInstances.java
StateSnapshot.java
SystemState.java
VmAdmissionControl.java
VmControl.java
VmMetadata.java
VmReplyTransform.java

/com/eucalyptus/cluster
Cluster.java
ClusterBootstrapper.java
ClusterMessageQueue.java
ClusterNodeState.java
ClusterState.java
ClusterThreadGroup.java
Clusters.java
Networks.java

/com/eucalyptus/cluster/handlers
AbstractClusterMessageDispatcher.java
AddressStateHandler.java
ClusterCertificateHandler.java
LogStateHandler.java
NetworkStateHandler.java
ResourceStateHandler.java
VmStateHandler.java
configuration
core
edu/ucsb/eucalyptus/cloud/entities: Includes basic beans which represent various objects in the system (e.g. vmtype, user, bucket, image)
distribution
dns
group-manager
image-manager
interface
key-manager
msgs
com/eucalyptus/bootstrap:
Depends, Provides, Resource
Component: each component has a resource provider.
ResourceProvider
Bootstrapper
SystemBootstrapper: includes methods to load and start all bootstrappers contained in jars.
BootstrapFactory: provides static methods to initialize bootstrappers, configuration resources and resource providers.
com/eucalyptus/event
AbstractNamedRegistry.java
ClockTick.java: clock tick event
Event.java: event abstraction
EventListener.java: event listener abstration
GenericEvent.java: adds message to an event
ListenerRegistry.java: a event listener registry
ReentrantListenerRegistry.java
StateEvent.java: represents an event that has state.
StatefulNamedRegistry.java
SystemClock.java: periodically triggers clocktick event.
com/eucalyptus/util

edu/ucsb/eucalyptus/msgs/
Files under this directory defines various message types.
Configuration.groovy
DNS.groovy
EventRecord.java
Messages.groovy
StorageController.groovy
Unimplemented.groovy
VPN.groovy
VmAddresses.groovy
VmBlockDevice.groovy
VmControl.groovy
VmImages.groovy
VmKeys.groovy
VmLocation.groovy
VmNetwork.groovy
VmSecurity.groovy
Walrus.groovy
storage-common
edu/ucsb/eucalyptus/cloud/ws:
ChunkedDataFile.java
CompressedChunkedFile.java

edu/ucsb/eucalyptus/storage
StorageManager.java: interface for S3?

edu/ucsb/eucalyptus/storage/fs:
FileIO.java
FileReader.java
FileSystemStorageManager.java: impl of StorageManager.
FileWriter.java
storage-controller
com/eucalyptus/bootstrap
BlockStorageBootstrapper: bootstrappers for block storage.
edu/ucsb/eucalyptus/cloud/ws
BlockStorage: EBS (to read). It uses StorageManager from storage-common as underlying impl.
StorageEventListener
/edu/ucsb/eucalyptus/storage/
AOEManager.java: implements StorageExportManager
BlockStorageChecker.java
BlockStorageManagerFactory.java: factory for LVM2Manager.
LVM2Manager.java: impls LogicalStorageManager
LogicalStorageManager.java: interface for logical volume management
StorageExportManager.java: interface for export of volume
/edu/ucsb/eucalyptus/ic
StorageController.java: handles messages.
walrus (S3 impl)
edu/ucsb/eucalyptus/cloud/ws:
ObjectReader.java
TorrentClient.java
TorrentCreator.java
Torrents.java
Tracker.java
WalrusControl.java
WalrusEventListener.java: event listener. Currently, listens to StopComponentEvent.
WalrusImageManager.java
WalrusManager.java
WalrusBlockStorageManager.java
WalrusStatistics.java
edu/ucsb/eucalyptus/ic:
Walrus.java: handles messages.
wsstack
www

Cluster Controller (CC)

It interacts with two parts: cloud controller and node controllers. Its interface is described in file wsdl/eucalyptus_cc.wsdl. Axis2/C is used to generate C stub functions from wsdl.
It has two parts in the code: client and server. It provides services using SOAP-based web service.

client

It relies on

generated CC client stub functions from *wsdl/eucalyptus_cc.wsdl*

Code:

cluster/CCclient.c
cluster/cc-client-marshal-adb.c (main file)
cluster/cc-client-marshal.h

Client functions make use of stub functions generated from wsdl using axis2/C to construct SOAP request messages. The generated tool (by make) can be used to interact with cluster controller services.

server

When you build this part, a shared object is generated and deployed to Axis2/C environment. The axis2/C service name is "EucalyptusCC".
To fulfill the requests, sometimes it needs to interact with node controller (e.g. get information of the instances running on a specific node). So this part relies on generated stubs from both wsdl/eucalyptus_cc.wsdl and wsdl/eucalyptus_nc.wsdl. It acts as a server to provide services for external parties while it also acts as a service client to interact with other services.
So, it relies on

generated NC client stub functions *wsdl/eucalyptus_nc.wsdl*
generated CC server stub functions *wsdl/eucalyptus_cc.wsdl*

Code:

cluster/handlers.c
cluster/handlers.h
cluster/server-marshal.c
cluster/server-marshal.h

Functions in *handlers.c* do the real work when a service request is received. If necessary, it sends service requests to node controller.
File *server-marshal.c* extracts the request parameters using generated stub functions and invokes corresponding functions in file *handlers.c*. So it acts like a mediator which converts web service request to function name and parameters.

Storage Controller

Provides storage services for other parts. However, the form is not web service, just functions. Other parts directly call storage controller functions.
Code

storage/walrus.c, storage/walrus.h
Provides functions:
int walrus_object_by_url (const char * url, const char * outfile, const int do_compress);
int walrus_object_by_path (const char * path, const char * outfile, const int do_compress);
int walrus_image_by_manifest_url (const char * url, const char * outfile, const int do_compress);
int walrus_image_by_manifest_path (const char * manifest_path, const char * outfile, const int do_compress);
int walrus_verify_digest (const char * url, const char * digest_path);
These functions are used to interact with walrus services (using REST interfaces instead of SOAP-based interfaces).
storage/storage.c, storage/storage.h
provides higher-level abstraction which eases interaction of application with storage system.
It is used by Node Controller and Cluster controller(? it is included as header file, but seems none of its functions are used).

A executable named Wclient is generated when you "make all".

Node Controller (NC)

Node controller manages the instances running on a physical machine. It also provides services in the form of web services.
Its interface is described in file wsdl/eucalyptus_nc.wsdl

Server

Provides various services to other parts.
Code:

node/handlers.c
node/handlers_default.c
node/handlers.h
node/handlers_kvm.c
node/handlers_xen.c
node/server-marshal.c
node/server-marshal.h

Similar to implementation of cluster controller, files *handlers.c* does the real work to interact with virtual machine monitoring. Currently, both kvm and xen are supported. The concrete work involved to interact with a specific type of virtualization implementation is delegated to functions in file handlers_kvm.c, handlers_xen.c or handlers_default.c based on the user configuration. Under the hood, libvirt is used to interact with virtualization implementation.
server-marshal.c just extracts information from the incoming service requests and invokes corresponding functions in file *handlers.c*.
When a service needs to interact with storage system (e.g. download an image, create an image), it makes use of Storage Controller.

Client

Access services provided by NC server.
Code:

node/client-marshal-adb.c
node/client-marshal.h
node/client-marshal-local.c
node/NCclient.c

NCclient.c contains the main function. It parses the command and dispatch the requests to corresponding functions in file *client-marshal-adb.c* or *client-marshal-local.c*.
If the service is not local, client-marshal-adb.c is used. Basically the request is transformed to SOAP message and sent to remote NC service.
If the service is local, client-marshal-local.c is used. The corresponding implementation functions are directly invoked without hassle of web service stuff.

Misc.

network

See http://open.eucalyptus.com/wiki/EucalyptusNetworking_v1.6 for networking configuration.
In managed mode,
Manipulates the dhcpd configuration file and use "brctl" command from package *bridge-utils*. Also it supports VLAN.

gatherlog

Another sets of services described in file wsdl/eucalyptus_gl.wsdl. Code is located under directory *gatherlog*.

util

tools

Do you know Windows 7 GodMode?

2010-01-08T11:49:00.000-05:00

See this post for details:

http://www.osnews.com/story/22691/Activate_Windows_7_s_Hidden_God_Mode_

Just try:

Create a new folder
Name it: GodMode.{ED7BA470-8E54-465E-825C-99712043E01C}
That's it

Random Ubuntu console notes

2010-01-06T16:26:00.001-05:00

Turn off beeps: http://www.cyberciti.biz/faq/how-to-linux-disable-or-turn-off-beep-sound-for-terminal/

Framebuffer

http://tldp.org/HOWTO/Framebuffer-HOWTO.html

http://www.mat.univie.ac.at/~gerald/laptop/vesafb.txt

Change resolution: http://ubuntuforums.org/showthread.php?t=215566
http://www.mepis.org/node/2992
http://en.wikipedia.org/wiki/VESA_BIOS_Extensions#Linux_video_mode_numbers
http://www.linuxquestions.org/questions/ubuntu-63/console-session-very-large-text-font-598857/
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/246269
http://ubuntuforums.org/showthread.php?p=5400183

vga=ask to ask the user to choose mode

Solution: http://ubuntuforums.org/showthread.php?p=3826742

Setup console: http://ubuntuforums.org/showthread.php?t=329369&highlight=boot+console+font

console-setup

Install fonts, script and services

fonts are installed to /usr/share/consolefonts/
script: /bin/setupconm /usr/bin/ckbcomp
services: /etc/init.d/console-setup, /etc/init.d/keyboard-setup (they are installed to rcS.d)

Configuration:
1) $HOME/console-setup
2) /etc/default/console-setup
Note: if 1) exists, 2) will not be executed at all!!!

console-tools vs. kbd

console-tools provides consolechars. However consolechars cannot recognize some fontfaces provided by console-setup.
console-tools installs service console-screen.sh, makes dumpkeys process file /etc/console-tools/remap
update-rc.d console-screen remove
For example, I tried
consolechars –v –f /usr/share/consolefonts/Uni3-Terminus20x10.psf.gz --tty=/dev/tty5

It gives error

Cannot (yet) load a non-seekable RAW file
read_simple_font(): Invalid argument

I found this post http://www.mail-archive.com/debian-bugs-dist@lists.debian.org/msg154310.html

I installed kbd (sudo apt-get install kbd). console-tools is automatically removed when you install kbd. Then I tried

setfont –v /usr/share/consolefonts/Uni3-Terminus20x10.psf.gz

It works!!.

kbd installs bunch of commands,

configuration: /etc/kbd/config, /etc/kbd/remap
kbd installs service console-screen.kbd.sh (installed to rcS.d)

Note: for both of console-tools and kbd, the installed services won’t be run if setupcon is present. In other words, if console-setup is installed, services installed by console-tools or kbd don’t run.

So, usually you should edit file /etc/default/console-setup to change configuration!!! Or copy it to your home directory.
Then use command
setupcon
to make it take effect immediately.

http://www.robodesign.ro/mihai/blog/customize-your-linux-terminal

apt-get install hwinfo

sudo dpkg-reconfigure console-setup

consolechars

setupcon

apt-get remove pkgname

dpkg –purge pkgname

apt-get purge pkgname

Firefox private key/certificate import/export

2010-01-04T23:19:00.000-05:00

Recently I am dealing with X.509 related stuff. The browser I am using is Firefox 3.5.3.

Generation and Import

The steps of generating and importing private key and certificate consist of:

Public/private key pair generation
You can use standalone utility (e.g. openssl) to generate them.
Then you generate a self-signed certificate for the public key generated in the first step.
Also, you can generate a Certificate Request message and send it to an external CA to apply for a certificate.
Now you have both private key and the associated certificate.
You need to put both of them into a single pkcs12 file which can be recognized by Firefox.
Then you can import the pkcs12 file into Firefox by clicking
Tools –> Options –> Advanced –> Encryption –> View Certificates –> Your Certificates –> Import…

If you use javascript, you can use crypto object to generate them.

Use crypto.generateCRMFRequest to
generate a key pair and create a Certificate Request message
Send the generated Certificate Request message to remote server
“The string found by accessing crmfObject.request is the base-64 encoded CRMF message to be sent to the CA/RA”
After your certificate request is approved, you will get a public key certificate
You can import the certificate into your browser.

Note: In this case, you don’t need to import the private key because it is imported automatically when function crypto.generateCRMFRequest is called.

Export

Export your private key and certificate from Firefox:

Tools –> Options –> Advanced –> Encryption –> View Certificates –> Your Certificates –> Backup…
Note: the keystore format supported is pkcs12.

Export trusted certificates from Firefox:

Tools –> Options –> Advanced –> Encryption –> View Certificates –> {corresponding tab} –>Export
Note: Firefox supports couple of different formats including PEM, PKCS#7, etc.

Chinese Character Encoding

2010-01-04T15:13:00.001-05:00

GB2312, GBK, GB18030

GBK IANA registration page: http://www.iana.org/assignments/charset-reg/GBK
From the page, I finally know that windows codepage 936 corresponds to GBK.

JEE 6 has been approved!

2009-12-25T13:41:00.001-05:00

See the ballot here: http://jcp.org/en/jsr/results?id=5025

ASF voted against the proposal. The comment ASF gave is:

"The Apache Software Foundation's vote is based on the point of view that this spec lead - Sun - is in violation of the JSPA

http://www.apache.org/jcp/sunopenletter.html

and therefore shouldn't be allowed to lead other JSRs until the above matter is resolved.

This vote is not a comment on the technical merits of the JSR. If not for the issue of the spec lead, the ASF would have otherwise voted "yes".

I read http://www.apache.org/jcp/sunopenletter.html. It seems the dispute originates from license of JCK (Java Compatibility Kit) which is needed to demonstrate compatibility of Java SE spec. Also IBM voted for the proposal but stated that it wants an open licensing model, etc. Interesting relationships among those vendors and foundations.

The problem has been there for several years, and it still has not been solved.

A ball probability question

2009-12-24T16:39:00.001-05:00

I was looking for a solution to a ball probability question which is described below. I found this page which exactly describes the same question: http://www.isibang.ac.in/~statmath/problems/august.html. However, it seems the page is not accessible now. So I copied the page and pasted it here. If this incurs any term/license problem, I am more than glad to remove the content.

Suppose Farrah gives you two bags. Bag A, contains 50 Red balls and Bag B, contains 50 Black Balls. She leaves the room asking you to rearrange the balls between the bags in any way you want without discarding any of the balls. She then returns and chooses a bag at random and then from the chosen bag, chooses a ball at random.

Question: Can you find the rearrangement that maximises her chance of choosing a black ball ?

Solution provided by Winners :

Soumya bhattacharya
Clearly,if one bag contains only black balls and the other bag atleast one black ball along with the red balls,then the prabability that Farrah choses a black ball is more than half.

Again if any of the bags contain equal no. of black and red balls ,then the other bag also contains them in equal number and (if none of them is an empty bag,in which case the probability of Farrah's chosing a black ball is only 1/4) the probability that Farrah choses a black ball is half.As we already have seen an example in which the probability of chosing a black ball is more than half,hence this case can't be the solution.

So,if an optimal case exist,in that each of the bags must contain unequal numbers of black and red balls.And hence,(as the total no. of black balls= the total no. of the red balls) the proportion of black balls in a bag will be more than half in one bag and less than half,in the other. let 'b' and 'd' denote the total no. of balls in 2 bags and 'a' and 'c' denote the total no. of black balls in them respectively.

W.L.O.G.,let us assume that a/b < 1/2 and c/d > 1/2.Now if we take out a black and a red ball from the 2nd bag and put them in the first,the probability of chosing a black ball changes from (1/2)*(a/b + c/d) to (1/2)*((a+1)/(b+2) + (c-1)/ (d-2)). Again as a/b < 1/2,hence we have (a+1)/(b+2) > a/b. And as c/d > 1/2, (c-1)/(d-2) > c/d.

Therefore, by this rearrangement,the probability of chosing a black ball increases.So we do it untill there is no red ball in the bag,which was initially containing d balls.( as there were more black balls initially in this,and we shift equal no. of red and black balls to the other bag.) Now, clearly if we shift all the black balls,leaving only one in the first bag to the bag that contains red balls also,then the ratio of blacks to reds increases in the 2nd bag ,whereas that in the 1st bag remains constant, in effect increasing the probability of chosing a black ball by Farrah.

So,starting with any arbitrary case we have reached the state of rearrangement, which has the probability of chosing a black ball more than or equal to that arbitrary case.So, we conclude that ,the rearrangement of the balls in which one bag contains only 1 black ball and the other bag contains the rest of the balls is maximum (0.7474).

Subhroshekhar Ghosh

First of all, we claim that the required arrangement is : one black ball in one of the jars and all the other balls in the other jar, when the probability p of getting a black ball turns out to be 1/2*1 + 1/2 * 49/99.

To, see this, we eliminate the cases 1.all the balls are in one jar (p=1/4) 2.there are equal no. of red and black balls in one jar (and hence in the other jar) (p=1/2).

We observe that for any labelling of the jars as 1 and 2, p=1/2*p(1)+1/2*p(2) where p(i) is the conditional probability of getting a black ball having chosen the ith jar. Also, as there are equal no. of red and black balls, in any configuration there will be exactly one jar in which no. of red balls >= no. of black balls, hence the conditional probability for that jar <= 1/2. Since we have eliminated cases 1 and 2, we will have it < 1/2.

Without loss of generality, we call this jar 1 and let it have r red and b black balls. Then p(1)=b/(r+b), b0. We have p(1)<1/2, so we consider 1/2 - p(1) = 1/2 * {(r-b)/(r+b)}. The restrictions imply (r-b)>= 1 and r+b <=99. Therefore, 1/2 - p(1) >= 1/2 * 1/99, so p(1) <= 49/99, equality holds if there are 50 red and 49 black balls in jar 1. The other conditional probability p(2) <= 1, equality holds if all balls in jar 2 are black. Hence, p will be maximised if both of these upper bounds are attained. But they are attained by the configuration mentioned in the beginning. In fact, it is clear from the equality conditions that this configuration is the only one that attains the upper bound.

Interesting “Epigrams on Programming”

2009-12-22T16:13:00.000-05:00

Recently I found this

Epigrams on Programming: http://www.pam1.bcs.uwa.edu.au/~michaelw/Perlis_Epigrams.html

Really interesting, terse and helpful. Among those epigrams, I like following ones a lot

Programming is an unnatural act.
Computer Science is embarrassed by the computer.
In man-machine symbiosis, it is man who must adjust: The machines can't.

Simplest RFC?

2009-11-07T15:57:00.000-05:00

Which is the simplest RFC? Maybe this one: http://tools.ietf.org/html/rfc863

DISCARD: the server just discards all received messages.

Windows7 7 Sins

2009-11-06T22:57:00.001-05:00

An interesting web site: http://en.windows7sins.org

Classloaders, shared directory and endorsed directory of Tomcat 5.5 and 6.0

2009-09-26T14:28:00.000-04:00

Tomcat 5.5 Class Loader: http://tomcat.apache.org/tomcat-5.5-doc/class-loader-howto.html
Tomcat 6.0 Class Loader: http://tomcat.apache.org/tomcat-6.0-doc/class-loader-howto.html

Tomcat 5.5

Class loader hierarchy
(from tomcat 5.5 document)

Class loaders:

Boostrap, System
Common:
visible to both Tomcat internal classes and to all web applications.
Related directories
- classes and resources in: $CATALINA_HOME/common/classes
- jar files in: $CATALINA_HOME/commons/endorsed, $CATALINA_HOME/commons/i18n and $CATALINA_HOME/common/lib
Catalina
include all classes and resources required to implement Tomcat 5 itself. These classes and resources are TOTALLY invisible to web applications.
Related directories:
- classes and resources in: $CATALINA_HOME/server/classes
- jars in: $CATALINA_HOME/server/lib
Shared
Shared resources are shared across all web applications. (They are not used by Tomcat internal classes).
Related directories
- classes and resources in: $CATALINA_BASE/shared/classes
- jars in: $CATALINA_BASE/shared/lib
WebappX
Load web app specific resources and jars.
Related directories
- classes and resources: <web-app-dir>/WEB-INF/classes
- jars: <web-app-dir>/WEB-INF/lib

Note: $CATALINA_BASE (not $CATALINA_HOME) is used class loader “Shared”.

#   CATALINA_HOME   May point at your Catalina "build" directory.
#
#   CATALINA_BASE   (Optional) Base directory for resolving dynamic portions
#                   of a Catalina installation. If not present, resolves to
#                   the same directory that CATALINA_HOME points to.

Tomcat 6.0

Class loader hierarchy
(from tomcat 6.6 document)
Similar to the hierarchy in tomcat 5.5.
The difference is that class loaders Catalina and Shared are not present in tomcat 6.6.
For Class loader Common, the related directories are different. In tomcat 6, classes, resources and jars under directory $CATALINA_HOME/lib (in tomcat 5.5, the directory is $CATALINA_HOME/common/classes, $CATALINA_HOME/common/lib, etc) are loaded by class loader Common.

These directories under which classes and resources are searched for can be changed by modifying configuration file $CATALINA_HOME/conf/catalina.properties.
Default values for class loader search directories are:

common.loader=${catalina.home}/lib,${catalina.home}/lib/*.jar
server.loader=
:shared.loader=

If you still want to use shared directory in the way specified in tomcat 5.5, you can change value of property shared.loader.
For example: shared.loader=${catalina.base}/shared/classes,${catalina.base}/shared/lib/*.jar

Endorsed directory

Tomcat 5.5

“Tomcat utilizes this mechanism by including the system property setting
-Djava.endorsed.dirs=$CATALINA_HOME/common/endorsed in the command line that starts the container. Therefore, you can replace the parser that is installed in this directory, and it will get used even on a JDK 1.4 system.”

Tomcat 6.0

Tomcat utilizes this mechanism by including the system property setting
-Djava.endorsed.dirs=$JAVA_ENDORSED_DIRS in the command line that starts the container.

Variable JAVA_ENDORSED_DIRS is specified in file setclasspath.sh:
JAVA_ENDORSED_DIRS="$BASEDIR"/endorsed

Variable BASEDIR is specified in file catalina.sh:
BASEDIR="$CATALINA_HOME"

In other words,
in tomcat 5.5, the default endorsed directory is $CATALINA_HOME/common/endorsed
in tomcat 6.0, the default endorsed directory is $CATALINA_HOME/endorsed

Apache ten years

2009-09-24T23:11:00.000-04:00

Some really nice articles/interviews about Apache Software Foundation:
http://dev.day.com/microsling/content/blogs/main/apachebirthday.html
ASF highlights: 1999-2009 http://www.apache.org/press/highlights.html
Details about how ASF works: http://www.apache.org/foundation/how-it-works.html
eWeek: 11 Apache Technologies that Have Changed Computing in the Last 10 Years
http://www.eweek.com/c/a/Application-Development/11-Apache-Technologies-that-Have-Changed-Computing-in-the-Last-10-Years-469693/

SVN: exclude some files out of version control (using svn:ignore)

2009-09-12T14:09:00.000-04:00

Sometimes, you don’t want SVN to include all you files in the source code directory because they may be temporary files generated by developers or tools (vimcreates .swp file when a file is opened in vim).
SVN book: http://svnbook.red-bean.com/en/1.5/svn.advanced.props.special.ignore.html

Two ways

o Global change: it is applied to all svn operations performed using the runtime configuration. In other words, it changes the configuration of svn application (not a specific directory/file under control of svn)

o Bind configuration to a specific version tree using svn:ignore property.

o The ignore pattern checking is applied only during adding of unversioned directories/files to svn control.
“Once an object is under Subversion's control, the ignore pattern mechanisms no longer apply to it. In other words, don't expect Subversion to avoid committing changes you've made to a versioned file”

o Its value contains a list of NEWLINE-delimited file pattern

o They don’t override global global-ignores configuration, but append to the list

o “the patterns found in the svn:ignore property apply only to the directory on which that property is set, and not to any of its subdirectories.”
Use switch –R if you want to apply it recursively.

o   Usage
svn propset svn:ignore -F file-contains-ignore-patterns path-ignore-to-be-applied
Examples:
    svn propset svn:ignore -F .svn-ignore .
    #recursive. If you change the direcotry tree after this command is run,
    #you need to return following command to make the newly created directories/files controlled also by ignore pattern.
    svn -R propset svn:ignore -F .svn-ignore .
    # edit a property
    svn propedit svn:ignore .     #edit the ignore patterns using the preconfigured editor.
Note:
*) If you use a file to specify ignore patterns, you must rerun the command
              "svn propset svn:ignore -F your-patter-file ."
    to make svn reread the file after the pattern file is changed.
*) One alternative is to use
              "svn propedit svn:ignore ."
       to edit the property configuration. Then the change you made immediately takes effect. However, the new property value will not be written back to the original pattern file. You can use command
    "svn propget svn:ignore . > pattern-file"
to export the new value to your pattern file.

o File Pattern
*: match any sequence of characters
?: match single character
[char-set]: match the specified char set.

o Command “svn add”
When you use wildcards, you should be careful because it may make svn bypass the ignores check.
See http://svnbook.red-bean.com/trac/ticket/115.
Basically, “svn add *” bypasses the ignores system. However, it has been fixed. So if you are using a latest svn version, it should not be a problem.
Anyway, use “svn add --force .” when you can (it makes svn check the whole specified directory tree).

o Command “svn status” won’t list ignored directories/files.
Use “svn status --no-ignore” to list status of all files including ignored ones.

test

2009-09-08T22:10:00.001-04:00

Hello, worl

2009-08-20T19:46:00.000-04:00

test post from linux using blogtk 1.1

Install local .deb packages

2009-07-09T17:31:00.000-04:00

Ususally, apt-get is used to install/update/remove packages. It is preferred because it can automatically resolve package dependency.

However, sometimes you must download a .deb package file and install it from local disk. dpkg can do this task.
dpkg -i package_name_here
However, dpkg does NOT resolve package dependency. So it is likely that the installed program won't work.
You can fix it by using command
apt-get -f install

Another solution is documented here:
http://www.debian.org/doc/manuals/apt-howto/ch-basico.en.html#s-dpkg-scanpackages.
The idea is to build a tarball (Packages.gz) that can be recognized by apt tools. Then you can install the program by using apt-get command. (In other words, .deb does not contain enough information which is required by apt tools).

Resources
APT HOWTO: http://www.debian.org/doc/manuals/apt-howto/

Some linux notes about hardware inspection

2009-07-09T15:05:00.001-04:00

PCI (Peripheral Component Interconnec)

PCI is the dominant bus standard. In linux, its device files are located at /proc/bus/pci.
Command lspci can be used to list your PCI devices. It can display <[domain id]:bus id:device id.function number> in addition to device type. You can use -v switch to get verbose information. Switch -k is useful to get corresponding kernel modules.
Resource: http://tldp.org/LDP/tlk/dd/pci.html

DMI (Desktop Management Interface)

From wiki

"Essentially, to the user, it is a table provided by the personal computer BIOS which can be parsed and which gives information about the BIOS and the computer system in a standardized way."

dmidecode is a tool on linux that can be used to get computer information (motherboard, BIOS, ...)via DMI.
DMI is part of SMBIOS.
Resources: http://en.wikipedia.org/wiki/Desktop_Management_Interface

USB

Command lsusb can be used to list your USB devices.

Resources

TLDP (The linux document project: http://en.tldp.org/): This web site contains a lots of useful HOWTO articles. Also it has in-depth articiles to introduce internal knowledge of linux kernel. Almost every aspect of linux is included.
A great PCI article: http://tldp.org/LDP/tlk/dd/pci.html
RedHat enterprise linux 5 Release Notes: http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5.2/html/Release_Notes/singles/relnotesU2-x86.html
RedHat linux deployment guide: http://www.redhat.com/docs/manuals/enterprise/RHEL-5-manual/Deployment_Guide-en-US/index.html
Although it contains some RedHat linux specific stuff, you still can find lots of common knowledge that is shared by other linux distros.
Linux devide list: http://www.lanana.org/docs/device-list/
Linux Foundation: Its publication includes some interesting papers which can be accessed http://www.linuxfoundation.org/collaborate/publications/.

Synergy bugs - left and down keys don't repeat

2009-07-09T14:38:00.001-04:00

I am using Synergy to share keyboard and mouse between two machines.

However, the left and down keys don't repeat. Here is the reported bug:
https://bugs.launchpad.net/ubuntu/+source/synergy/+bug/281546
Also a workaround is given (https://bugs.launchpad.net/ubuntu/+source/synergy/+bug/281546/comments/5).
The idea is basically to set the two keys 'autorepeat' using following commands:

xset r 116 #for down key
xset r 113 #for left key

You should replace with keycodes of your keyboard.
Some information about this bug is included in http://ubuntuforums.org/showthread.php?t=965393.

Non-Ascii characters in MIME headers

2009-03-01T19:01:00.000-05:00

Message Header Extensions for Non-ASCII Text: http://tools.ietf.org/html/rfc2047
The text is encoded in this ways:

   encoded-word = "=?" charset "?" encoding "?" encoded-text "?="
   charset = token    ; 
   encoding = token   ; 
   token = 1*<Any CHAR except SPACE, CTLs, and especials>
   especials = "(" / ")" / "<" / ">" / "@" / "," / ";" / ":" / "
               <"> / "/" / "[" / "]" / "?" / "." / "="
   encoded-text = 1*<Any printable ASCII character other than "?" or SPACE>

Supported encodings include "Q"(similar to quoted-printable) and "B" (Base64).

For example,

=?gb2312?Q?[CSDN]=C2=DB=CC=B3=D3=C3=BB=A7=BC=A4=BB=EE=CD=A8=D6=AA?=

represents text "[CSDN]论坛用户激活通知".
Note: SPACE and TAB are not allowed in encoded text. SPACE must be encoded using "=20".

How to organize scientific papers?

2009-03-01T14:38:00.000-05:00

I have hundreds of papers to organize and the number is increasing. And then I found this post http://stochastix.wordpress.com/2007/10/02/how-to-organize-scientific-papers/ which is quite helpful.

Tools:
Zotero: http://www.zotero.org/ It is a FireFox extension.
Jabref: http://jabref.sourceforge.net/ open source reference manager
CiteULike: http://www.citeulike.org online reference manager
Evernote: Online note management (www.evernote.com)

Commercial ones
Reference Manager
EndNote
...

Vim advanced topics

2009-01-30T19:48:00.000-05:00

As I am using vim frequently, gradually I find it necessary to grasp some advanced vim topics. Sometimes, search engines don't return the useful posts/resources in the first page.

Vim built-in functions, internal variables, data structures, etc. http://vimdoc.sourceforge.net/htmldoc/eval.html
Filetype http://www.vim.org/htmldoc/filetype.html
Command line mode http://vimdoc.sourceforge.net/htmldoc/cmdline.html
Options and commands:
http://vimdoc.sourceforge.net/htmldoc/options.html
http://vimdoc.sourceforge.net/htmldoc/various.html
Configuration: http://peox.net/articles/vimconfig.html
Other topics (FAQs): http://www.cs.utah.edu/~kad/vim/vimnotes.shtml#17
Vim Buffer Resources
http://www.vim.org/htmldoc/windows.html
http://vim.wikia.com/wiki/Vim_buffer_FAQ

File manipulation commands:

Information getting
Ctrl+g, :f, :file	print some info of current file in command line
:ls, :buffers, :files	list
{count}Ctrl+g	1Ctrl+g: print current file name with full path 2Ctrl+g: currentbuffer number is printed also
g Ctrl+g	print cursor position in 5 ways: Column, Line, Word, Character and Byte.
{Visual} g Ctrl+g	print Word, Character, Line, and Byte counts of selected text
Editing switch
:view {file}	same as :edit, except the file is opened in readonly mode.
:file name	set current file name to name. The old name becomes alternate file name.
:0file	Remove the name of the current buffer.
:edit[!]	Re-load current file. Useful when the file is modified outside of vim.
:edit[!] {file}	Edit a new file.
Ctrl + ^	Edit alternate file(equivalent to ":e #").By default, the alternate file is the previously edited file
:{count} Ctrl + ^	go to a specific buffer
:edit #[count]	Edit the [count]th buffer
:b[uffer] [count]	Edit the [count]th buffer
:enew[!]	edit a new unnamed buffer
Find and Edit
:find[!] {file}	Find {file} in path and then edit
:{count}find[!] {file}	Find {count}-th {file} in path and then edit
[count] gf	Edit the [count]-th file in path whose name is under the cursor. If [count] is omitted, just get the first match. The file in opened in current window.
[count] Ctrl-w gf	Edit the [count]-th file in path whose name is under the cursor. It is opened in a new tab.
[count] Ctrl-w f	Edit the [count]-th file in path whose name is under the cursor. It is opened in a newly split window in current tab.
{Visual}[count]gf	selected text is used as filename to search
[count]gF	Similar to the commands above. If a number follows the filename, after opening the file, cursor would be moved to the specified position.
{Visual}[count]gF
Writing
:w[rite][!]	write current buffer to current file.
:{range}w[rite][!]	write specified lines to current file
:{range}w[rite][!] {file}	write specified lines (or whole buffer) to {file}
:{range}w[rite][!] >>	append specified lines to current file
:{range}w[rite][!] >> {file}	append specified lines to {file}
:[range]w[rite] !{cmd}	feed specified lines into {cmd} as standard input
Insert a file
:r[ead] [file]	insert file {file} after the cursor
:{range}r[ead] {name}	insert file {file} below the specified line
:[range]r[ead] !{cmd}	Execute {cmd} and insert its standard output after the cursor or below the specified line. E.g. :read !ls

Window manipulation

Command	Explanation
Ctrl-w s, Ctrl-w Ctrl-s :[N]sp[lit] [file]	Split current window into two horizontally.Option splitbelow controls where to put newly created window.
Ctrl-w v, Ctrl-w Ctrl-v :[N]vsp[lit] [file]	Split current window into two vertically. Option splitright controls where to put newly created window.
Ctrl-w n, Ctrl-w Ctrl-n :[N]new [file]	Create a new window with an empty file in it or with the specified file. split horizontally
:[N]vnew [file]	Create a new window with an empty file in it or with the specified file. split vertically.
:[N]sv[iew] {file}	same as :split, but set 'readonly' option
:[N]sf[find] {file}	same as :split, search for {file} in path.
Ctrl-w Ctrl-^, Ctrl-w ^	Does ":split #". Split window into two and edit alternate file. When a count is given, it is equivalent to ":split #N". Split and edit buffer N
Explictly set position of window
:vertical {cmd}	Explicitly set position of a newly created window which is generated by invoking {cmd}
:leftabove {cmd} :aboveleft {cmd}
:rightbelow {cmd} :belowright {cmd}
:topleft
:botright
Cursor movement and window movement
Ctrl-w k Ctrl-w j Ctrl-w h Ctrl-w l	Move cursor to Nth window above current one. Move cursor to Nth window below current one Move cursor to Nth window left of current one Move cursor to Nth window right of current one
Ctrl-w w Ctrl-w W Ctrl-w t Ctrl-w b Ctrl-w p	move cursor to window below/right of current one. move cursor to window above/left of current one. move cursor to top-left window move cursor to bottom-right window move cursor to previous window
Ctrl-w K Ctrl-w J Ctrl-w H Ctrl-w L	move window to be at the very top, with full width of screen at the very bottom, with full width of screen at the far left, with full height of screen at the far right, with full height of screen
Ctrl-w r Ctrl-w R	rotate windows downwards/rightwards upwards, leftwards
Ctrl-w x	exchange current window with next one or with Nth window if count is given. The two exchanged windows must be located in the same row or column.
Ctrl-w T	Move current window to a new tab

Buffer
A buffer is a file loaded into memory for editing.
Unlisted buffer: It means it exists, but it is not listed in buffer list.
active buffer: (1) displayed in a window (2) loaded
hidden buffer: (1) is not displayed in a window, (2) loaded into memory
inactive buffer:(1) is not displayed (2) not loaded (3) Options are remembered.
unlisted buffer: some buffers are not listed in buffer list. when you apply :bdelete or :bwipeout to a buffer, it would be deleted from buffer list.
hidden buffer -> unhidden buffer: start to edit it with any command or delete it with :bdelete command.

:q[uit] [!] quit current window. The buffer becomes inactive.
For following three commands, modified buffers are never abandoned (never becomes inactive).
:close [!]   close current window. (1) if 'hidden' is set, the buffer becomes hidden (2) else abandon it (becomes inactive).
:hid[e]      Quite current window and state of the buffer depends on option bufhidden.
:only[!]     Make the current window the only one in current tab. All other windows are closed.

Direct buffer editing command

# Remove association between a buffer and a file. The buffer is now just a buffer without any content.
:bd[elete][!] [N]      Unload buffer [N] (default: current buffer) and delete it from buffer list.
:bdelete[!] {bufname}  Unload and delete buffer corresponding to {bufname}
:bdelete[!] N1 N2 ...  Unload and delete buffer N1, N2, ...
:N,Mbdelete[!]         Unload and delete all buffers in range N to M

# The buffer is removed thoroughly which means the buffer does not exist any more.

:[N]bwipout[!]             Like :bdelete, but really delete the buffer. 
:bw[ipeout][!] {bufname}   Everything releated to the buffer is lost.
:N,Mbw[ipeout][!]
:bw[ipeout][!] N1 N2 ...

# memory is freed. When you switch to that buffer later, the content would be read from the file.
:[N]bun[load][!]       The memory allocated for those specified
:bun[load][!] [N]      buffers is freed. But the buffers remain in the
:bunload[!]{bufname}   buffer list.
:N,Mbunload[!]
:bunload[!] N1 N2 ...

Important options:
hidden: off - a buffer is unloaded when it is abandoned.
                   When (1) the buffer is modified and
                            (2) 'autowrite' is off or writing is not possible
                            (3) the '!' flag was used.
                   a buffer becomes hidden even if hidden option is off.
            on - a buffer becomes hidden when it is abandoned.
bufhidden: what happens when a buffer becomes hidden.
    <empty> - follow hidden option
    hide - the buffer becomes hidden (not unloaded)
    unload - unload the buffer
    delete - delete the buffer from buffer list (like using :bdelete)
    wipe - wipe out buffer from buffer list (like using :bwipeout)
When "unload", "delete" or "wipe" is used, changes in a buffer are load without any warning.

Usually, if option hidden is on, the operations of buffer manipulation commands depend on option bufhidden.
If option hidden is off, buffer is unloaded

A list of buffer/arg switch command from vim help doc:

args list		       buffer list	   meaning ~
1. :[N]argument [N]	11. :[N]buffer [N]	to arg/buf N
2. :[N]next [file ..]	12. :[N]bnext [N]	to Nth next arg/buf
3. :[N]Next [N]		13. :[N]bNext [N]	to Nth previous arg/buf
4. :[N]previous	[N]	14. :[N]bprevious [N]	to Nth previous arg/buf
5. :rewind / :first	15. :brewind / :bfirst	to first arg/buf
6. :last		16. :blast		to last arg/buf
7. :all			17. :ball		edit all args/buffers
			18. :unhide		edit all loaded buffers
			19. :[N]bmod [N]	to Nth modified buf

  split & args list	  split & buffer list	   meaning ~
21. :[N]sargument [N]   31. :[N]sbuffer [N]	split + to arg/buf N
22. :[N]snext [file ..] 32. :[N]sbnext [N]      split + to Nth next arg/buf
23. :[N]sNext [N]       33. :[N]sbNext [N]      split + to Nth previous arg/buf
24. :[N]sprevious [N]   34. :[N]sbprevious [N]  split + to Nth previous arg/buf
25. :srewind / :sfirst	35. :sbrewind / :sbfirst split + to first arg/buf
26. :slast		36. :sblast		split + to last arg/buf
27. :sall		37. :sball		edit all args/buffers
			38. :sunhide		edit all loaded buffers
			39. :[N]sbmod [N]	split + to Nth modified buf

40. :args		list of arguments
41. :buffers		list of buffers

More Misc. commands
:scriptnames    a list of sourced script files in the order they were first sourced.
:verbose set [option]
:verbose map
:echo g:colors_name
:Ctrl-w T
:read !{cmd}
:tab {cmd}    execute the external command and open the result file in a new tab. E.g. :tab split
:tab split filename
:tabedit filename
:tab ball    open all buffers, each in a separate tab page.
:tabnew |r !dir    execute command dir and redirect its output to a file opened in a new tab page.
:tabnew |:E
:echo functionname(argument) call an internal function and returns the output
:call functionname(argument)    call an internal function and returned value is discarded.
:exe string_expr    execute the commands contained in the string in Ex mode.

Windows file association
For example, if you want to associate .txt files with vim, use following commands:

	ftype VIM="e:\program files\vim\vim72\gvim.exe" --remote-tab-silent "%1"
	assoc .txt=VIM

Replace the path with your own vim installation directory..
If you want to maximize the window, use
ftype VIM="e:\program files\vim\vim72\gvim.exe" -c "simalt ~x" --remote-tab-silent "%1"

More VIM text editing command

2009-01-26T22:39:00.000-05:00

The more I use VIM, the more it surprises me.
Here, I list some useful commands in everyday text editing. I have known some of them for long time. But others were learnt recently.

Command	Explanation	Note
Delete
cc	delete current lines and change to insert mode
dd	delete current lines
c{motion}	delete until the character {motion} moves to and change to insert mode.
d{motion}	delete until the character {motion} moves to
C	delete characters until the end of the line. Then change to insert mode
D	delete characters until the end of the line.
s	delete [count] characters and change to insert mode
x	delete [count] characters under and after the cursor
:[range]d	delete those lines in the range
S	delete [count] lines and change to insert mode
Replace
R	Enter Replace mode
gR
Line joining
[range]J	join [count] lines	Steps of execution of these commands (1) delete leading white space on the next line, (2) if there is trailing white space for current line the next line starts with a ')', delete <EOL> else if the 'joinspaces' option is on and current line ends up with a '.', '!' or '?' replace <EOL> with two spaces else replace <EOL> with one space
{Visual}J	join the selected lines
:[range]j[!]	join range lines. With !, the join doesn't insert/remove any spaces.
[range]gJ	join [count] lines	Don't insert or delete any spaces.
{VIsual}gJ	join the selected lines	Don't insert or delete any spaces.

Some commands useful in source code editing

Commands	Explanation
<{motion}	shift covered lines leftwards by 1 shiftwidth
>{motion}	shift covered lines rightwards by 1 shiftwidth
{Visual}[count]<	shift selected lines leftwards by 1 shiftwidth
{Visual}[count]>	shift selected lines rightwards by 1 shiftwidth
[range]<	shift selected lines leftwards by 1 shiftwidth
[range]>	shift selected lines rightwards by 1 shiftwidth
<<	shit current line leftwards by 1 shiftwidth
>>	shit current line rightwards by 1 shiftwidth

Resource
http://www.vim.org/htmldoc/change.html

Text formatting in VIM

2009-01-26T13:39:00.000-05:00

How to set options for VIM?

using set command.
Examples:

Setting	Explanation
set paste	set options for commands that don't need parameters.
set filetype=java	Directly set options for commands that accept parameters.
set guioptions-=option	Remove a specific option from the configuration of a command.
set guioptions+=option	Add a specific option to the configuration of a command.

Basic concept
Sentence/Paragraph/Section: http://www.vim.org/htmldoc/motion.html#sentence

Text format
http://www.vim.org/htmldoc/change.html#formatting
Alignment

Setting	Explanation	Examples
`:[range]ce[nter] [width]`	center lines	:.,+3 center 80
`:[range]ri[ght] [width]`	right alignment	:% right
`:[range]le[ft] [indent]`	left alignment.(Unit of identation is space)	:left 4

Format

Command	Explanation	Examples	Note
gq{motion}	Format lines that {motion} moves over. Cursor is put where {motion} moves to	gqap format a paragraph	Motion: http://www.vim.org/ htmldoc/motion.html
gw{motion}	Format lines that {motion} moves over. Put cursor back to the original position.	gwj	Motion: http://www.vim.org/ htmldoc/motion.html
gqgq gqq	Format current line
{Visual}gq	Format highlighted text

How to control formatting setting of gq command?
Formatting does not change empty lines, but it changes lines only containing white spaces.

(1) textwidth
This option controls length of every formatted line.
If option textwidth is 0, the formatted line length is the screen width (max value is 79).

(2) autoindent
If this option is on, Vim uses indent of the first line for the following lines.

(3) joinspaces (boolean)
Used when lines are joined together. By default, it is on. Use set nojoinspaces to turn it off.

(4) formatprg
Set an external program to format text. In this case, the textwidth and other options have no effect on text formatting.

(5) formatoptions
Most useful options:
t: Auto-wrap text using textwidth. (does not apply to comments)
c: Auto-wrap commens using textwidth.
q: Allow formatting of comments with gq command. When using gq command in comments, blank lines and lines only with comment leaders and white spaces are considered as paragraph delimiters.
r: auto insert comment leader after hitting 'Enter' in Insert mode.
o: auto insert comment leader after hitting 'o' or 'O'.
a: auto format paragraphs when text is inserted or deleted.
w: A trailing non white space ends a paragraph.
Note: Formatting would be applied anyway no matter whether the paragraphs are delimited correctly.
You can use set formatoptions-=a to disable automatic formatting.
More options: http://www.vim.org/htmldoc/change.html#fo-table

(6) comments
This option controls how to format comments (usually in source code).
Default value is

"s1:/*,mb:*,ex:*/,://,b:#,:%,:XCOMM,n:>,fb:-"

s: start of a three-piece comment
s{digit}: add extra indent to middle part. Unit is space (not tab).
s-{digit}: remove extra indent to middle part. Unit is space (not tab).
m: middle part of a three-piece comment. Middle part is left adjusted with start part of the comment by default. This is controlled by option l.
b: blanks required after the specified string.
e: end of a three-piece comment
x: In C++ comment, just type / to end a comment when middle-comment string is inserted. The space between middle-part and / would be removed automatically.
n: recognize numbered list(see below for details)
More options: http://www.vim.org/htmldoc/change.html#format-comments
My setting is

set textwidth=79
set formatoptions=tcqron

How to format numbered lists?
This sometimes is really tricky.
See this post: http://objectmix.com/editors/332035-vim-how-create-bullet-list-numbered-list.html
Usually, textwidth must be set and formatoptions must include nwt.
Also see option formatlistpat(flp) for how to set list header. That is used by vim to recognize numbered lists.

MyProxy and proxy certificate

2009-01-22T16:46:00.000-05:00

All of following discussion is based on PKI (public key infrastructure).

What's important in terms of security?
integrity of the user's identity credential(certificate + public/private key pair).
Note: What is important is not just a single part of the credential, but also integrity of different parts. If a certain component is compromised, the whole credential is not secure any more.

how to protect private key?
(1) Store it in desktop
May not be a good idea because usaually desktop is not protected professionally. It is vulnerable to keystrok loggers, trojan ...
(2) Smart card
A specific piece of hardware provides improved security for key storage.
Pros: allows keys to be used for signing and encryption without allowing the keys to be exported from the card's protected memory.
Cons: cost and support
(3) Virtual smart card
Similar to the physical smart card approach. It is a software system which protects private keys on secure servers without allowing the keys to be exported.

Grid computing
Usually, the user has a certificate issued by a CA and it has long life time (maybe several years). Before successul running of a task, the certificate must be authenticated by grid infrastructure. Tasks may be submitted and launched from different hosts.
(*) So it is not convenient to ask users to copy their credentials manually across different machines.
(*) It is not secure to put long-term credentials to many hosts in the grid.
It would be great if users can obtain a short-time(hours or days) credential which can be used for authentication during execution of a task.
There are two solutions:
(1) Online CA
Every time the user needs a short-time credential, he/she just sends certificate request to CA and CA returns the newly created certificate.
Drawback: cost. Adding a new CA needs a lot of extra work in terms of time, money, negotiating policies...
(2) Online credential repository
This solution makes use of X.509 proxy certificate described below.
MyProxy uses the third approach.

Proxy Certificate (RFC 3820)
Key idea: An entity delegates its rights or a subset of its rights to another entity which can behave on behalf of the original entity.
Security
Restrictions can be placed on the PC be means of policies.
Terminology
EEC(End Entity Certificate): an X.509 Public Key Certificate issued to an end entity by a CA.
PC(Proxy Certificate): can be issued by EEC or another PC.
PI(Proxy Issuer): an entity with an EEC or PC that issues a PC. The issued PC is signed using the corresponding private key.
Properties of PC
(*) Signed by EEC or PC
(*) It can sign another PC. Cannot sign EEC
(*) Its public/private key pair is different from that of the proxy issuer.
(*) Its identity is derived from identity of the original ECC that signed the PC.
Note: its identity is unique so that it can be used as an independent identity.
(*) It contains a new X.509 extension to indicate that it is a PC and to place restrictions on use of the PC.
Procedure
If an entity A wants to delegate its rights to entity B, the procedure is
(1) Entity B creates a private/public key pair
(2) Entity B creates a request for a PC and sends it to entity A
(3) Entity A creates a PC, singed by the private key of A's certificate (EEC or another PC).
In this process, restrictions can be placed on the newly created PC. For example, validity period...
(4) Entity A sends the certificate back to entity B.
Note: the private key of entity A is stored on the local file system.
Significant advantages
Private keys are never transferred over the internet.

How MyProxy works?
Proxy storing
This is similar to the procedure described above in section <Proxy Certificate> / <Procedure>. Entity A is the end user and entity B is MyProxy server. In step (4), end user sends both the newly created Proxy Certificate and the entire certificate chain to the server. Also access control can be imposed by means described below.
Proxy retrieval
Similar to proxy storing, except in reverse. Access control policies must be checked to see whether the client is allowed to retrieve the certificate.
Access Control
Server administrator can control who can store/retrieve credentials by:
    (1) Requiring client-side TLS authentication
    (2) Configuring a regular expression that must match the client certificate's subject for storage requests
    (3) Configuring a regular expression that must match the client certificate's subject for retrieval requests
Credential owner can control access to their certificates by :
    (1) setting a password when storing the certificates
     The password is not stored in the server. It is used to encrypt server's private key which would be used to sign the proxy certificates.
    (2) settting regular expression on the client certificate's subject that would retrieve the certificate.
Renewal
    to be filled soon

To be investigated in the future
MyProxy and OGSI (Open Grid Services Infrastructure):
MyProxy is encapsulated as web services.
MyProxy and WS-Resource Framework

Uniform Resource Identifier(URI) and Uniform Resource Locators (URL) - (RFC 1738 and RFC 3986)

2009-01-21T23:56:00.001-05:00

Character Escape
URI consists of a set of characters.
uric = reserved | unreserved | escaped

Reserved characters
Principle: a character is reserved if the semantics of the URI changes if the character is replaced with its escaped escaped encoding.

    gen-delims  = ":" / "/" / "?" / "#" / "[" / "]" / "@"
    sub-delims  = "!" / "$" / "&" / "'" / "(" / ")" / "*" / "+" / "," / ";" / "="

Their usage within URI component is limited to their reserved purpose. If the data would conflict with the reserved purpose, it must be escaped.
Characters in the "reserved" set are NOT reserved in all contexts. The reserved characters in a URI component is defined by that specific component.

Unreserved characters
a-z A-z 0-9 "-" "_" "." "~"
These characters can be escaped WITHOUT changing the semantics of the URI.
But this should NOT be done UNLESS the escape is necessary.

Disallowed characters
Some characters are disallowed for various reasons. To use those characters, they MUST be escaped.
Disallowed US-ASCII Characters:
control:      <US-ASCII coded characters 0x00-0x1F and 0x7F>
space:        <US-ASCII coded character 0x20>
delimiters: < > # % "
unwise:       { } | \ ^ [ ] `

When to escape?
When a character does not have a representation using an unreserved character, it must be escaped. It includes:
(1) data that does not correspond to printable characters (ANSII coding)
(2) disallowed characters
Note: here, whether a character is unreserved is context-specific.

Escape sequences:
A "%" followed by hex representation of the character.
escaped = "%" hex hex
E.g. %20 %35
Uppercase hexadecimal digits should be used in percent-encoding!

Syntax
Generic URI syntax:
<scheme>:<scheme-specific-part>
Interpretation of scheme-specific-part depends on the scheme.
<scheme>://<authority><path>?<query>

scheme
alpha *( alpha | digit | "+" | "-" | "." )

authority
URI component authority can be internet-based server or a scheme-specific registry.
authority (server based) = username@host:port
userinfo = *( unreserved | escaped |";" | ":" | "&" | "=" | "+" | "$" | "," )
About domain label:

"The rightmost domain label of a fully qualified domain name will never start with a digit, thus syntactically distinguishing domain names from IPv4 addresses, and may be followed by a single "." if it is necessary to distinguish between the complete domain name and any local domain."

Query
query = *uric
Within a query component, the characters ";", "/", "?", ":", "@", "&", "=", "+", ",", and "$" are reserved.

Fragment
Fragment is not part of a URI, but is often used in conjunction with a URI.
URI-reference = [ absoluteURI | relativeURI ] [ "#" fragment ]

from RFC 2396:
"The semantics of a fragment identifier is a property of the data resulting from a retrieval action, regardless of the type of URI used in the reference.
A fragment identifier is only meaningful when a URI reference is intended for retrieval and the result of that retrieval is a document for which the identified fragment is consistently defined."

Relative URI reference
to be continued in the future.

Specific schemes

scheme	syntax	Explanation	Note
file	file://<host>/<path>	Access a file on a specific host. <host> can be "localhost" or empty to indicate local host. E.g. file:///usr/home	Unlike http and ftp, It does not specify an internet protocol to access the files.
ftp	ftp://<host>:<port>/ <cwd1>/<cwd2>/.../<cwdN>/ <name>;type=<typecode>	<cwd1> through <cwdN> are strings and <typecode> can be "a", "i" or "d". If <typecode> is "d", <name> is used as the argument of NLIST command.	Within the <name> or a CWD component, / and ; must be escaped. E.g. ftp://test.com/%2Froot/a.txt
mailto	mailto:<mail-address>	RFC 2822 specifies the format of internet messages.	Usually, "%" must be escaped.
http	http://<host>:<port>/ <path>?<query>

Resources
URI working group: http://labs.apache.org/webarch/uri/

Some resources of symbolic learning and connectionist

2009-01-14T01:21:00.001-05:00

This post http://www.cs.indiana.edu/classes/c661/sym-subsym.html gives some breif introduction and comparison of these two methods. It is pretty good.

Also I found this paper - Comparing connectionist and symbolic learning method which compares these two methods based on some experimental results. It is an old paper but I think maybe it is worth reading.

After searching on web, I found this book: Perspectives of Neural-Symbolic Integration(http://www.neural-symbolic.org/). Its name seems to imply that this book will introduction the combination of these two methods. I have not read it yet.

MySQL error 1045(28000): Access denied for user ...

2009-01-07T17:27:00.000-05:00

Problem
I have a database called hibernate in my MySQL server and I grant privileges to a user using this command
    > grant all on hibernate.* to 'testuser'@'%' identified by 'testuser';
Wildchar % means no matter where the user logs in mysql server from, the access should be allowed for further checking/verification.
When I tried to log in as the newly created user from local host, I got this error:
    ERROR 1045 (28000): Access denied for user 'testuser'@'localhost' (using password: YES)
Also I tried some suggested solutions:
   > FLUSH PRIVILEGES:
   > SET PASSWORD FOR 'testuser'@'%'='testuser';
None of them works.

My solution:
Using following two commands:
> grant all on hibernate.* to 'testuser'@'%' identified by 'testuser'; > grant all on hibernate.* to 'testuser'@'localhost' identified by 'testuser';

Theoretically, wildchar % should have included all hosts. But it may not cover localhost. Who knows.

Install and Configure MySQL as a non-root user on Linux

2009-01-07T14:59:00.000-05:00

Recently, I want to install MySQL on a server in my lab. It should not be difficult considering the wide usage of MySQL. I must install MySQL as a non-root user. This results in difficulties of installation.
My target OS is Red Hat Enterprise Linux Server release 5.2.
My username is gerald and my home directory is /home/gerald.

MySQL website provides a binary package(rpm) for RedHat. I downloaded package MySQL-server-community-5.0.67-0.rhel5.i386.rpm.
After executing command
> rpm -q -p MySQL-server-community-5.0.67-0.rhel5.i386.rpm -i
I got following output:

Name        : MySQL-server-community       Relocations: (not relocatable)
Version     : 5.0.67                       Vendor: MySQL AB
Release     : 0.rhel5                      Build Date: Mon 04 Aug 2008 03:31:42 PM EDT
Install Date: (not installed)              Build Host: blade11.mysql.com
Group       : Applications/Databases       Source RPM: MySQL-community-5.0.67-0.rhel5.src.rpm
Size        : 42294654                     License: GPL
Signature   : DSA/SHA1, Wed 06 Aug 2008 05:36:22 AM EDT, Key ID 8c718d3b5072e1f5
Packager    : MySQL Product Engineering Team <build@mysql.com>
URL         : http://www.mysql.com/
Summary     : @COMMENT@ for Red Hat Enterprise Linux 5

See the red text above?
So It did not work because the rpm package should be installed as root.

As a result, I must install MySQL from source. Steps:

Installation
(1) Download and untar MySQL-5.1.30. Then Change location into the top-level directory of the unpacked distribution.
(2) Configure
CFLAGS="-O3" CXX=gcc CXXFLAGS="-O3 -felide-constructors -fno-exceptions -fno-rtti" ; \ ./configure --prefix=/home/gerald/share/mysql \ --enable-assembler \ --with-mysqld-ldflags=-all-static \ --with-client-ldflags=-all-static \ --with-mysqld-user=gerald \ --with-unix-socket-path=/home/gerald/share/mysql/tmp/mysql.sock \ --localstatedir=/home/gerald/share/mysql/data
--localstatedir: overrides the default location for database directories (normally /usr/local/var for installation from source)
--with-unix-socket-path: where unix socket file would be stored. (must be accessible)
--with-mysqld-ldflags=-all-static: compile statically linked daemon programs
--with-client-ldflags=-all-static: compile statically linked mysql client programs
--enable-assembler: Use assembler versions of some string functions if available
--prefix: Install architecture-independent files in the specified directory.

Values of some of options above should be replaced with your own paths.
Also this option --enable-thread-safe-client may should be applied in your case if your client programs use multi-thread.
Description of all available configuration options is here http://dev.mysql.com/doc/refman/5.1/en/configure-options.html.
(3) make
(4) make install

Post-installation
http://dev.mysql.com/doc/refman/5.1/en/unix-post-installation.html
(5) Create and edit configuration file (also called option file) my.cnf.
http://dev.mysql.com/doc/refman/5.1/en/option-files.html
My my.cnf file looks like this:

[mysqld]
user=gerald
basedir=/home/gerald/share/mysql
datadir=/home/gerald/share/mysql/data
port=3307
socket=/home/gerald/share/mysql/tmp/mysql.sock

[mysqld_safe]
log-error=/home/gerald/share/mysql/log/mysqld.log
pid-file=/home/gerald/share/mysql/mysqld.pid

[client]
port=3307
user=gerald
socket=/home/gerald/share/mysql/tmp/mysql.sock

[mysqladmin]
user=root
port=3307
socket=/home/gerald/share/mysql/tmp/mysql.sock

[mysql]
port=3307
socket=/home/gerald/share/mysql/tmp/mysql.sock

[mysql_install_db]
user=gerald
port=3307
basedir=/home/gerald/share/mysql
datadir=/home/gerald/share/mysql/data
socket=/home/gerald/share/mysql/tmp/mysql.sock

On Unix, MySQL programs read startup option file from the following files:

Filename	Purpose
`/etc/my.cnf`	Global options
`/etc/mysql/my.cnf`	Global options (as of MySQL 5.1.15)
`SYSCONFDIR/my.cnf`	Global options
`$MYSQL_HOME/my.cnf`	Server-specific options
`defaults-extra-file`	The file specified with `--defaults-extra-file=path`, if any
`~/.my.cnf`	User-specific options

I use the fourth option. So I set environment variable MYSQL_HOME using this command:
    > export MYSQL_HOME=/home/gerald/share/mysql
This environment variable contains the directory where mysql is installed.
If you don't want to manually set the environment variable every time you start up the mysql daemon, you can add it to file ~/.bash_profile(if you are using bash).
Then copy the option file to the directory specified by MYSQL_HOME.
Note: options specified in my.cnf can also be supplied on the command line when you execute a mysql program.
(6) Change location to the directory where mysql is installed (in my case, it is /home/gerald/share/mysql).
(7) Initialize system tables using command
    >bin/mysql_install_db
Options of this command are specified in our option file $MYSQL_HOME/my.cnf. Of course, you can specify the options in command line. But I think using the option file is more convenient.
I got a warning: [Warning] Ignoring user change to 'gerald' because the user was set to 'mysql' earlier on the command line. I don't know how to fix it now. But mysql seems to work well despite the warning.
Note: If you install MySQL using a binary package, usually this step is automatically done when you install the package.
(8) mkdir log
create log directory under which log files are saved. This must match the configuration of option log-error in my.cnf.
In my case, log-error is set to /home/gerald/share/mysql/log/mysqld.log
(9) Start up mysql using command
    > bin/mysqld_safe &
(10) Verify that mysql daemon runs well. Use any of following commands:

> bin/mysqladmin version
> bin/mysqladmin ping
> bin/mysqladmin status
> bin/mysqladmin variables

Initial Account Security
http://dev.mysql.com/doc/refman/5.1/en/default-privileges.html
By default, password of root user is empty. In other words, anyone who can access the host can also access information of your mysql database. You can set password of root user using
> mysqladmin -u root password newpwd
or
> mysqladmin -u root -h host_name password newpwd

Account management statements
http://dev.mysql.com/doc/refman/5.1/en/account-management-sql.html

Privilege system
http://dev.mysql.com/doc/refman/5.1/en/privilege-system.html
Usually, you need to check table mysql.user:
    > select * from mysql.user;
How can you know the indentity used by mysql?
After running mysql successfully, use command
    > select CURRENT_USER();
Set password of a user:
    > SET PASSWORD FOR 'someuser'@'somehost' = PASSWORD('newpass');
Grant privileges:
Usually, commands are like this:

    > GRANT ALL ON mydb.* TO 'someuser'@'somehost';
    > GRANT ALL ON mydb.mytbl TO 'someuser'@'somehost';
    > GRANT SELECT (col1), INSERT (col1,col2) ON mydb.mytbl TO 'someuser'@'somehost';

Revoke privileges:
> revoke all on mydb.* from 'someuser'@'somehost';
Remove user:
> drop user 'someuser'@'somehost'

Resources
Related environment variables:
http://dev.mysql.com/doc/refman/5.1/en/environment-variables.html
List of programs/tools in the mysql package:
http://dev.mysql.com/doc/refman/5.1/en/programs.html

Ubuntu error "I've detected a panel already running"

2009-01-05T13:37:00.000-05:00

Prerequisite: I use GNOME, not KDE.
I want to install Nvidia driver to my old Ubuntu machine. However, it prompts that a X server is running and the installation can not proceed. Obviously, X server must be killed to install the driver. To press Ctrl+Alt + Backspace does NOT do the work because it RESTARTS the X server instead of kills it.
After searching online, I found that following command worked for me:

sudo /etc/init.d/gdm stop

After I installed the driver, I used command startx to start GNOME. At that time the error "I've detected a panel already running ..." came out. I found this command worked:

killall gnome-panel
killall gdm
startx

In other words, the processing gnome-panel is not terminated even if you press Ctrl+Alt+Backspace to kill the X server.

Version matching of JSP, JSTL and Servlet

2009-01-03T20:34:00.000-05:00

Working with JSP, JSTL, Servlet and Container is not an easy job if you download and configure every part by yourself. The most important problem is version matching. Also web.xml must be correctly configured to make use of the right version of Servlet/JSP.

Web app descriptor (web.xml)
To make use of correct version of JSP/JSTL/EL, you should read following posts which give detailed information you need to know to write web.xml:
http://faq.javaranch.com/java/ServletsWebXml

Some useful resources I found:
http://faq.javaranch.com/java/ElOrJstlNotWorkingAsExpected
http://forum.springframework.org/archive/index.php/t-19866.html
http://blog.csdn.net/eviliw/archive/2007/12/17/1944270.aspx
http://faq.javaranch.com/java/JstlTagLibDefinitions

On Sun's website, it is so hard to find a download link for JSTL 1.2. Usually you will be directed to the JSTL spec site.
You can download JSTL 1.2 from here: https://maven-repository.dev.java.net/repository/jstl/jars/.
Note for JSTL 1.2 there is just one jar file instead of two.
In JSTL 1.1/1.0, there are two jars : jstl.jar and standard.jar.
You can download JSTL 1.1 from here http://jakarta.apache.org/site/downloads/downloads_taglibs-standard.cgi and JSTL 1.0 from http://jakarta.apache.org/site/downloads/downloads_taglibs-standard.cgi.

Version match
JSTL 1.0 : Servlet 2.3 : JSP 1,2 (tomcat 4)
JSTL 1.1 : Servlet 2.4 : JSP 2.0 (tomcat 5)
JSTL 1.2 : Servlet 2.5 : JSP 2.1 (tomcat 6)
In old versions of JSP, EL is not enabled by default!! You can enable EL manually by using JSP page directory.
JSTL jars are not included in tomcat distributions so far. You need to download and deploy JSTL by yourself.

JSTL 1.0 specifies a set of custom tag libraries based on the JSP 1.2 API. There are four separate tag libraries, each containing custom actions targeting a specific functional area. This table lists each library with its recommended tag prefix and default URI:

Description	Prefix	Default URI
Core	`c`	`http://java.sun.com/jstl/core`
XML Processing	`x`	`http://java.sun.com/jstl/xml`
I18N & Formatting	`fmt`	`http://java.sun.com/jstl/fmt`
Database Access	`sql`	`http://java.sun.com/jstl/sql`

JSTL 1.1 specifies a set of custom tag libraries based on the JSP 2.0 API. There are five separate tag libraries, each containing custom actions targeting a specific functional area. This table lists each library with its recommended tag prefix and default URI:

Description	Prefix	Default URI
Core	`c`	`http://java.sun.com/jsp/jstl/core`
XML Processing	`x`	`http://java.sun.com/jsp/jstl/xml`
I18N & Formatting	`fmt`	`http://java.sun.com/jsp/jstl/fmt`
Database Access	`sql`	`http://java.sun.com/jsp/jstl/sql`

One missed tag in the table is "functions".
It seems that you also can use tag URI like this: http://java.sum.com/jstl/core_rt. And it works in my application. However, I have not investigated what's going on behind the scene. So this may or may not work generally. Instead, you should always use the new tag URI.

Prefixes and URL of JSTL 1.2 Tag Libraries are the same as that of JSTL 1.1.

Troubleshooting
(1) If you get following error "According to TLD or attribute directive in tag file, attribute test does not accept any expressions", it is possible that you don't specify tag prefix and URI correctly. You may be using JSTL 1.2 while you specify tag URI of JSTL 1.0 wrongly. Take Core as an example, tag URI in JSTL 1.1/1.2 is http://java.sun.com/jsp/jstl/core while tag URI in JSTL 1.0 is http://java.sun.com/jstl/core.
(2) If your EL/JSP is displayed directly without evaluation. it is highly possible that your web.xml file specifies wrong JSP version. See this post http://faq.javaranch.com/java/ServletsWebXml.

HTML table cell text wrap

2008-12-29T22:02:00.000-05:00

By default, if you set width of table cell, words would be wrapped automatically. A single word would not be split into pieces. Usually, an English word is not so long. But, some "words" may be very long. For example, URLs without word delimiter would be considered as a single word.
CSS property word-break can be used to force split a long word into more than one line.
usage: work-break:break-all
Unfortunately, that does not work on FireFox :-( I searched the internet and found that FireFox does not support that property. Besides, there is no equivalent in FireFox.

How to use Maven2

2008-12-26T03:30:00.001-05:00

Download and install maven: http://maven.apache.org/download.html.

Running Maven
http://maven.apache.org/guides/getting-started/maven-in-five-minutes.html
http://maven.apache.org/guides/getting-started/index.html
Generally the local repository is provided in USER_HOME/.m2/repository.

Configuration
http://maven.apache.org/guides/mini/guide-configuring-maven.html
Three levels:

Project - most static configuration occurs in pom.xml.
There should be a project configuration for every project. The configuration in pom.xml overrides the same configuration in other two levels. A user does not need to specify every aspect of the configuration. Those unspecified aspects are inherited from Maven's default Super POM.
Introduction to POM: http://maven.apache.org/guides/introduction/introduction-to-the-pom.html.
POM reference: http://maven.apache.org/pom.html
Installation - this is configuration added once for a Maven installation
This configuration file is generally located in M2_HOME/conf/settings.xml.
NOTE: This location can be overridden with the system property:
-Dorg.apache.maven.global-settings=/path/to/global/settings.xml
Guide: http://maven.apache.org/guides/mini/guide-configuring-maven.html
Reference: http://maven.apache.org/ref/2.0.8/maven-settings/settings.html
Detailed reference: http://maven.apache.org/settings.html
User - this is configuration specific to a particular user
This configuration file is generally located in USER_HOME/.m2/settings.xml.
NOTE: This location can be overridden with the system property:
-Dorg.apache.maven.user-settings=/path/to/user/settings.xml

Build your own private/internal repository:
This article introduces how to create a repository using Artifactory: http://www.theserverside.com/tt/articles/article.tss?l=SettingUpMavenRepository. In addition, the author also compares some mainstream maven remote repository managers including Standard maven proxy, Dead simple Maven Proxy, Proximity and Artifactory.
In my case, I also use Artifactory and deploy it to tomcat. It has a nice web-based interface. Artifactory uses database(derby I think) to store various repository data so a user can not know the repository content by directly looking at the directory.

Deploy your artifacts to remote repository by using maven-deploy plugin:
http://maven.apache.org/plugins/maven-deploy-plugin/usage.html
(1) If the artifacts are built by using Maven, you should use deploy:deploy Mojo.
In your pom.xml, element <distributionManagement/> should be inserted to tell Maven how to deploy current package. If your repository is secured, you may also want to configure your settings.xml file to define corresponding <server/> entries which provides authentication information.
Command: mvn deploy.
(2) If the artifacts are NOT built by using Maven, you should use deploy:deploy-file Mojo.
Sample command:
mvn deploy:deploy-file -Dpackaging=jar -Durl=file:/grids/c2/www/htdocs/maven2
-Dfile=./junit.jar -DgroupId=gridshib -DartifactId=junit -Dversion=GTLAB

FAQ:
(1) What does maven standard directory layout look like?
http://maven.apache.org/guides/introduction/introduction-to-the-standard-directory-layout.html
(1) How to specify parent artifact in pom.xml?
Read http://maven.apache.org/guides/introduction/introduction-to-the-pom.html.
(2) If a dependent package can not be download from central Maven repository, three methods can be used to deal with it:

"
Install the dependency locally using the install plugin. The method is the simplest recommended method. For example:
mvn install:install-file -Dfile=non-maven-proj.jar -DgroupId=some.group -DartifactId=non-maven-proj -Dversion=1
Notice that an address is still required, only this time you use the command line and the install plugin will create a POM for you with the given address.
Create your own repository and deploy it there. This is a favorite method for companies with an intranet and need to be able to keep everyone in synch. There is a Maven goal called deploy:deploy-file which is similar to the install:install-file goal (read the plugin's goal page for more information).

Set the dependency scope to system and define a systemPath. This is not recommended, however, but leads us to explaining the following elements:
"

(2) How to add new repository?
Put following code snippet into pom.xml or settings.xml.

<repository>
  <id>your-new-repository-id</id>
  <name>New Maven Repository </name>
  <layout>default</layout>
  <url>Address of the new repository</url>
  <snapshots>
    <enabled>enable-it?</enabled>
  </snapshots>
  <releases>
    <enabled>enable-it?</enabled>
  </releases>
</repository>

(3) How to disable default central maven repository?
Put following snippet into your pom.xml.

<repository>
  <id>central</id>
  <name>Maven Repository Switchboard</name>
  <layout>default</layout>
  <url>http://repo1.maven.org/maven2</url>
  <snapshots>
    <enabled>false</enabled>
  </snapshots>
  <releases>
    <enabled>false</enabled>
  </releases>
</repository>

(4) How can I package source code without run test?
Feed parameter -Dmaven.test.skip=true into the command line.
Note this property is defined by maven plugin surefire.
(5) Why does "mvn clean" delete my source code?
In your pom.xml, if content of element <directory> nested in element <build> is "./", "mvn clean" will delete all content in current directory including the src directory.
There are two more elements which can be used to specify locations of compiled classes.
outputDirectory: The directory where compiled application classes are placed.
testOutputDirectory: The directory where compiled test classes are placed.
(6) How to add resources into built package?
http://maven.apache.org/guides/getting-started/index.html#How_do_I_add_resources_to_my_JAR.
http://maven.apache.org/guides/getting-started/index.html#How_do_I_filter_resource_files
(7) Sometime, you want to use some libraries at compilation time and you don't want Maven to add them into your package(jar or war). How to do that?
Use dependency scope "provided" instead of default "compile". Read this post for details:
http://maven.apache.org/general.html#scope-provided. And this post elaborates Maven's dependency mechanism: http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html#Dependency_Scope
(8) How to build a war instead of jar?
Use Maven WAR Plugin: http://maven.apache.org/plugins/maven-war-plugin/usage.html.
First you should set entry packaging in pom.xml to war.
<packaging>war</packaging>
Then you can use one of the following commands to build your war:
mvn package mvn compile war:war mvn compile war:exploded mvn compile war:inplace
Also you can filter resources of your web app. See this post http://maven.apache.org/plugins/maven-war-plugin/examples/adding-filtering-webresources.html.
(9) How to make war and jar at the same time?
By default your source code is compiled into class files and placed into directory WEB-INF/classes. Sometimes you may want to build a jar and then put it into directory WEB-INF/lib.
http://communitygrids.blogspot.com/2007/11/maven-making-war-and-jar-at-same-time.html
http://maven.apache.org/plugins/maven-war-plugin/war-mojo.html
http://maven.apache.org/plugins/maven-war-plugin/faq.html#attached
(10) help plugin can be used get information about a project, plugin or the system.
http://maven.apache.org/plugins/maven-help-plugin/
mvn help:describe -DgroupId=org.apache.maven.plugins -DartifactId=maven-compiler-plugin -Dfull=true
mvn help:describe -DgroupId=org.apache.maven.plugins -DartifactId=maven-compiler-plugin
mvn help:system
mvn help:all-profiles
mvn help:active-profiles

Properties reference:
http://docs.codehaus.org/display/MAVENUSER/MavenPropertiesGuide

During my using of Maven2, I encountered several bugs:
(1) targetPath support on the webResources war plugin parameter:
http://jira.codehaus.org/browse/MWAR-54
(2) maven-war-plugin webResources -- relative path:
http://www.mail-archive.com/users@maven.apache.org/msg77274.html
http://jira.codehaus.org/browse/MNG-2382
http://jira.codehaus.org/browse/MWAR-79
http://jira.codehaus.org/browse/MWAR-77

Install and Configure Ubuntu 8.10

2008-12-26T02:43:00.000-05:00

Recently, I got a used desktop which would be used as linux server. I installed Ubuntu 8.10. The installation was easy. I downloaded the Ubuntu .iso file and burned it to a CD. Then just installed Ubuntu using the CD.

Vim upgrade
By default, Ubuntu just installed vim-tiny whose functionalities may not satisfy end user's requirements. For example, opening a directory using vim-tiny would fail rather than list all files in the directory in vim. So I installed vim-full which includes all vim features using this command:
sudo apt-get install vim-full
Then usually vim is configured to meet user's specicial requirements. In my case, the configuration file $HOME/.vimrc looks like:

set tabstop=4
set shiftwidth=4
set wrapmargin=8
set smartindent	"smart indentation
set expandtab  	"expand tabs to spaces
set ruler 		"display ruler on bottom right corner
set nu			"display line number
set incsearch	"turn on incremental search
set hlsearch		"highlight search result

:colorscheme ron 
:filetype indent on		"enable special indentation rules according to file type.

"highlight the 81-th character in each line
au BufEnter * ":/\%81c"
"in default configuration, textwidth option is set
"so I want to override the default value.
au BufRead * set tw=0

"when you open a file, cursor is moved to previous position when you edited the file last time.
au BufReadPost * if line("'\"") > 0|if line("'\"") <= line("$")|exe("norm '\"")|else|exe "norm $"|endif|endif

:syntax on		"turn on syntax detection and highlight
"stop cursor blinking. Only available when compiled with GUI enabled,
"and for MS-DOS and Win32 consol
set guicursor=a:blinkon0

All available options of vim can be read here: http://www.vim.org/htmldoc/index.html

Browser
Firefox is installed by default which is the default web browser. Several addons were installed manually.

Tab Mix Plus	Tab browsing with added boost.
All-in-One gestures	Support mouse gestures.

Firebug	Web development Evolved
RestTest	Allow users to send HTTP requests with customized heads and data.
Poster	A developer tool for interacting with web services and other web resources that lets you make HTTP requests, set the entity body, and content type. Similar to RestTest. But Poster works on FF3 while RestTest does not.
HttpFox	An HTTP analyzer
Web Developer	Adds a menu and a toolbar with various web developer tools

FireShot	Take a screenshot and edit it
Google Notebook	Firefox addon for google notebook application
QuickRestart	Add a "Restart Firefox" item to the "File" menu.
MenuEditor	Customize application menus
FlashGot	Enables single and massive downloads using external download managers. This addon itself is not a download program.

Terminal configuration
I like "green on black" color theme.

Keyboard shortcuts
Click System -> Keyboard Shortcuts to display the shortcut setting dialog. Shortcuts I use very often include:
Super + s     #start a terminal
Super + m    #toggle maximization state
Super + n     #minimize window
Ctrl+Alt+L    #lock screen
Alt + F2       #Show run application dialog
Alt + Tab     #switch between different windows
Ctrl + Alt + Left/Right/Up/Down    #switch to different workspaces

Following serveral shortcuts are available only after you install compiz and enable corresponding components.
Super + E #Expo key
Super + Tab #another window switcher
Super + leftclick #move the window

Wireless network configuration
Wired network connection is established successfully. However, after I moved my desktop, it is not close to the router and I don't want to connect them using a cable.
I have a D-Link System AirPlus G DWL-G122 Wireless USB Adapter. After I plugged it in, I used command lsusb to see whether the device was detected. It was detected but it did not work. Obviously, a driver is needed to make it work. I read this post: https://help.ubuntu.com/community/WifiDocs/Driver/Ndiswrapper. It states

D-Link DWL-G122 USB Wireless device: As of December 2008, Ubuntu 8.10 provides full 'out of the box' support for this device, using the rt73usb driver. In this case, there is no need to use ndiswrapper at all and there is no need to make any changes to the default /etc/modprobe.d/blacklist file."

But on my machine, the wireless adapter still did not work even after the module rt73usb was loaded. I guess the reason is that the device mentioned in the manual and my device are different although they seem the same.
Then I followed the instructions on post https://help.ubuntu.com/community/WifiDocs/Driver/Ndiswrapper and it worked well.
Network manager is a convenient tool to configure your network.
http://projects.gnome.org/NetworkManager/
https://help.ubuntu.com/community/NetworkManager

Network Management
Some packages and their useful commands to manage network devices

Supported commands by package "net-tools":
ifconfig: configure the kernel-resident network interfaces.

Supported commands by package "wireless-tools":
iwconfig: it is dedicated to the wireless interfaces. It can be used to set the parameters of NIC which are specific to wireless connection.
iwlist: display additional information from a wireless NIC. "iwlist scan" returns a list of available wireless networks.

Supported commands by package "network-manager":
NetworkManager: network management daemon.
nm-tool: utility to report state of network manager in text mode.
nm-system-settings:

Supported commands by package "network-manager-gnome":
nm-applet: a graphical networkmanager applet. It displays an icon in notification area(usually at top right corner) for managing network devices and connections. Usually, it is started up automatically when the system boots up.
nm-connection-editor: display a graphcial connection configuration tool.

Supported commands by package "gnome-nettool":
gnome-nettool : GNOME network tools. This tool can be used to display detailed network information.

gnome-network-preference : Set network proxy preferences

Compiz window manager
A good reference: https://help.ubuntu.com/community/CompositeManager/CompizFusion
Compiz provides some very cool features I like.
Use command ccsm to display CompizConfig Settings manager.
More shortcuts:
Super + E             #Expo key. supported by "Expo" component.
Super + Tab          #another window switcher. supported by "Shift Switcher" component and "Ring Switcher" component.
Alt + left-button    #move the window. supported by "Move Window" component.
Alt + mid-button   #resize the window. supported by "Resize Window" component.

Input Method
To input Chinese, I added Chinese language support. Executing command "gnome-language-selector" would display "language support" dialog and you can select any language in the list you want to use. Releated packages are installed automatically.
SCIM (smart comman input method) is installed by default. Try command "scim-setup" to configure scim. Use command "im-switch -z en_US -s scim" to switch input method to scim. Then restart X. scim would be started automatically.
Also you can manually start scim using command "scim -d".
To make the lookup table follow the cursor, in scim setup dialog uncheck
FrontEnd -> Global Setup -> Embed Preedit String into client window
and
Panel -> GTK -> Embedded lookup table

Misc. Useful packages I installed:
(*) sudo apt-get install gnome-device-manager
As its name implies, it is a device manager with GUI. Then it can be accessed by clicking Applications->System Tools -> Device Manager.
(*) Totem is installed by default which is a video player.
(*) sudo apt-get install vlc #vlc player
Also, I tried to find a video player in Ubuntu repositories which can play .rm and .rmvb files. None of vlc, Kmplayer and mplayer can do that job. It seems that the main problem is license which means including real codec is illegal. Finally I downloaded linux version of realplayer from official real web site and installed it successfully.
(*) sudo apt-get install sun-java6-jdk #jdk 6
Then use update-alternatives --config java to set default java executable.
(*) Package "Transmission" is installed by default which is a BitTorrent client program.
(*) sudo apt-get install deluge-torrent (a bittorrent client)
(*) sudo apt-get install d4x (a download manager)
(*) sudo apt-get install gwget (another download manager)

More tips
How to take screenshots: http://tips.webdesign10.com/how-to-take-a-screenshot-on-ubuntu-linux
How to view btchina in Linux?
http://mozilla.sociz.com/viewthread.php?tid=2367
First install greasemonkey addon and then install the script at http://userscripts.org/scripts/show/33286 (click the button "Install" at top right corner). After successful installation, you can set the options of the script by clicking Tools->GreaseMonkey->User Script Commands -> option-here.
If you use FireFox on Windows, addon IETab can be used.
Flashget on Linux?
Candidates: gwget and WebDownloader for X (d4x) (can be installed from Ubuntu repositories). wxDownload Fast and trueDownloader(maybe must be installed from source).
Pick the ones you like from this list http://en.wikipedia.org/wiki/List_of_download_managers.
How to configure multiple versions of a program?
use utility update-alternatives.

Latex resources

2008-12-20T21:10:00.000-05:00

[How to insert figures to Latex document]
ftp://ftp.tex.ac.uk/tex-archive/info/epslatex.pdf
http://www.miwie.org/tex-refs/html/index.html
FAQ: http://www.physics.ohio-state.edu/~faqomatic/fom-serve/cache/103.html

[Screen presentation]
http://www.math.uakron.edu/~dpstory/pdf_present.html
http://www.miwie.org/presentations/html/index.html

References
http://www-h.eng.cam.ac.uk/help/tpl/textprocessing/teTeX/latex/latex2e-html/index.html
http://www-h.eng.cam.ac.uk/help/tpl/textprocessing/