Utilize shell scripts,linux commands, open source tools,java, to Maximize the Power of Linux.
Focused on working with linux and shell, search engine technology including Chinese segmenter
Any questions please contact me at gmail: david.ullua

1/29/2007

Mysql starting error: ... STOPPING server from pid file /var/lib/mysql/roboo-lvzg.pid , mysqld ended

When I start mysqld server (in slackware linux 11), got error as follows:

root@kseeker-lvzg:~# mysqld_safe --user='mysql' &

[1] 10624

root@kseeker-lvzg:~# Starting mysqld daemon with databases from /var/lib/mysql

STOPPING server from pid file /var/lib/mysql/kseeker-lvzg.pid

070129 18:55:45 mysqld ended

oh, you need to su mysql and run mysql_install_db to fix the problem.

root@kseeker-lvzg:~# su mysql

mysql@kseeker-lvzg:/root$ mysql_install_db

Installing all prepared tables

Fill help tables

To start mysqld at boot time you have to copy support-files/mysql.server

to the right place for your system

PLEASE REMEMBER TO SET A PASSWORD FOR THE MySQL root USER !

To do so, start the server, then issue the following commands:

/usr/bin/mysqladmin -u root password 'new-password'

/usr/bin/mysqladmin -u root -h kseeker-lvzg password 'new-password'

See the manual for more instructions.

You can start the MySQL daemon with:

cd /usr ; /usr/bin/mysqld_safe &

You can test the MySQL daemon with the benchmarks in the 'sql-bench' directory:

cd sql-bench ; perl run-all-tests

Please report any problems with the /usr/bin/mysqlbug script!

The latest information about MySQL is available on the web at

http://www.mysql.com

Support MySQL by buying support/licenses at http://shop.mysql.com

java knowledge tree

this java knowlege tree is composed and exported by FreeMind 0.8 (http://freemind.sourceforge.net) a little regret is the expand and collapse function is lost on the blog.





1/25/2007

利用~/.netrc文件自动ftp下载文件 (当天或昨天的tomcat日志)

home目录中添加文件.netrc,内容如下:

#download files begin with test under user david's home

machine 172.16.100.30 login david password 3287

macdef init

binary

prompt

!clear

mget test*

!clear

bye


#get the log of yesterday:

#echo "catalina.`date --date='1 days ago' +%Y-%m-%d `.log" | ftp www.kseeker.com

#download file specified by the second arg (the log filename)

machine www.kseeker.com login david password pass92342@#$@**(89

macdef init

binary

prompt

cd /home/tomcat/logs

mget $2

bye


注意:bye后要有一空行,.netrc文件的属性须是600,

可以使用这个命令下载172.16.100.30上面/home/david/目录test开头的文件:%ftp 172.16.100.30


可以使用这个命令来下载服务器上昨天的tomcat日志(文件名如: catalina.2007-01-24.log):

%echo "catalina.`date --date='1 days ago' +%Y-%m-%d `.log" | ftp www.kseeker.com

man netrc, 可以看到更多信息:

NETRC(5) BSD File Formats Manual NETRC(5)

NAME

netrc, .netrc -- user configuration for ftp

DESCRIPTION

This file contains configuration and autologin information for the File Transfer Protocol client ftp(1).

The .netrc file contains login and initialization information used by the auto-login process. It resides in the user's

home directory. The following tokens are recognized; they may be separated by spaces, tabs, or new-lines:

......

1/22/2007

KMail提示用户可能忘了附件的有趣功能

今天发邮件的时候,KMail居然提示我"可能有附件,要不要附加文件,还是原样发送?"
原来我的邮件的内容中含有attachment这个单词, 新建一封邮件, 里面有"附件"这个词,发送的时候KMail也提示是否要附加附件。
看起来KMail会自动检测邮件中是否有attachment,或者"附件"的单词存在,存在的时候,如果没有附加附件,在发送时会提醒是否附加附件。

这项功能实在蛮有趣的,开发者考虑的还真周到,象我这种粗心的人,曾饱受outlook发送邮件忘记附件之苦, 用KMail问题都解决了。

use crontab and mail command to backup your firefox bookmarks

add the following line to ~/cron.yourname , (yourname is your real login name)

30 12 * * * date '+%m/%d/%Y' | mail -s "bookmark"" `date '+%m/%d/%y'`" -a /home/yourname/.mozilla/firefox/68k8bqg9.default/bookmarks.html receiver1@abc.com receiver2@abc.com

maybe 68k8bqg9.default should be replaced by your actual bookmarks directory.

and run:

%crontab ~/cron.yourname

then crontab would run at 12:30 am to send firefox bookmark to your email receiver1 and receiver2.

date '+%m/%d/%Y' : to show date as "01/23/2007"

it use filter to current date as the mail's content. and use -a to specify the attachment.

it use -s spesify subject as: "bookmark"" `date '+%m/%d/%y'`" ,

here date is backquoted.

send mail with attachment in command line

if it is a normal text file, you can use redirection to send the file as
content:

mail -s "hello, david" someone@abc.com someone2@kseeker.com < attachfile

if it is a binary file, or you want to send text file as real attachment:

mail -s "hello, david" -a attachfile someone@abc.com someone2@kseeker.com

btw, mail is often the alias of mailx command in linux.

1/15/2007

add authentication for awstats in apache

awstats does not require an authentication by default.
but we can use apache's auth_module to add authentication for a specified directory( in this case is awstats).

assume awstats config file is /etc/awstats/awstats.kseeker.com.conf ,
awstats is installed at /usr/local/awstats/, awstats.pl is located at /usr/local/awstats/wwwroot/cgi-bin.
and httpd.conf is located at /etc/httpd/conf/.

step 1: config awstats to enable authentication access
%vi /etc/awstats/awstats.kseeker.com.conf, find the line contains AllowAccessFromWebToAuthenticatedUsersOnly, and the line contains
AllowAccessFromWebToFollowingAuthenticatedUsers, change them as follows:
AllowAccessFromWebToAuthenticatedUsersOnly=1
AllowAccessFromWebToFollowingAuthenticatedUsers="david"

step 2: enable apache authentication for awstats directory
%vi /etc/httpd/conf/httpd.conf, set AllowOverride to All for "/usr/local/awstats/wwwroot/cgi-bin" directory:

<Directory "/usr/local/awstats/wwwroot/cgi-bin">
AllowOverride All
Options None
Order allow,deny
Allow from all
</Directory>

step 3: add authentication file and .htacess file, add user , and restart apache

%cd /usr/local/awstats/wwwroot/cgi-bin
%htpasswd -c userfile david //create user file, it will prompt for user's password.
%cat > .htaccess
AuthUserFile /usr/local/awstats/wwwroot/cgi-bin/userfile
AuthName "Restricted Area For Customers"
AuthType Basic
require valid-user
Ctrl + D

% apachectl restart

vim与gvim 转换highlight的文本到html网页

今天发现用gvim居然可以把vim中着色显示的文本文件(c, php, sh, java, ...)转换成html,
且保留显示的格式和颜色,不必再为发布源代码到blog上显示格式混乱的问题了。

在gvim中点syntax => convert to html, 这样会产生一个后缀名加上html的文件,
然后保存即可(:w ).如果要对.html文件或者源文件作适当的编辑,
可以用Ctrl+ww 可以切换源文件与产生的html文件的窗口。

google 了一下"vim convert to html", 原来vim中也可以使用

:runtime! syntax/2html.vim 命令来保存按语法高亮显示的文本到html文件。

参考: http://vimdoc.sourceforge.net/vimfaq.html

24.14. How do I convert a source file (.c, .h, etc) with the Vim syntax
highlighting into a HTML file?

You can use the 2html.vim script to convert a source file into a HTML file

with the Vim syntax highlighting. Use the following command:
:runtime! syntax/2html.vim

For more information, read

:help convert-to-HTML

1/12/2007

Use eclipse with ant to deploy your java project -- from beginning of env setup to ant build

1.export project to ant build file.
2.run junit testing default build file
3.make your own build file,add unit test and junitreport to build.
4.create target of : makepackage,deployapp
5.run ant under linux terminal or win command line, or add it to scheduled tasks.
Q1: "Ant-junit, build failed: Could not create task or type of type: junit"
Q2: "ant: /bin/sh^M: bad interpreter: No such file or directory."

This tutorial shows how to use ant within eclipse to build, test, make package and deploy your app using eclipse step by step. Any questions, please feel free to contact me at david.ullua at gmail, or leave a comment. The tutorial is tested under eclipse 3.1.2, ant 1.6.5. in other higher version in the future should also works. The tutorial is oriented for readers without any knowledge about make,ant or similar build tools, while also can be as a reference for middle experienced developer.

1. export project to ant build file.
Open your java project in eclipse,select your project, right click mouse, click "export", choose "General" => "Ant BuildFiles", click "Finish", it will generate a build.xml file. The default build file includes project build target and junitreport target. we can use the default build.xml as a start.

Open build.xml, here is the first line( space after > is added for web display ) :
< basedir="." default="build" name="twoway-analyzer">< /project>
It mean the build file would take project path as basedir, and use build target as default task when running "ant" or "ant -f build.xml" in project dir without args specifying target(s).

Look at this line in build.xml:
< depends="build-subprojects,build-project" name="build">
the line means build task depends on 2 sub-tasks, and does not contain any other independent actions in the build project. that's to say, if we run "ant -f build.xml", or "ant build", it will call 2 sub-tasks: build-subprojects, build-project.

ok, now you can right click "build.xml" => "Run As" => "Ant Build" (choose the one which you can select task else it would build default tasks) , then choose targets to excute, click "run", it will run the build task.

If you met the error of "Could not create task or type of type: junit", see Q1 in the end of this article as reference.


2. run junit testing default build file
the default build file contains tests and junitreport,
you can merge seperate unit tests to one target as follows,
it combines all 4 unit tests into one target called unitTest:

<target name="unitTest">
<mkdir dir="${junit.output.dir}"/>
    <junit fork="yes" printsummary="withOutAndErr">
<formatter type="xml"/>
<test name="com.roboo.segmenter.unitTest.DictionaryTest" todir="${junit.output.dir}"/>
<test name="com.roboo.segmenter.unitTest.MMTokenizerTest" todir="${junit.output.dir}"/>
<test name="com.roboo.segmenter.unitTest.OriginTokenizerTest" todir="${junit.output.dir}"/>
<test name="com.roboo.segmenter.unitTest.RMMTokenizerTest" todir="${junit.output.dir}"/>
<classpath refid="twoway-analyzer.classpath"/>

target>
now in eclipse you can choose unitTest to run all tests.
and you can choose junitreport, this time the report would be generated. Attention if you have chosen junitreport before unitTest, then need to adjust targets order to make sure that unitTest is done before junitreport. else the report would be generated by old data.


3.make your own build file,add unit test and junitreport to build.

every time when you export Ant Buildfiles, the default build.xml will be override. so just make your own build file. save build.xml to custbuild.xml. Also we want ant run not only as eclipse plugin but also in terminal by a single command, ok, we can add these targets to the depends of build target. find this line in custbuild.xml(which we copied from build.xml):

<target depends="build-subprojects,build-project" name="build"/>

change it to :
<target depends="build-subprojects,build-project,unitTest,junitreport" name="build"/>

now can run "ant -f custbuild.xml" in command line to run project build,unitTest and junitreport in 1 step.

when running ant, if you met the error of "ant: /bin/sh^M: bad interpreter: No such file or directory.", see Q2 as reference in the end of this article.


4.create target of : makepackage,deployapp
now we want to make a jar package for the project, then add the following target:

-- jar task to make package -->
<target name="makepackage">
< echo message="Making jar package ....."/>
< delete file="${package.name}${package.version}.jar" />
< jar destfile="${package.name}${package.version}.jar"basedir="bin" excludes="*.*,**/allwords.dict" update="false">
-- exclude normal files under bin directory -->


don't forget to add the target name to the depends for "build" target.
here package.name and package.version is predefined in the properties of the build file(after xml file's root element):
<property name="package.name" value="roboosegmenter"/>
<property name="package.version" value="1.3.2.0"/>

after making the jar package, we need to deploy it to web apps.
here app_lib_dir,tomcat_web_lib_dir is predefined in the properties of the build file(after xml file's root element)
 -- deploy segmenter lib to app_lib_dir and tomcat_web_lib -->
<target name="deployapp">
<delete>
<fileset dir="${app_lib_dir}" includes="${package.name}*jar"/> < /delete>

<copy file="${package.name}${package.version}.jar" todir="${app_lib_dir}"/>

<delete>
<fileset dir="${tomcat_web_lib_dir}" includes="${package.name}*.jar"/> < /delete

<copy file="${package.name}${package.version}.jar" todir="${tomcat_web_lib_dir}"/>
don't forget to add the deployapp target to your build define, like this:
<target depends="build-subprojects,build-project,unitTest,junitreport,makepackage,deployapp"name="build"/>


5.run ant under linux terminal or windows command line, or add it to scheduled tasks.

to build project, run unit test, make jar package and deploy app:
%ant -f custbuild.xml

to build project do not run unit test or deploy:
%ant -f custbuild.xml unitTest

to build project and run just unitTest without deploy:
%ant -f custbuild.xml build-project unittest


Q1: Ant - junit, build failed: Could not create task or type of type: junit

When running JUnit tasks in ant and you get the following error:

BUILD FAILED
path\to\...\build.xml:259: Could not create task or type of type: junit.
Ant could not find the task or a class this task relies upon.

To fix this problem, you need to copy a JUnit.jar file to your eclipse/plugins/org.apache.ant_1.65/lib directory,
Go into Eclipse then click on Window->Preferences->ant->Runtime,
select 'Ant Home Entries (Default). Click on the button 'Add External JARs'. Locate the junit.jar file you copied, select it and hit 'OK'. Hit 'Apply', then 'OK' and try your ant task again. That should fix it.


Q2:when i run ant in unix/linux terminal it shows the following error: eclipse/plugins/org.apache.ant_1.6.5/bin/ant: /bin/sh^M: bad interpreter: No such file or directory.
A2:caused by file format under dos/windows is different to unix ,
%fromdos <> file2 or use dos2unix to change the file format.

1/11/2007

Eclipse and gaim font settings under KDE

When I opened gaim under KDE, the menus and dialogs are very ugly. And I have use eclipse day by day under KDE about 2 months. The dialog and menu, and editor fonts are fairly ugly for both English and Chinese. So I decide to change these fonts.

In eclipse Windows -> Preferences, -> General -> Appearance -> Color and Fonts, type "font" as filter , all fonts settings would be shown, dialog font, text editor font, java editor font, console fonts, etc. I change all fonts to WenQuanYi BitMap Song which support Chinese perfectly.

However in eclipse settings it's own menu font cannot be changed. after googled "elipse menu font" and "eclipse 菜单字体", found the solution:
download and install gtk-qt-engine here .

Then under the KDE Control-Center -> Appearence && Themes (外观和主题) , a submenu called "GTK Styles and Fonts" would be available, you can change the gtk Fonts here.

Now gaim also look fine.


The eclipse and gaim font problem's root cause is that :
1.Eclipse use SWT to show its UI; SWT is implemented by windows/gtk/motif
2.Running Eclipse on linux/gtk, need gtk libraries.
3.KDE is based on QT not GTK, so it need gtk-qt-engine to config GTK Style and Fonts. gtk-qt-engine is used to do it.

to implement SVN password changing function by shell and php

We have install SVN as our version control system. It doesn't provide a remote password changing function on the web. I googled on the internet and didn't found a good solution. So I wrote a script in bash and a php front end page to do it.
(if you need a jsp or perl front end page, you can write one yourself)

Here's the shell script to change password for svn:

#!/bin/sh
#changepass.sh , function: change password for svn
#usage: changepass.sh username oldpass newpass newpassretype
if [ ! "$#" = 4 ]
then
echo "Usage:`basename $0` username oldpass newpass newpassretype"
exit 1
fi

passfile="/home/svn/passwd"

username=
oldpass=
newpass1=
newpass2=

username=$1
oldpass=$2
newpass1=$3
newpass2=$4

if [ ! $newpass1 = $newpass2 ]
then
#echo "password does not match!";
echo "password does not match!"
exit 2
fi

export NEWPASS=$newpass1
export USERNAME=$username

i=`sed /^"$username"[" ""\t"]*=[" ""\t"]*$oldpass\$/p $passfile -n|wc -l`
if [ $i = 1 ]
then
awk '{ if(index($0,ENVIRON["USERNAME"])==1) {print ENVIRON["USERNAME"] " = " ENVIRON["NEWPASS"] }else {print $0} }' $passfile > /home/svn/newpass

rm /home/svn/newpass 2>&1 1> /dev/null
cp /home/svn/newpass $passfile

echo 'password changed!'
elif [ `sed /"$username"/p $passfile -n |wc -l` > 0 ]
then
echo ' username and password does not match'
else
echo 'could not find username'
fi


here's the php front end changepass.php , it is for php 5, if you use php 4, maybe you need use $HTTP_POST_VARS instead $_POST ( just guess, i haven't test it):



if(strlen($_POST["pwd1"]) >0 ){
if($_POST["pwd1"]!=$_POST["pwd2"]){
print("Passwords do not match, try again.

\n");


}else{

$LoginName = $_POST["loginname"];
$OldPassword = $_POST["oldpwd"];
$NewPass1 = $_POST["pwd1"];
$NewPass2 = $_POST["pwd2"];
$retval='';
#echo $LoginName;
#echo $OldPassword;
// user apache or nobody(who runs httpd daemon) needs the permission of passwd file and temp file
$command = "/mnt/apache/changepass.sh ".$LoginName." ".$OldPassword." ".$NewPass1." ".$NewPass2 ;

#$command = "/bin/cat /mnt/apache/changepass.sh "." 2>&1";
$command = $command." 2>&1";

#system("/bin/echo abc",$retval);
system($command, $retval);

echo "
" ;echo $retval;echo '
';


if($retval!=0){
print("
Error while setting password:
\n");

}

}
}
else{
;#echo "parameters not set!";
}
?>





Login Name:
Old Password:
New Password:
Repeat Password:


the php page and shell scripts should works ok, but the php page shows nothing after click "submit" in fedora, however in slackware it is ok.

After debug step by step, found that shell script error message would not be ouput on the web page, so i add this line: $command = $command." 2>&1";

And finally found that SELinux would block some function on the web server. I disabled SELinux in the firewall settings, and it works.



difference on class between php version 4.4.4 and 5.2.0

Different PHP versions behalves differently. The only two distribution on php.net is 4.4.4 and 5.2.0. The experience of the article describes class difference between 4.4.4 and 5.2.0 is that:
1). class members variable is decorated by "private" in 5.2.0, as "var" in 4.4.4
2). class functions is decorated by "public function" in 5.2.0, as "function" in 4.4.4


Several days ago, i wrote a php proxy for wap.roboo.com to transfer jsp response to a php server. And grabed a HTTPRequest class from google, the class code is as follows:

class HTTPRequest
{
var $_fp; // HTTP socket
var $_url; // full URL
var $_host; // HTTP host
var $_protocol; // protocol (HTTP/HTTPS)
var $_uri; // request URI
var $_port; // port

// scan url
function _scan_url()
{
....
}

// constructor
function HTTPRequest($url)
{
$this->_url = $url;
$this->_scan_url();
}

// download URL to string
function DownloadToString()
{
....
}
}

I use Slackware 11 to do most works, slack use php 4.4.4 by default. the HTTPRequest works ok on my machine. After deploy it on another fedora machine which has php 5.0 by default, it fails. Then I updated it to php 5.2.2, with the help of one colleague, I changed the class as follows to support 5.2.2:

class HTTPRequest
{
private $_fp; // HTTP socket
private $_url; // full URL
private $_host; // HTTP host
private $_protocol; // protocol (HTTP/HTTPS)
private $_uri; // request URI
private $_port; // port

// scan url
public function _scan_url()
{
....
}

// constructor
public function HTTPRequest($url)
{
$this->_url = $url;
$this->_scan_url();
}

// download URL to string
public function DownloadToString()
{
....
}
}

By the way, i use vi(vim actually) to edit source, just use :0,$s/var/private/g , :0,$s/function/public function/g to do the substitution.

After disabled SELinux on fedora , it works ok.

About Me

I am a senior developer and a team leader with 3 years development experience in Suzhou, China, focus on mobile web search, linux, Java and machine learning in NLP (natural language processing). My goal is to improve people's life with computer technology.