OpenBSD port of Sphinx

Posted by face on February 28, 2008

Sphinx Free open-source SQL full-text search engine

Update July 16th, 2008: Sphinx sphinx-0.9.8 has been released and this port has been updated.

Here is a OpenBSD port of Sphinx, Free open-source SQL full-text search engine. Sphinx is a wonderfully fast and memory efficient deep text search engine. I have found integrates nicely with Ruby and Ruby-On-Rails.

I hope at some point to get this port committed to the OpenBSD CVS repository. In the meantime this will be the ports home. Till it hits the OpenBSDs repository, it will live in my SVN repository here.

You can download a snapshot of the source code for the port here: sphinx.tgz. So far this port has been tested with MySQL and Postgresql on OpenBSD 4.2 and OpenBSD 4.3-betaOpenBSD-4.4-beta on the i386 platform with the Ultrasphinx Ruby-On-Rails plugin.

Instructions to build the port

If your ports tree is not already prepped, please begin by prepping your ports tree.

The port will build much faster if you install the prerequisites via binaries:

sudo su
export PKG_PATH=""
# or, setenv PKG_PATH ""
pkg_add libiconv mysql-server mysql-client

Now we are ready to build the sphinx port:

cd /usr/ports/textproc
lynx --source "" > sphinx.tgz
tar xvfz sphinx.tgz 
cd sphinx
make install  # or make package
There is also a Postgresql flavor. Follow the instructions above, installing Postgresql instead of MySQL and then instead of make install do:

env FLAVOR=pgsql make install

Thanks, and feedback is always welcome!

Digg! Delicious! Technorati Blinklist Furl Reddit

raspell shared library problems on OpenBSD

Posted by face on February 26, 2008

--- extconf.rb.orig     Tue Feb 26 12:03:53 2008
+++ extconf.rb  Tue Feb 26 11:52:16 2008
@@ -1,5 +1,7 @@
 require "mkmf"
+$LIBS += " -lstdc++ -laspell"

I recently installed the raspell gem on my development box as part of an evaluation of ultrasphinx.

Unfortunately, the native extension to raspell doesn’t link in the dynamic libraries it uses. A fix is provided below.

Note, using the raspell with ultrasphinx also caused a ruby core dump, this time in an assertion from the configuration for aspell. OpenBSD uses aspell-0.50.5 and after upgrading aspell to 0.60.5, raspell started working fine.

Here is the output from the dynamic library problem:

** Starting Rails with development environment...
ruby:/usr/local/lib/ruby/gems/1.8/gems/raspell-1.1/lib/ undefined symbol 'new_aspell_config'
lazy binding failed!
/home/face/urevz/vendor/plugins/ultrasphinx/lib/ultrasphinx/spell.rb:33: [BUG] Segmentation fault
ruby 1.8.6 (2007-09-24) [i386-openbsd4.2]

A little poking around and I noticed the native library,, doesn’t link in libaspell!

A quick fix is to add the libs to extconf.rb and rebuild the library:

  1. cd /usr/local/lib/ruby/gems/1.8/gems/raspell-1.1/ext
  2. edit extconf.rb and add the line $LIBS = ” -lstdc+ -laspell”
  3. sudo make clean
  4. sudo make
  5. sudo cp ../lib/.

Digg! Delicious! Technorati Blinklist Furl Reddit

Current Ruby and Ruby-Gem binaries for the current release of OpenBSD

Posted by face on February 12, 2008

Ruby-Forge Header

Here is how to install i386 binaries (please choose an OpenBSD mirror near you…and please buy a t-shirt):

sudo su

#for ksh/bash
export PKG_PATH=""

#or, for tcsh: setenv PKG_PATH=

# Now for Ruby
pkg_add ""

# Optional, but Recommend for Rails
pkg_add ""
pkg_add ""

# Now you could install rails if you wanted:
gem install rails

Rubyforge doesn’t allow you to have the same filename, even if it is in a different sub-package and release. Therefore, amd64 and sparc64 binaries are distributed directly from

For AMD64 (which runs in IA64 of course):

sudo su
export PKG_PATH=""

# or, setenv PKG_PATH ""

pkg_add ruby- ruby-iconv- ruby-gems-1.0.1

# Perhaps Ruby-On-Rails:
gem install rails
For sparc64:

sudo su
export PKG_PATH=""

# or, setenv PKG_PATH ""

pkg_add ruby- ruby-iconv- ruby-gems-1.0.1

# Maybe Rails?
gem install rails

Thanks, I find these useful and I hope you do too. I now do all my development from a OpenBSD (patched) Desktop and have made these binaries as secure as possible. All binaries are produced behind pf firewalls.

I have a old Mac Mini I can install macppc and produce binaries…if there is a demand.

Puffy swimming

Digg! Delicious! Technorati Blinklist Furl Reddit

undelete / unrm for OpenBSD 4.2 with dls

Posted by face on January 14, 2008

While nightly backups and SVN are great, sometimes we make a mistake and rm something we need back. This article should also be useful to anyone who needs to do forensic analysis of a filesystem. This technique should also work under freebsd, netbsd, linux, solaris, dos, windows, etc. just the installation part of tools would be different and you might need a -t option to dls.

I did this yesterday on my development box under the /home partition. The first thing is to try to ensure nothing overwrites the deleted inodes.

In this example /home is /dev/wd0e. You can use df to determine your setup.

Get on the console and bring her into single user mode:
shutdown  now
# or, if you are not alone and want to give peope 2 minutes:   shutdown  +2

Now that we are in single user mode, unmount the disk

umount /home

Ok, the inodes are safe. Let’s undelete them. On OpenBSD we have, The Sleuth Kit an evolution of TCT and unrm is now called dls.

sudo su
export PKG_PATH="" 
# or, setenv PKG_PATH "" 
pkg_add sleuthkit

# Now, we will need tcl for the comeforth script referenced below.  Intall if you don't have:
pkg_add tcl-8.4.7p5
ln -s  /usr/local/bin/tclsh8.4 /usr/bin/tclsh

Now my /var has lots of free space. The following command will find all deleted inodes and place them in a file.

dls  /dev/wd0e > /var/tmp/undelete.bin

You can also yous fls on the raw device to report on directory information (file names). To get information on deleted sub directories, you would need to determine which inodes are directories and then use fls on those inodes.

If it is a text or source file you are looking for, you can just use something like less to search undelete.bin now.

Otherwise, if you want to extract files, I recommend comeforth. TCT does not build correctly on OpenBSD anymore. It is possible to get enough of TCT working for lazarus to run, but comeforth is much faster.

First, download and open comeforth. Now make it executable chmod +x ~face/comeforth/comeforth. We should already have tclsh installed above so now we run comeforth which is an interactive script. I am looking for Ruby-On-Rails files under app. So first I create a file of regular expressions that will match the output of the file command for the files I want to retrieve:

echo 'ASCII' > files
echo 'HTML' >> files
Comeforth is an interactive script. I accepted all the defaults except for the File type regex which I set to < filesHere is a typescript from a session:
Script started on Mon Jan 14 15:13:05 2008
e5:/var/tmp> ~face/comeforth/comeforth

comeforth 1.12, Copyright (c) 2003-2004 Danamis Associates (
This program comes with ABSOLUTELY NO WARRANTY; this is free software, and you
are welcome to redistribute it under certain conditions; for details view the
GNU Lesser General Public License at

Data file: undelete.bin

Data block size: 4096

Recovery directory: recov

File type regex
([?] for help): < files

- Found 2 regexes to use in 'files'.

Block work dir: comeforth-5045.tmp

Progress indicator block interval: 24

Start at block: 1

Scanning data for matching blocks...
24, 0.01%, 1411.8 per sec, 2.9 min rem...

# Lots of output deleted ...

Finished scanning filesystem data in 0:09:26.

Inspect and assemble files? ([y]es/[q]uit): q

And thats it. You now have all your deleted files in recov. You can use grep to find the specific files you want. If the arglist is too long, then break it down with find. For example, lets say you where looking for the ruby class UsersController:

find . -print | xargs -L 10000 grep "^class UsersController"

You may have to delete some nulls at the end of your files. I tried dls -b but that didn’t seem to help.

I hope this is useful to someone else out there…


Digg! Delicious! Technorati Blinklist Furl Reddit

The official OpenBSD Ruby 1.8.6-111 ports have been checked into CVS

Posted by face on December 31, 2007

Thanks to Bernd Ahlers, the official OpenBSD ruby 1.8.6-111 ports are now in the OpenBSD CVS Repository.

I have updated my post and associated binaries Ruby 1.8.6p110 on OpenBSD 4.2 to use the official 111 port instead of mine.

Digg! Delicious! Technorati Blinklist Furl Reddit

rubygems-1.0.1 on OpenBSD

Posted by face on December 23, 2007

I started a real update to the port but was unable to complete it quickly. I have more pressing coding tasks and can’t spend the time now to complete the port.

My short term solution is to just use ruby gems to update itself (assuming you already have gem installed, if not see below).

sudo gem update --system

Then, if you want to, update some installed gems (like rails -> 2.0.2)

sudo gem update rails

I updated my systems which run my ruby-1.8.6p110 port. However it should work with the stock ruby and gem which you can install via OpenBSD ports:

sudo su
# setenv PKG_PATH "" 
export  PKG_PATH="" 
pkg_add ruby ruby-gems ruby-iconv

gem update --system

If anyone has a pointer to a completed native port, please let me know.

Digg! Delicious! Technorati Blinklist Furl Reddit

Ruby 1.8.6-111 on OpenBSD 4.2 (and Ruby-On-Rails)

Posted by face on November 04, 2007

Update: December 31, 2007: Official OpenBSD 1.8.6 patch level 111 ports have been checked into CVS. This post and the associated binaries have been updated to use the official port. My unofficial port has been discarded.

This article describes building a ruby port from source. If you want the binaries produced by this article, I have released them via this Rubyforge project.

If your ports tree is not already prepped, please begin by prepping your ports tree.

There is a message that you may run out of memory without modifying login.conf. My login.conf was already modified. My login is a member of the staff group:

We will be using the official OpenBSD port from CVS.

We will start by installing the prerequisites as binaries because if you have a fast internet connection, it is faster than building the ports from src:
sudo su
setenv PKG_PATH "" 
# or export PKG_PATH="" 
pkg_add libiconv gdbm tk-8.4.7p1

Now lets update the ruby port from CVS HEAD and build it. I did this on December 31, 2007 and the snapshot of the source code to the port can be found here: ruby.tgz

export   # pls choose a mirror near you
# setenv CVSROOT
cd :/usr/ports/lang/ruby
cvs up -rHEAD .
make package
Now we can install it:
sudo su
setenv PKG_PATH './:'
# or:  export PKG_PATH='./:'
cd /usr/packages/i386/all
pkg_add ruby- 
# Or, of you already have ruby installed, try:  pkg_add -r ruby-
And for a bonus, lets install Ruby-On-Rails:
pkg_add ruby-gems ruby-iconv-  mysql-server mysql-client

# Update gems to version 1.0.1, apparently needed for rails 2.0.2 but will break things like ruby-openid-1.1.4
gem update --system

gem install rails
# For Rails 1.2.6
# gem install -v 1.2.6 rails

# Now let's optimize ruby for mysql
gem install mysql

# finally, mongrel rocks
gem install mongrel --include-dependencies
gem install mongrel_cluster

If this is a new MySQL install, don’t forget to follow the instructions in /usr/local/share/doc/mysql/README.OpenBSD.


Digg! Delicious! Technorati Blinklist Furl Reddit

Prepping Ports for OpenBSD

Posted by face on November 04, 2007

This follows 15.3 – Working with ports. Here are the main steps:

Your /etc/mk.conf should look similar to this:

The assumption is you have placed the 4.2 ports.tar.gz in /tmp:
sudo su
cd /usr
tar xpfz /tmp/ports.tar.gz
chgrp -R wsrc /usr/ports
find /usr/ports -type d -exec chmod g+w {} \;
mkdir -p /usr/distfiles /usr/obj/ports /usr/packages
chgrp -R wsrc /usr/distfiles /usr/obj/ports /usr/packages
chmod g+w /usr/distfiles /usr/obj/ports /usr/packages

No ensure you are part of the wsrc group. You should be able to do the following as a normal user:

setenv CVSROOT "" 
#or, export
cd /usr
cvs -q up -P -rOPENBSD_4_2 ports

Reference: 15.3 – Working with ports

Digg! Delicious! Technorati Blinklist Furl Reddit

Building A OpenBSD-4.2-current (patched) CD

Posted by face on November 04, 2007

I have to update several OpenBSD systems so I find it easiest to make a 4.2-current (patched) distribution. This can also be burned to a cd so new installs or upgrades via cd are also patched.

So, I have just upgraded to 4.2 OpenBSD. I am now going to build a patched distribution, patching the system in the process.

Let’s do it:

sudo su
# setenv CVSROOT "" 
export CVSROOT="" 
cd /usr
mv src src-4.1
# rm -rf src-4.1 &
mkdir src
cd /usr/src
tar xpfz /tmp/src.tar.gz
tar xpfz /tmp/sys.tar.gz 

cd /usr
cvs -q up -P -rOPENBSD_4_2 src 
cd src/sys/arch/i386/conf

#use GENERIC for non MP or HT servers
/usr/sbin/config GENERIC.MP
cd ../compile/GENERIC.MP
make clean && make depend && make

# if you did not do the 'set image' step above, then just 'make install'
rm -f /
ln / /
cp bsd /nbsd
mv /nbsd /


Log back in as root after reboot, you should see the kernel now has a name like “OpenBSD 4.2-stable (GENERIC.MP) #0”. The -stable means the eratta patches where applied.

cd /usr/src
rm -rf /usr/obj/*
cd /usr/src
make obj
cd /usr/src/etc && env DESTDIR=/ make distrib-dirs
cd /usr/src
make build
# This will take a while


When the build completes, your system is up to date. If you have only 1 machine, you are done. However, I need patched distribution sets for my other boxes:

export DESTDIR=/usr/dest
export RELEASEDIR=/usr/rel
# setenv DESTDIR /usr/dest
# setenv RELEASEDIR /usr/rel
cd /usr/src/distrib/crunch && make obj depend all install
test -d ${DESTDIR} && mv ${DESTDIR} ${DESTDIR}.old && rm -rf ${DESTDIR}.old &
cd /usr/src/etc
make release
cd /usr/src/distrib/sets/
sh checkflist

We now have a release in /usr/rel. I combine this with an official distribution to get the x-windows sets as I am not building them here. Basically, I copy everything except MD5 from /usr/rel into the 4.2/i386 dist dir that has everything from the ftp server except install42.iso. Then I edit the MD5 to include the new checksums. Now put everthing under a directory structure OpenBSD/4.2/i386. For this example, say it is all in /tmp/OpenBSD/4.2/i386 we can now make the iso cd image:

cd /tmp
mkhybrid -A "OpenBSD-4.2-stable-i386" -P "Me" -V  "OpenBSD-4.2-stable-i386" -r -b 4.2/i386/cdbr -c 4.2/i386/boot.catalog -o OpenBSD-4.2-Stable-i386.iso OpenBSD


Digg! Delicious! Technorati Blinklist Furl Reddit

Remote Upgrade of OpenBSD 4.1 to 4.2

Posted by face on November 04, 2007

The first assumption is that you have a remote OpenBSD server running 4.1 with a serial port console.

Before you start, read the OpenBSD 4.2 upgrade faq here . For the most part, you will be following that faq. This is simply a supplemental guide to get things rolling…

I’m upgrading a i386 system. bsd.rd is the install/upgrade kernel. If we already have a remote system running, we can upgrade bsd.rd first, then boot from it and upgrade the rest of the system.

sudo cp /bsd.rd /bsd.rd-4.1
sudo cp bsd.rd /bsd.rd
sudo shutdown -h now "Upgrading to 4.2"  # if you don't get a boot prompt, read the NOTE below

Now get on the serial console. You should see that the os is halted and Please press any key to reboot.. When you see the boot prompt type:

boot bsd.rd

Digg! Delicious! Technorati Blinklist Furl Reddit

Flex 2 SDK on OpenBSD 4.1

Posted by face on October 15, 2007

Java is a prerequisite. I tried building a simple mxml file with kaffe on OpenBSD, but kaffe failed with a CDATA error for flex 2 and then simply dumped core flex 3 beta. So before we start, lets build and install JDK for OpenBSD from source. Once you have built jdk-1.5, install it and ensure $JAVA_HOME is set and $JAVA_HOME/bin is on your path.

If you don’t already have wget and unzip installed, we can install them now:

sudo csh
setenv PKG_PATH
pkg_add wget unzip

The flex sdk is free and comes in a multi-platform binary format. Adobe forgot to put it in a directory, so lets make one for it now and then grab and unzip the sdk:

sudo mkdir /usr/local/flex2sdk-2.01
sudo ln -s /usr/local/flex2sdk-2.01 /usr/local/flex
cd /tmp
cd /usr/local/flex
sudo unzip /tmp/
rm -f  /tmp/
sudo chmod +x /usr/local/flex/bin/mxml

That should do it. Lets try it out. Place the following in a file called hello.mxml:

<?xml version="1.0" encoding="utf-8"?>
<mx:Application xmlns:mx="" horizontalAlign="center" verticalAlign="center">
  <mx:Panel title="Yo">
    <mx:Text htmlText="Hello &lt;b&gt;new&lt;/b&gt; World!" />

And now compile it:

/usr/local/flex/bin/mxmlc hello.mxml

And you should end up with hello.swf:

Digg! Delicious! Technorati Blinklist Furl Reddit

JDK on OpenBSD 4.1

Posted by face on October 15, 2007

JDK 1.5 in the 4.1 ports is crufty at this point. I recommend back porting the current HEAD version (assuming you already have your ports tree configured and up to date, are part of the wsrc group, and the wsrc group has write permissions to /usr/ports. If my assumption is wrong, please configure your ports tree as suggested here 15.3 – Working with ports ):

Before we start lets get the manual files and put them in /usr/distfiles:

Now Lets update the port, install prerequisites as binaries to save time, and build the package (You probably need to up datasize-cur in /etc/login.conf for the user building java to be 1024M, mine was when I just built it):

setenv CVSROOT
setenv PKG_PATH
sudo pkg_add gtar-1.16.1p0 zip unzip bzip2 openmotif libiconv kaffe jikes nspr gmake wget
cd /usr/ports/devel/jdk/1.5
cvs up -rHEAD .

This updated jdk/1.5/Makefile will fail in OpenBSD 4.1 after several hours with the following error: /bin/sh: syntax error: `(' unexpected. I tracked this down to the \(TM\) in the COMMENT-main and COMMENT-jre in the Makefile. Remove these \(TM\)s in those two comment variables in the Makefile with your favorite editor and we can continue.

cd /usr/distfiles
cd /usr/ports/devel/jdk/1.5
make makesum # Need to rebuild checksums as 4.2 OpenBSD uses longer keys
make package  # This will take a while...a long while

After several hours we should have a shiny new native JDK for OpenBSD. We can install it now:

unsetenv PKG_PATH
pkg_add /usr/packages/i386/all/jdk-
chmod -R +r,+x /usr/local/jdk-1.5.0   #Let's let everyone run it

Digg! Delicious! Technorati Blinklist Furl Reddit

Unofficial Guide to Install OpenBSD on a Dedicated Server

Posted by face on October 14, 2007

Update: I have dumped for poor customer service and slow connectivity to my "100Mbps" dedicated server. I now lease my servers from and couldn't be happier.
I used to lease 6 dedicated servers from 1&1. I used to like 1and1 because they had an excellent network backbone that peers with about everyone (though now it is slow and I think oversold where my dedicated servers were). Combined with their raw network speed is all the automated tools I need for self service (serial console, web based hard boot, web based rescue boot into linux, web based reimage, dns). Customer service is adequate if you have no problems (24x7 fast to answer the phone, however, they can't do much as their systems are automated and they have no two way communication with the Admins). is German engineered...go figure. They offer managed services as well (windows) but if you are reading this guide you probably don't need that!

I love OpenBSD because it is lean, fast, secure, free, and did I mention secure. I started using OpenBSD many years ago for firewalls. These days I use it for everything that doesn't need massive thread scaling (java or MySQL at high load MP hardware). What a bonus to have the best firewall on the market built into every server.

Digg! Delicious! Technorati Blinklist Furl Reddit

OpenBSD Guide to configure an Anonymous ftp server.

Posted by face on October 14, 2007

This is a quick guide to configure PF and fptd for an OpenBSD 4.1 system.

All the software necessary comes preinstalled with OpenBSD 4.1 so all we need is some configuration.

NOTE: The following commands assume you have just installed OpenBSD 4.1 and haved not done any previous configuration. If this is not a virgin OpenBSD system make sure you study the current configuration and know what you are doing before cutting and pasting the commands below. Use the commands below at your own risk.

First, I assume you already have your basic firewall configured and running. I guess if your box is behind another firewall you might choose not to have one. If you don't have one you can install this this sample pf.conf to /etc/pf.conf before continuing (just change vr0 to be your interface going to the internet).

Now, let's configure everything. I use /var/www/ftp as my anonymouse root (www so my chrooted Apache can provide HTTP access to my ftp server dirs):

mkdir -p /var/www/ftp
groupadd ftp
useradd -c 'Anonymous ftp' -d /var/www/ftp -g ftp -s /sbin/nologin ftp
cd /var/www/ftp
mkdir -p bin etc pub
chmod 555 . pub
chmod 511 bin etc
cd etc
cp /etc/pwd.db .
cp /etc/group .
echo 'Welcome to my server....All transfers are logged.' >> motd
cd ..
chmod -R -w .
echo '# Set to NO if ftpd is running out of inetd'  >> /etc/rc.conf.local
echo 'ftpd_flags="-DllUSA"    # for non-inetd use: "-D"' >> /etc/rc.conf.local

echo 'net.inet.ip.porthifirst=6000'       >> /etc/sysctl.conf
echo 'net.inet.ip.porthilast=60100'       >> /etc/sysctl.conf

echo ''               >> /etc/pf.conf
echo '# FTP'              >> /etc/pf.conf
echo 'pass in on $ext_if proto {tcp,udp} from any to any \'   >> /etc/pf.conf
echo '  port { ftp, ftp-data, 6000:60100 } keep state'    >> /etc/pf.conf
echo ''               >> /etc/pf.conf

pfctl -f /etc/pf.conf
/usr/libexec/ftpd -DllUSA

That is it. Of course you might actually want to put some read only files in /var/www/ftp/pub/ for people to anonlymously get.

man ftpd
man pf

Digg! Delicious! Technorati Blinklist Furl Reddit