Update July 16th, 2008: Sphinx sphinx-0.9.8 has been released and this port has been updated.

Here is a OpenBSD port of Sphinx, Free open-source SQL full-text search engine. Sphinx is a wonderfully fast and memory efficient deep text search engine. I have found integrates nicely with Ruby and Ruby-On-Rails.

I hope at some point to get this port committed to the OpenBSD CVS repository. In the meantime this will be the ports home. Till it hits the OpenBSDs repository, it will live in my SVN repository here.

You can download a snapshot of the source code for the port here: sphinx.tgz. So far this port has been tested with MySQL and Postgresql on OpenBSD 4.2 and OpenBSD 4.3-betaOpenBSD-4.4-beta on the i386 platform with the Ultrasphinx Ruby-On-Rails plugin.

Instructions to build the port

If your ports tree is not already prepped, please begin by prepping your ports tree.

1
2
3
4
5
6

sudo su
export PKG_PATH="ftp://ftp2.usa.openbsd.org/pub/OpenBSD/4.3/packages/i386"
# or, setenv PKG_PATH "ftp://ftp2.usa.openbsd.org/pub/OpenBSD/4.3/packages/i386"
pkg_add libiconv mysql-server mysql-client
exit

Now we are ready to build the sphinx port:

1
2
3
4
5
6

cd /usr/ports/textproc
lynx --source "http://myutil.com/ports/4.4/sphinx.tgz" > sphinx.tgz
tar xvfz sphinx.tgz 
cd sphinx
make install  # or make package

1
2

env FLAVOR=pgsql make install

Thanks, and feedback is always welcome!

1
2
3
4
5
6
7
8
9
10
11

--- extconf.rb.orig     Tue Feb 26 12:03:53 2008
+++ extconf.rb  Tue Feb 26 11:52:16 2008
@@ -1,5 +1,7 @@
 require "mkmf"
 
+$LIBS += " -lstdc++ -laspell"
+
 have_header("ruby.h")
 have_header("aspell.h")
 have_library("aspell")

I recently installed the raspell gem on my development box as part of an evaluation of ultrasphinx.

Unfortunately, the native extension to raspell doesn’t link in the dynamic libraries it uses. A fix is provided below.

Note, using the raspell with ultrasphinx also caused a ruby core dump, this time in an assertion from the configuration for aspell. OpenBSD uses aspell-0.50.5 and after upgrading aspell to 0.60.5, raspell started working fine.

** Starting Rails with development environment...
ruby:/usr/local/lib/ruby/gems/1.8/gems/raspell-1.1/lib/raspell.so: undefined symbol 'new_aspell_config'
lazy binding failed!
/home/face/urevz/vendor/plugins/ultrasphinx/lib/ultrasphinx/spell.rb:33: [BUG] Segmentation fault
ruby 1.8.6 (2007-09-24) [i386-openbsd4.2]

A little poking around and I noticed the native library, raspell.so, doesn’t link in libaspell!

A quick fix is to add the libs to extconf.rb and rebuild the library:

1. cd /usr/local/lib/ruby/gems/1.8/gems/raspell-1.1/ext
2. edit extconf.rb and add the line $LIBS = ” -lstdc+ -laspell”
3. sudo make clean
4. sudo make
5. sudo cp raspell.so ../lib/.

Here is how to install i386 binaries (please choose an OpenBSD mirror near you…and please buy a t-shirt):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

sudo su

#for ksh/bash
export PKG_PATH="ftp://ftp3.usa.openbsd.org/pub/OpenBSD/4.2/packages/i386"

#or, for tcsh: setenv PKG_PATH=ftp://ftp3.usa.openbsd.org/pub/OpenBSD/4.2/packages/i386

# Now for Ruby
pkg_add "http://rubyforge.org/frs/download.php/32289/ruby-1.8.6.111.tgz"

# Optional, but Recommend for Rails
pkg_add "http://rubyforge.org/frs/download.php/32290/ruby-gems-1.0.1.tgz"
pkg_add "http://rubyforge.org/frs/download.php/32291/ruby-iconv-1.8.6.111.tgz"

# Now you could install rails if you wanted:
gem install rails

Rubyforge doesn’t allow you to have the same filename, even if it is in a different sub-package and release. Therefore, amd64 and sparc64 binaries are distributed directly from MyUtil.com.

1
2
3
4
5
6
7
8
9
10

sudo su
export PKG_PATH="http://myutil.com/ports/4.2/amd64/:ftp://ftp3.usa.openbsd.org/pub/OpenBSD/4.2/packages/amd64/"

# or, setenv PKG_PATH "http://myutil.com/ports/4.2/amd64/:ftp://ftp3.usa.openbsd.org/pub/OpenBSD/4.2/packages/amd64/"

pkg_add ruby-1.8.6.111 ruby-iconv-1.8.6.111 ruby-gems-1.0.1

# Perhaps Ruby-On-Rails:
gem install rails

1
2
3
4
5
6
7
8
9
10

sudo su
export PKG_PATH="http://myutil.com/ports/4.2/sparc64/:ftp://ftp3.usa.openbsd.org/pub/OpenBSD/4.2/packages/sparc64/"

# or, setenv PKG_PATH "http://myutil.com/ports/4.2/sparc64/:ftp://ftp3.usa.openbsd.org/pub/OpenBSD/4.2/packages/sparc64/"

pkg_add ruby-1.8.6.111 ruby-iconv-1.8.6.111 ruby-gems-1.0.1

# Maybe Rails?
gem install rails

Thanks, I find these useful and I hope you do too. I now do all my development from a OpenBSD (patched) Desktop and have made these binaries as secure as possible. All binaries are produced behind pf firewalls.

I have a old Mac Mini I can install macppc and produce binaries…if there is a demand.

While nightly backups and SVN are great, sometimes we make a mistake and rm something we need back. This article should also be useful to anyone who needs to do forensic analysis of a filesystem. This technique should also work under freebsd, netbsd, linux, solaris, dos, windows, etc. just the installation part of tools would be different and you might need a -t option to dls.

I did this yesterday on my development box under the /home partition. The first thing is to try to ensure nothing overwrites the deleted inodes.

In this example /home is /dev/wd0e. You can use df to determine your setup.

shutdown  now
# or, if you are not alone and want to give peope 2 minutes:   shutdown  +2

Now that we are in single user mode, unmount the disk

umount /home

Ok, the inodes are safe. Let’s undelete them. On OpenBSD we have, The Sleuth Kit an evolution of TCT and unrm is now called dls.

sudo su
export PKG_PATH="ftp://ftp2.usa.openbsd.org/pub/OpenBSD/4.2/packages/i386" 
# or, setenv PKG_PATH "ftp://ftp2.usa.openbsd.org/pub/OpenBSD/4.2/packages/i386" 
pkg_add sleuthkit

# Now, we will need tcl for the comeforth script referenced below.  Intall if you don't have:
pkg_add tcl-8.4.7p5
ln -s  /usr/local/bin/tclsh8.4 /usr/bin/tclsh

Now my /var has lots of free space. The following command will find all deleted inodes and place them in a file.

dls  /dev/wd0e > /var/tmp/undelete.bin

You can also yous fls on the raw device to report on directory information (file names). To get information on deleted sub directories, you would need to determine which inodes are directories and then use fls on those inodes.

If it is a text or source file you are looking for, you can just use something like less to search undelete.bin now.

Otherwise, if you want to extract files, I recommend comeforth. TCT does not build correctly on OpenBSD anymore. It is possible to get enough of TCT working for lazarus to run, but comeforth is much faster.

First, download and open comeforth. Now make it executable chmod +x ~face/comeforth/comeforth. We should already have tclsh installed above so now we run comeforth which is an interactive script. I am looking for Ruby-On-Rails files under app. So first I create a file of regular expressions that will match the output of the file command for the files I want to retrieve:

echo 'ASCII' > files
echo 'HTML' >> files

Script started on Mon Jan 14 15:13:05 2008
e5:/var/tmp> ~face/comeforth/comeforth

comeforth 1.12, Copyright (c) 2003-2004 Danamis Associates (http://danamis.com).
This program comes with ABSOLUTELY NO WARRANTY; this is free software, and you
are welcome to redistribute it under certain conditions; for details view the
GNU Lesser General Public License at http://www.gnu.org.

Data file: undelete.bin

Data block size: 4096

Recovery directory: recov

File type regex
([?] for help): < files

- Found 2 regexes to use in 'files'.

Block work dir: comeforth-5045.tmp

Progress indicator block interval: 24

Start at block: 1

Scanning data for matching blocks...
24, 0.01%, 1411.8 per sec, 2.9 min rem...

# Lots of output deleted ...

Finished scanning filesystem data in 0:09:26.

Inspect and assemble files? ([y]es/[q]uit): q

And thats it. You now have all your deleted files in recov. You can use grep to find the specific files you want. If the arglist is too long, then break it down with find. For example, lets say you where looking for the ruby class UsersController:

find . -print | xargs -L 10000 grep "^class UsersController"

You may have to delete some nulls at the end of your files. I tried dls -b but that didn’t seem to help.

I hope this is useful to someone else out there…

http://wiki.sleuthkit.org/index.php?title=Help_Documents http://wiki.sleuthkit.org/index.php?title=Tools_Using_TSK http://www.linuxhaxor.net/2007/12/26/undelete-files-in-linux-with-lazarus-and-unrm/

Thanks to Bernd Ahlers, the official OpenBSD ruby 1.8.6-111 ports are now in the OpenBSD CVS Repository.

I have updated my post and associated binaries Ruby 1.8.6p110 on OpenBSD 4.2 to use the official 111 port instead of mine.

I started a real update to the port but was unable to complete it quickly. I have more pressing coding tasks and can’t spend the time now to complete the port.

My short term solution is to just use ruby gems to update itself (assuming you already have gem installed, if not see below).

sudo gem update --system

Then, if you want to, update some installed gems (like rails -> 2.0.2)

sudo gem update rails

I updated my systems which run my ruby-1.8.6p110 port. However it should work with the stock ruby and gem which you can install via OpenBSD ports:

sudo su
# setenv PKG_PATH "ftp://ftp2.usa.openbsd.org/pub/OpenBSD/4.2/packages/i386" 
export  PKG_PATH="ftp://ftp2.usa.openbsd.org/pub/OpenBSD/4.2/packages/i386" 
pkg_add ruby ruby-gems ruby-iconv

gem update --system

If anyone has a pointer to a completed native port, please let me know.

This article describes building a ruby port from source. If you want the binaries produced by this article, I have released them via this Rubyforge project.

If your ports tree is not already prepped, please begin by prepping your ports tree.

staff:\
        :datasize-cur=1024M:\
        :datasize-max=infinity:\
        :maxproc-max=256:\
        :maxproc-cur=128:\
        :ignorenologin:\
        :requirehome@:\
        :tc=default:

We will be using the official OpenBSD port from CVS.

sudo su
setenv PKG_PATH "ftp://ftp2.usa.openbsd.org/pub/OpenBSD/4.2/packages/i386" 
# or export PKG_PATH="ftp://ftp2.usa.openbsd.org/pub/OpenBSD/4.2/packages/i386" 
pkg_add libiconv gdbm tk-8.4.7p1
exit

Now lets update the ruby port from CVS HEAD and build it. I did this on December 31, 2007 and the snapshot of the source code to the port can be found here: ruby.tgz

export CVSROOT=anoncvs@anoncvs3.usa.openbsd.org:/cvs   # pls choose a mirror near you
# setenv CVSROOT anoncvs@anoncvs3.usa.openbsd.org:/cvs
cd :/usr/ports/lang/ruby
cvs up -rHEAD .
make package

sudo su
setenv PKG_PATH './:ftp://ftp2.usa.openbsd.org/pub/OpenBSD/4.2/packages/i386'
# or:  export PKG_PATH='./:ftp://ftp2.usa.openbsd.org/pub/OpenBSD/4.2/packages/i386'
cd /usr/packages/i386/all
pkg_add ruby-1.8.6.111 
# Or, of you already have ruby installed, try:  pkg_add -r ruby-1.8.6.111

pkg_add ruby-gems ruby-iconv-1.8.6.111  mysql-server mysql-client

# Update gems to version 1.0.1, apparently needed for rails 2.0.2 but will break things like ruby-openid-1.1.4
gem update --system

gem install rails
# For Rails 1.2.6
# gem install -v 1.2.6 rails

# Now let's optimize ruby for mysql
gem install mysql

# finally, mongrel rocks
gem install mongrel --include-dependencies
gem install mongrel_cluster

If this is a new MySQL install, don’t forget to follow the instructions in /usr/local/share/doc/mysql/README.OpenBSD.

Enjoy!

This follows 15.3 – Working with ports. Here are the main steps:

Your /etc/mk.conf should look similar to this:

SUDO=/usr/bin/sudo
WRKOBJDIR=/usr/obj/ports
DISTDIR=/usr/distfiles
PACKAGE_REPOSITORY=/usr/packages
USE_SYSTRACE=Yes

sudo su
cd /usr
tar xpfz /tmp/ports.tar.gz
chgrp -R wsrc /usr/ports
find /usr/ports -type d -exec chmod g+w {} \;
mkdir -p /usr/distfiles /usr/obj/ports /usr/packages
chgrp -R wsrc /usr/distfiles /usr/obj/ports /usr/packages
chmod g+w /usr/distfiles /usr/obj/ports /usr/packages
exit

No ensure you are part of the wsrc group. You should be able to do the following as a normal user:

setenv CVSROOT "anoncvs@anoncvs3.usa.openbsd.org:/cvs" 
#or, export CVSROOT=anoncvs@anoncvs3.usa.openbsd.org:/cvs
cd /usr
cvs -q up -P -rOPENBSD_4_2 ports

Reference: 15.3 – Working with ports

I have to update several OpenBSD systems so I find it easiest to make a 4.2-current (patched) distribution. This can also be burned to a cd so new installs or upgrades via cd are also patched.

So, I have just upgraded to 4.2 OpenBSD. I am now going to build a patched distribution, patching the system in the process.

Let’s do it:

sudo su
# setenv CVSROOT "anoncvs@anoncvs3.usa.openbsd.org:/cvs" 
export CVSROOT="anoncvs@anoncvs3.usa.openbsd.org:/cvs" 
cd /usr
mv src src-4.1
# rm -rf src-4.1 &
mkdir src
cd /usr/src
tar xpfz /tmp/src.tar.gz
tar xpfz /tmp/sys.tar.gz 

cd /usr
cvs -q up -P -rOPENBSD_4_2 src 
cd src/sys/arch/i386/conf

#use GENERIC for non MP or HT servers
/usr/sbin/config GENERIC.MP
cd ../compile/GENERIC.MP
make clean && make depend && make

# if you did not do the 'set image bsd.mp' step above, then just 'make install'
rm -f /obsd.mp
ln /bsd.mp /obsd.mp
cp bsd /nbsd
mv /nbsd /bsd.mp

sync
reboot

Log back in as root after reboot, you should see the kernel now has a name like “OpenBSD 4.2-stable (GENERIC.MP) #0”. The -stable means the eratta patches where applied.

cd /usr/src
rm -rf /usr/obj/*
cd /usr/src
make obj
cd /usr/src/etc && env DESTDIR=/ make distrib-dirs
cd /usr/src
make build
# This will take a while

reboot

When the build completes, your system is up to date. If you have only 1 machine, you are done. However, I need patched distribution sets for my other boxes:

export DESTDIR=/usr/dest
export RELEASEDIR=/usr/rel
# setenv DESTDIR /usr/dest
# setenv RELEASEDIR /usr/rel
cd /usr/src/distrib/crunch && make obj depend all install
test -d ${DESTDIR} && mv ${DESTDIR} ${DESTDIR}.old && rm -rf ${DESTDIR}.old &
mkdir -p ${DESTDIR} ${RELEASEDIR}
cd /usr/src/etc
make release
cd /usr/src/distrib/sets/
sh checkflist

We now have a release in /usr/rel. I combine this with an official distribution to get the x-windows sets as I am not building them here. Basically, I copy everything except MD5 from /usr/rel into the 4.2/i386 dist dir that has everything from the ftp server except install42.iso. Then I edit the MD5 to include the new checksums. Now put everthing under a directory structure OpenBSD/4.2/i386. For this example, say it is all in /tmp/OpenBSD/4.2/i386 we can now make the iso cd image:

cd /tmp
mkhybrid -A "OpenBSD-4.2-stable-i386" -P "Me" -V  "OpenBSD-4.2-stable-i386" -r -b 4.2/i386/cdbr -c 4.2/i386/boot.catalog -o OpenBSD-4.2-Stable-i386.iso OpenBSD

Thanks!

The first assumption is that you have a remote OpenBSD server running 4.1 with a serial port console.

Before you start, read the OpenBSD 4.2 upgrade faq here . For the most part, you will be following that faq. This is simply a supplemental guide to get things rolling…

I’m upgrading a i386 system. bsd.rd is the install/upgrade kernel. If we already have a remote system running, we can upgrade bsd.rd first, then boot from it and upgrade the rest of the system.

ftp ftp://ftp2.usa.openbsd.org/pub/OpenBSD/4.2/i386/bsd.rd
sudo cp /bsd.rd /bsd.rd-4.1
sudo cp bsd.rd /bsd.rd
sudo shutdown -h now "Upgrading to 4.2"  # if you don't get a boot prompt, read the NOTE below

Now get on the serial console. You should see that the os is halted and Please press any key to reboot.. When you see the boot prompt type:

boot bsd.rd

Java is a prerequisite. I tried building a simple mxml file with kaffe on OpenBSD, but kaffe failed with a CDATA error for flex 2 and then simply dumped core flex 3 beta. So before we start, lets build and install JDK for OpenBSD from source. Once you have built jdk-1.5, install it and ensure $JAVA_HOME is set and $JAVA_HOME/bin is on your path.

If you don’t already have wget and unzip installed, we can install them now:

sudo csh
setenv PKG_PATH ftp://ftp2.usa.openbsd.org/pub/OpenBSD/4.1/packages/i386
pkg_add wget unzip

The flex sdk is free and comes in a multi-platform binary format. Adobe forgot to put it in a directory, so lets make one for it now and then grab and unzip the sdk:

sudo mkdir /usr/local/flex2sdk-2.01
sudo ln -s /usr/local/flex2sdk-2.01 /usr/local/flex
cd /tmp
wget http://download.macromedia.com/pub/flex/sdk/flex2_sdk_hf1.zip
cd /usr/local/flex
sudo unzip /tmp/flex2_sdk_hf1.zip
rm -f  /tmp/flex2_sdk_hf1.zip
sudo chmod +x /usr/local/flex/bin/mxml

That should do it. Lets try it out. Place the following in a file called hello.mxml:

<?xml version="1.0" encoding="utf-8"?>
<mx:Application xmlns:mx="http://www.adobe.com/2006/mxml" horizontalAlign="center" verticalAlign="center">
  <mx:Panel title="Yo">
    <mx:Text htmlText="Hello &lt;b&gt;new&lt;/b&gt; World!" />
  </mx:Panel>
</mx:Application>

And now compile it:

/usr/local/flex/bin/mxmlc hello.mxml

And you should end up with hello.swf:

JDK 1.5 in the 4.1 ports is crufty at this point. I recommend back porting the current HEAD version (assuming you already have your ports tree configured and up to date, are part of the wsrc group, and the wsrc group has write permissions to /usr/ports. If my assumption is wrong, please configure your ports tree as suggested here 15.3 – Working with ports ):

• bsd-jdk15-patches-6.tar.bz2 from www.eyesbeyond.com
• jdk-1_5_0_12-solaris-i586.tar.Z from java.sun.com

Now Lets update the port, install prerequisites as binaries to save time, and build the package (You probably need to up datasize-cur in /etc/login.conf for the user building java to be 1024M, mine was when I just built it):

setenv CVSROOT anoncvs@anoncvs3.usa.openbsd.org:/cvs
setenv PKG_PATH ftp://ftp2.usa.openbsd.org/pub/OpenBSD/4.1/packages/i386
sudo pkg_add gtar-1.16.1p0 zip unzip bzip2 openmotif libiconv kaffe jikes nspr gmake wget
cd /usr/ports/devel/jdk/1.5
cvs up -rHEAD .

This updated jdk/1.5/Makefile will fail in OpenBSD 4.1 after several hours with the following error: /bin/sh: syntax error: `(' unexpected. I tracked this down to the \(TM\) in the COMMENT-main and COMMENT-jre in the Makefile. Remove these \(TM\)s in those two comment variables in the Makefile with your favorite editor and we can continue.

cd /usr/distfiles
wget http://www.java.net/download/tiger/tiger_u12/jdk-1_5_0_12-fcs-src-b04-jrl-02_may_2007.jar
wget http://www.java.net/download/tiger/tiger_u12/jdk-1_5_0_12-fcs-bin-b04-jrl-02_may_2007.jar
wget http://www.apache.org/dist/xml/xalan-j/xalan-j_2_7_0-bin.tar.gz
cd /usr/ports/devel/jdk/1.5
make makesum # Need to rebuild checksums as 4.2 OpenBSD uses longer keys
make package  # This will take a while...a long while

After several hours we should have a shiny new native JDK for OpenBSD. We can install it now:

unsetenv PKG_PATH
pkg_add /usr/packages/i386/all/jdk-1.5.0.12.tgz
chmod -R +r,+x /usr/local/jdk-1.5.0   #Let's let everyone run it

Update: I have dumped 1and1.com for poor customer service and slow connectivity to my "100Mbps" dedicated server. I now lease my servers from m5hosting.com and couldn't be happier.

I love OpenBSD because it is lean, fast, secure, free, and did I mention secure. I started using OpenBSD many years ago for firewalls. These days I use it for everything that doesn't need massive thread scaling (java or MySQL at high load MP hardware). What a bonus to have the best firewall on the market built into every server.

mkdir -p /var/www/ftp
groupadd ftp
useradd -c 'Anonymous ftp' -d /var/www/ftp -g ftp -s /sbin/nologin ftp
cd /var/www/ftp
mkdir -p bin etc pub
chmod 555 . pub
chmod 511 bin etc
cd etc
cp /etc/pwd.db .
cp /etc/group .
echo 'Welcome to my server....All transfers are logged.' >> motd
cd ..
chmod -R -w .
echo '# Set to NO if ftpd is running out of inetd'  >> /etc/rc.conf.local
echo 'ftpd_flags="-DllUSA"    # for non-inetd use: "-D"' >> /etc/rc.conf.local

echo 'net.inet.ip.porthifirst=6000'       >> /etc/sysctl.conf
echo 'net.inet.ip.porthilast=60100'       >> /etc/sysctl.conf

echo ''               >> /etc/pf.conf
echo '# FTP'              >> /etc/pf.conf
echo 'pass in on $ext_if proto {tcp,udp} from any to any \'   >> /etc/pf.conf
echo '  port { ftp, ftp-data, 6000:60100 } keep state'    >> /etc/pf.conf
echo ''               >> /etc/pf.conf

pfctl -f /etc/pf.conf
/usr/libexec/ftpd -DllUSA

References:
man ftpd
man pf