NativeSearchContrib

By default TWiki searches its database by forking the standard grep program in a sub-process to search through the text files in the database cache. This is usually pretty fast, but unfortunately in some environments (specifically with mod_perl) forking a sub-process is very expensive, and another approach is needed.

NativeSearch is a c-code module that can be plugged into TWiki to replace the default searching algorithm. It publishes an XS interface that allows it to be called from Perl, without having to fork a sub-process. It has comparable performance to the standard forking search in most environments, but is often significantly faster than forked search when TWiki is run under a CGI accelerator. The effect is most pronounced with mod_perl, but is also there for SpeedyCGI.

You require shell access to the server to install this module. Refer to TWiki:Codev.NativeSearch for a list of systems where the module has been successfully compiled and installed (and please update it if you find a new one).

Installation Instructions

Before you start, there are some prerequisites.

1. you must have ExtUtils::MakeMaker installed (available from CPAN)
2. you must have a C compiler (gcc has been tested). Most Unix/Linux systems will have this already. On Win32 MinGW is recommended.
3. you will need make (a standard development tool). Unix/Linux users will probably have this already; others may have to search. You are recommended to use nmake on Win32.
4. finally you will need the pcre library (required for perl compatible regular expressions). This is a standard Linux library and will be available in an RPM or Debian package or whatever for your system. It is also available from GnuWin32 for Windows. You need both the runtime library (binary) and also the developer header files, which are usually shipped in a seperate package.

You do not need to install anything in the browser to use this extension. The following instructions are for the administrator who installs the extension on the server where TWiki is running.

Like many other TWiki extensions, this module is shipped with a fully automatic installer script written using the BuildContrib.

• If you have TWiki 4.1 or later, you can install from the configure interface (Go to Plugins->Find More Extensions)
  • The webserver user has to have permission to write to all areas of your installation for this to work.
• If you have a permanent connection to the internet, you are recommended to use the automatic installer script
  • Just download the NativeSearchContrib_installer perl script and run it.
• Notes:
  • The installer script will:
    • Automatically resolve dependencies,
    • Copy files into the right places in your local install (even if you have renamed data directories),
    • check in new versions of any installed files that have existing RCS histories files in your existing install (such as topics).
    • If the $TWIKI_PACKAGES environment variable is set to point to a directory, the installer will try to get archives from there. Otherwise it will try to download from twiki.org or cpan.org, as appropriate.
    • (Developers only: the script will look for twikiplugins/NativeSearchContrib/NativeSearchContrib.tgz before downloading from TWiki.org)
  • If you don't have a permanent connection, you can still use the automatic installer, by downloading all required TWiki archives to a local directory.
    • Point the environment variable $TWIKI_PACKAGES to this directory, and the installer script will look there first for required TWiki packages.
      • $TWIKI_PACKAGES is actually a path; you can list several directories separated by :
    • If you are behind a firewall that blocks access to CPAN, you can pre-install the required CPAN libraries, as described at http://twiki.org/cgi-bin/view/TWiki/HowToInstallCpanModules
• If you don't want to use the installer script, or have problems on your platform (e.g. you don't have Perl 5.8), then you can still install manually:
  1. Download and unpack one of the .zip or .tgz archives to a temporary directory.
  2. Manually copy the contents across to the relevant places in your TWiki installation.
  3. Check in any installed files that have existing ,v files in your existing install (take care not to lock the files when you check in)
  4. Manually edit LocalSite.cfg to set any configuration variables.
  5. Run configure and enable the module, if it is a plugin.
  6. Repeat from step 1 for any missing dependencies.

Note if you are installing this extension in TWiki version < 4.2, please add these definitions to your Main.WebPreferences, if they are not already there:

• Set SYSTEMWEB = %TWIKIWEB%
• Set USERSWEB = %MAINWEB%

Building the c-code

cd to the root directory in your TWiki installation. Now, assuming you have administrator (root) access to the server:

$ cd tools/native_search
$ perl Makefile.PL
$ make install
$ perl test.pl -i -l NativeTWikiSearch test.pl Makefile.PL NativeTWikiSearch.xs

Make sure that perl test.pl returns something sane, and NOT a Segmentation fault. (test.pl is simply a stub interface to the native search, so it behaves like grep. The test above simply greps for the string "NativeSearch" in the files Makefile.PL and NativeTWikiSearch.xs)

If you do not have administrator (root) access to the server, you can still install. Follow the steps above, but use perl Makefile.PL INSTALLBASE=/your/dir to install to a different directory. Exactly where you install depends on your system configuration.

Now, in configure, under "Store Settings", you should be able to select the search algorithm TWiki::Store::SearchAlgorithms::Native (click the button first if you have TWiki 4.2 or later).

Run a TWiki search, and watch the web server error log for any problems.

Because performance can differ widely between two apparently similar webserver configurations, you are recommended to do your own benchmarking and select the search algorithm that performs best for you.

Contrib Info

Another great TWiki extension from the WikiRing - working together to improve your wiki experience!

Many thanks to the following sponsors for supporting this work:

• Wind River Systems http://windriver.com

Contrib Author(s):	Crawford Currie http://c-dot.co.uk
Copyright:	© WikiRing 2007
License:	GPL (Gnu General Public License)
Contrib Version:	15064 (27 Sep 2007)
Change History:
Dependencies:	Name Version Description ExtUtils::MakeMaker >=0 Required for building makefiles
Contrib Home:	http://twiki.org/cgi-bin/view/Plugins/NativeSearchContrib
Feedback:	http://twiki.org/cgi-bin/view/Plugins/NativeSearchContribDev
Appraisal:	http://twiki.org/cgi-bin/view/Plugins/NativeSearchContribAppraisal

Related Topics: TWikiContribs, DeveloperDocumentationCategory, AdminDocumentationCategory, TWikiPreferences