xyzzy xyzzy…

June 2013
M	T	W	T	F	S	S
« Sep
	1	2
3	4	5	6	7	8	9
10	11	12	13	14	15	16
17	18	19	20	21	22	23
24	25	26	27	28	29	30

spamassassin: learning from inbox and junk folders

filed terribly early in the morning by Dr. Dirk Husemann in: hacking,linux,void

average time to read 0:26 minutes

while tweaking the spam filters on our mail server i finally took the step of adding a cron job to learn from the inbox and junk folders of each user. as we are using spamassassin as part of our spam defense this basically involves a couple of invocations of sa-learn to

learn the “ham” from each users inbox folder
learn the “spam” from each users junk folder

below is the shell script that gets invoked by cron once a day:

#!/bin/bash
echo "updating spamassassin bayesian spam/ham filter"
echo
for userDir in /home/*; do

    user=$(basename $userDir)
    ham=$userDir/Maildir/{cur,new}
    spam=$userDir/Maildir/.Junk/{cur,new}

    echo "    learning from $user"

    echo "       spam: $spam"
    /usr/bin/sa-learn --no-sync --spam $spam | while read line; do
	echo "            $line"
    done

    echo "        ham: $ham"
    /usr/bin/sa-learn --no-sync --ham $ham | while read line; do
	echo "            $line"
    done

    echo
done
echo "syncing:"
/usr/bin/sa-learn --sync | while read line; do
    echo "    $line"
done
echo
echo "stats:"
sa-learn --dump magic | while read line; do
    echo "    $line"
done

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

#!/bin/bash

echo "updating spamassassin bayesian spam/ham filter"

echo

for userDir in /home/*; do

user=$(basename $userDir)

ham=$userDir/Maildir/{cur,new}

spam=$userDir/Maildir/.Junk/{cur,new}

echo " learning from $user"

echo " spam: $spam"

/usr/bin/sa-learn --no-sync --spam $spam | while read line; do

echo " $line"

done

echo " ham: $ham"

/usr/bin/sa-learn --no-sync --ham $ham | while read line; do

echo " $line"

done

echo

done

echo "syncing:"

/usr/bin/sa-learn --sync | while read line; do

echo " $line"

done

echo

echo "stats:"

sa-learn --dump magic | while read line; do

echo " $line"

done

the while read line; do ... done bits are there so that i can nicely indent the output of sa-learn.

works rather nicely.

no comment so far

all content posted on these pages is an expression of my own mind. my employer is welcome to share these opinions but then again he might not want to.

adding a USB monitor

filed around lunchtime by Dr. Dirk Husemann in: hacking,linux

average time to read 2:37 minutes

i’ve long been using an external 22″ monitor with my ubuntu linux powered X200 thinkpad. while ubuntu maverick (10.10) has some issues with attaching and detaching the second monitor and subsequent suspend–resume cycles (the second suspend after a detach would not resume, d’oh), ubuntu lucid (10.04.02) works just fine (as befitting a long-term-support release).

for quite a while i’ve had the old 20″ monitor which i had been using previously sitting on my desktop along with an USB DisplayLink adapter — the idea being to hook the old monitor up as well as a third monitor¹. the displaylink driver provided by ubuntu lucid seemed to work properly — the monitor’s screen would turn green on connecting it and a framebuffer device (/dev/fb1) and so i tried various recipes floating around on ubuntu forums and elsewhere — all promising to achieve a grand unified desktop comprising all three monitors — and they either didn’t work or if they achieved the grand unified desktop it was unusable.

so, after another prolonged period during which the monitor and the displaylink adapter gathered even more dust, i tried a different approach: give up on the grand unified desktop goal and instead just try to make use of the monitor. the idea this time was to start up a VNC server, then use a VNC client to directly render the server into the framebuffer device (/dev/fb1) provided by linux’s displaylink driver. the vncserver bit is actually quite easy:

[sourcecode language="bash"] vncserver -name hidden -geometry 1600×1200 -depth 16 :42 [/sourcecode]

which starts a VNC server for the :42 display.

the VNC client bit turned out to be a bit more difficult. ubuntu lucid does have directvnc client which is “a vnc client for the linux framebuffer device”². that client does seem to require keyboard and mouse access and in some configurations did not work at all or locked up my keyboard (not the mouse, though, funnily enough) or crashed the running X session, so no points on that one. further research luckily turned up vnc2dl by none other than quentin stafford-fraser one of the original VNC developers (and also the inventor of the webcam it seems). vnc2dl seemed a bit more promising and — after slightly modifying³ dldevice.c — did do the job:

[sourcecode language="bash"] sudo vnc2dl :42 [/sourcecode]

next up was fusing display :42 to my main display :0 on a keyboard and mouse level so that i could just move the mouse pointer over to the left and end up on display :42 — here x2x (in the equally named ubuntu package) came into play:

[sourcecode language="bash"] x2x -west -to :42 >/dev/null 2>&1 & [/sourcecode]

and, hey, presto!, both displays are linked mouse and keyboard wise.

only thing still bothering me was that cut and paste was not working. to fix that required adding

[sourcecode language="bash"] vncconfig -nowin & [/sourcecode]

to my .vnc/xstartup file.

to have firefox run on display :42 required creation of a new firefox profile — i added that to .vnc/xstartup as well.⁴

yep, even more screen real estate; can’t have enough of that: the ur-IDE emacs in one screen, instant messaging app pidgin on the other (for communicating with the team), the third screen would be really useful to host a firefox window with the API docs and so forth. ↩
see man directvnc for more information about that one. ↩
vnc2dl in the version on quentin’s github is hard-wired to 1280×1040/24bpp which my old monitor doesn’t quite grok, changing the wiring to 1600×1200/16bpp made it more grokkable for my setup. ↩
to get the new profile synchronized with the default profile i used firefox’s recently added Firefox Sync add-on. ↩

comments (2)

all content posted on these pages is an expression of my own mind. my employer is welcome to share these opinions but then again he might not want to.

fixing failing suspend on maverick system

filed around lunchtime by Dr. Dirk Husemann in: linux

average time to read 1:00 minutes

i recently got my first real desktop system — as in: not a thinkpad. it’s an 8 core, 8GB, 1 terabyte system with an nvidia graphics chipset allowing two large displays to be powered: really sweet. installing ubuntu maverick 10.10 from the alternate install CD¹ was simple and straight forward. installing the latest nvidia drivers from the x-updates ppa and running nvidia-xconfig provides the correct configuration for the 1920×1200 display (second display on order and, except for suspend–resume, everything works fine.

the issue with suspend–resume is that the machine goes through the motions of suspending but then just comes back d’oh. googling around for “maverick suspend problem” i stumble over a post in the ubuntu forums citing the maverick release notes —

When the XHCI module is loaded for USB 3.0 operation the system cannot suspend. Manually unloading XHCI will allow suspend to complete normally. To avoid future suspend problems, the workaround is to add SUSPEND_MODULES=”xhci-hcd” to /etc/pm/config.d/unload_module then the system can suspend normally.

— hmph. following those instructions, suspend–resume are working.

…to be able to use disk level encryption! the home directory encryption offered by the normal ubuntu install CD suffers from path length issues and also stability issues. disk level encryption has so far worked quite well and we use it on all our machines now. ↩

no comment so far

all content posted on these pages is an expression of my own mind. my employer is welcome to share these opinions but then again he might not want to.

debianized rezzme

filed mid-morning by Dr. Dirk Husemann in: from the grid,linux

average time to read 0:45 minutes

the latest rezzme now contains the necessary code to build debian/ubuntu dpkgs — making it as easy to install rezzme packages on the latest debian/ubuntu systems as it already is for windows and mac OSX machines as the dpkg format allows me to explicitly specify the prerequisites (such as PyQt4).

while working on the dpkg support i also found out that debian/ubuntu has the very useful /etc/firefox-3.0, /etc/firefox-3.5, and /etc/thunderbird directories that allow me to provide package specific javascript to, for example, add a protocol handler for the rezzme: URI scheme. that along with gnome’s gconf tool gives us the same seamless rezzme user experience as on windows or mac OSX — unfortunately, it seems that nothing comparable exists for redhat/fedora based systems.¹

also the windows build code has been cleaned up and the no longer required dependency on setuptools removed.

at least i couldn’t find anything comparable; if you know how to do this — independent of minor version upgrades of firefox and thunderbird — i’d appreciate it very much to learn about that. ↩

comments (2)

all content posted on these pages is an expression of my own mind. my employer is welcome to share these opinions but then again he might not want to.

using F19/F20 to switch tabs in thunderbird 3

filed in the early evening by Dr. Dirk Husemann in: hacking,linux

average time to read 0:57 minutes

i just switched over from thunderbird 2 to the new sparkly thunderbird 3 — and i like it very much. particularly the tabbed user interface is quite nice but also lightning 1.0 seems to cope much better with those lotus notes generated calender invites

what did bother me was that i couldn’t find an easy to use function for keyconfig to bind the page left and page right keys (aka F19 and F20) of my X200 keyboard to “previous tab” and “next tab”. after some digging i stumbled across the mozilla thunderbird 3 tabmail page which then allowed me to figure out how to create the code for keyconfig:

[sourcecode language="javascript"] // next tab var windowManager = Components.classes['@mozilla.org/appshell/window-mediator;1']. getService(Components.interfaces.nsIWindowMediator); var winId = windowManager.getMostRecentWindow("mail:3pane"); var tabmail = winId.document.getElementById(‘tabmail’) tabmail.switchToTab((tabmail.tabContainer.selectedIndex + 1) % tabmail.tabInfo.length) [/sourcecode] and [sourcecode language="javascript"] // previous tab var windowManager = Components.classes['@mozilla.org/appshell/window-mediator;1']. getService(Components.interfaces.nsIWindowMediator); var winId = windowManager.getMostRecentWindow("mail:3pane"); var tabmail = winId.document.getElementById(‘tabmail’) tabmail.switchToTab((tabmail.tabContainer.selectedIndex + tabmail.tabInfo.length – 1) % tabmail.tabInfo.length) [/sourcecode]

copy & paste the code snippets and use keyconfig’s “add key” feature to bind the code to whatever key you’d like to use.

comments (2)

all content posted on these pages is an expression of my own mind. my employer is welcome to share these opinions but then again he might not want to.

fixing ugly {emacs, firefox} fonts on (x)ubuntu karmic

filed mid-afternoon by Dr. Dirk Husemann in: hacking,linux

average time to read 1:41 minutes

i recently switched all my systems from running kubuntu hardy heron 08.04.02 to xubuntu karmic koala 09.10¹ — a rather pleasant switch over. the only thing that was bothering me was the really crappy fonts in firefox and emacs. in firefox, no matter what i configured in firefox itself or in the appearance settings dialog of xubuntu (tried both gnome-control-center and XFCE’s appearance dialog) the fonts would still be blurry and below a certain size unreadable almost. in emacs i was suffering from the blurriness plus humongous menu fonts (i am getting older, i know, but so far i don’t need that large a print, really).

after a lot of research i finally managed to track it down to the following:

firefox doesn’t seem to be really interested in what the user’s desktop settings say, the only guy it will listen to is fontconfig configured via /etc/fonts/conf.d!
to get rid of the blurry fonts i needed to drop the 10-hinting-{full,medium,slight}.conf from fontconfig’s /etc/fonts.conf.d directory

that fixed the blurry fonts in firefox and emacs.

to get emacs menu’s back to a more sensible size i had to edit $HOME/.emacs.d/gtkrc — changing the default face via M-x customize-face only affected the buffer and modeline fonts but not the menu itself. i’m using the DejaVu Sans fonts, so my gktrc file looks like this:

style "user-font"
{
    font_name="DejaVu Sans 8"
}
widget_class "*" style "user-font"
gtk-font-name="DejaVu Sans 8"

style "default"
{
    font_name="DejaVu Sans 8"
    bg[NORMAL] = { 1.0, 1.0, 1.0 }
}

that one fixed emacs.

yes, i did try kubuntu 09.10 in the hopes that the KDE 4 gang had gotten their act together and had produced by now something as usuable (as opposed to blinky, flashy, sparkly) as KDE 3.5 — alas, that hope was in vain: a lot of the features that i’d valued in KDE 3.5 were still missing, usability was still not on par with KDE 3.5′s usability — but, hey, it’s blinky, flashy, sparkly, surely that counts for something? no, not really. i’m less interested in being an onlooker in the continuing GUI-blinky, flashy, sparkly pissing contest with windows/mac osx, and rather interested in a desktop that i can adapt to my needs without having to go on an archaeological expedition into the innards of KDE 4. sorry about that.

and, also, yes, i know about the KDE 3.5 repo for karmic — tried that, in fact, it mostly works, but with producing “cannot contact session manager” (no matter which session manager i tried) that was no longer usable either. really sorry. ↩

no comment so far

all content posted on these pages is an expression of my own mind. my employer is welcome to share these opinions but then again he might not want to.

japanese language support for OpenSim

filed mid-afternoon by Dr. Dirk Husemann in: from the grid,hacking,linux

average time to read 1:49 minutes

a colleague of mine in japan contacted us asking whether our in-world collaboration tools could be extended to support japanese characters — he had tried our internal, public sametime 3d servers and had noticed that while in-world chat and instant messaging worked just fine, our flipcharts, brain storming boards, calendar, and voting tools did just display rather boring little rectangles

not good. even worse when it means lost leads. so i set out to investigate. since OpenSim is being used in japan, it was a fair assumption that we might have a font problem here. this was confirmed by a twitter from adamfrisby:

@DrScofield so it should load fine.Only issue you need to be aware of is fonts – you need to use a font with the correct charset availible.

so, first port of call: how are we specifying the font to use for text rendering? digging through the OpenSim source i ended up in the VectorRenderModule:

    private void GDIDraw(string data, Graphics graph, char dataDelim)
    {
        Point startPoint = new Point(0, 0);
        Point endPoint = new Point(0, 0);
        Pen drawPen = new Pen(Color.Black, 7);
        string fontName = &quot;Arial&quot;;
        float fontSize = 14;
        Font myFont = new Font(fontName, fontSize);
        SolidBrush myBrush = new SolidBrush(Color.Black);
        ...

1

2

3

4

5

6

7

8

9

10

private void GDIDraw(string data, Graphics graph, char dataDelim)

{

Point startPoint = new Point(0, 0);

Point endPoint = new Point(0, 0);

Pen drawPen = new Pen(Color.Black, 7);

string fontName = "Arial";

float fontSize = 14;

Font myFont = new Font(fontName, fontSize);

SolidBrush myBrush = new SolidBrush(Color.Black);

...

gulp: we’ve got it hard coded

so, next question: can we change it dynamically? looking through the code, it seems like we might:

    ...
    else if (nextLine.StartsWith(&quot;FontName&quot;))
    {
        nextLine = nextLine.Remove(0, 8);
        fontName = nextLine.Trim();
        myFont = new Font(fontName, fontSize);
    }
    ...

1

2

3

4

5

6

7

8

...

else if (nextLine.StartsWith("FontName"))

{

nextLine = nextLine.Remove(0, 8);

fontName = nextLine.Trim();

myFont = new Font(fontName, fontSize);

}

...

except, hmph, there is no osSetFontName() OSSL function

a couple of SMOPs later, we had osSetFontName() and could also configure the default font name in OpenSim.ini, good. now to figure out which linux font would do the trick.

i tried:¹

Arial — no good: doesn’t have japanese script support
DejaVu Sans/Serif — again, no good, no japanese script support
Kochi Mincho — works!

— the configuration for that one is:

[VectorRender]
    font_name = "Kochi Mincho"

my next quest was to see whether i could find a font that would support the whole range: european scripts as well as japanese and also chinese.

…unfortunately, while windows users have Arial Unicode MS there is no such complete beast available to linux users — yes, there are fonts such as bitstream cybit, but their license conditions either don’t allow commercial use or only for a single user. so, if you are running under linux you need to localize your OpenSim installation via OpenSim.ini.

but! at least we figured out how to get japanese character sets supported in OpenSim on linux and can now configure one of our sametime 3d servers to support japanese OpenSim users properly!

lest you get the wrong impression: i know no japanese, instead i’m using google translate to produce suchs gems as “OpenSimは岩！”² ↩
and hoping that the good folks at google don’t do a “Έχω τρεις όρχεις” on me… ↩

no comment so far

all content posted on these pages is an expression of my own mind. my employer is welcome to share these opinions but then again he might not want to.

threading opensim through mono needles

filed mid-morning by Dr. Dirk Husemann in: from the grid,hacking,linux

average time to read 1:17 minutes

our Sametime3D test server is running a number of regions, some of them are rather on the monster side of things with gazillion scripts (ah, those brainstorming boards). until recently our opensim startup scripts would just grap a copy of mono and let it loose on OpenSim.exe and things would be working just fine.

sometime over the course of the last 10–14 days opensim must have crossed some (invisible) line drawn in the sandy thread beaches of monomania, as all of a sudden we noticed that three-avatar-meetings were working fine (chat & movement work) but as soon as we were joined by the fourth avatar, things came to a grinding halt: no movement, no chat (voice was still working but that’s to be expected as SL voice is pretty much running independent most of the time from what’s going on inside the region).

after some nose cone polishing, some more befuddled staring at the log files, and not finding anything immediately obvious, i went and discussed this problem with fellow opensimers on our intranet IRC opensim channel — sean had the good idea of mentioning the MONO_THREADS_PER_CPU environment variable. checking our startup script (just to make sure) showed that we were not setting it — and, thus, it was at its default setting of 50…

…so we went ahead and added it, setting it to 500. why 500? good question. and a question to which there is no clear answer: ask two opensimers and you’ll likely get three different recommendations, i’ve heard 75, 500, 1000, and also 2000. ralf haifisch recently tried to obtain clarity on this issue but so far the exact value for MONO_THREADS_PER_CPU remains a bit of magic.

and, guess what? that did the trick, it seems. we got over the three-avatar-limit.

comments (2)

all content posted on these pages is an expression of my own mind. my employer is welcome to share these opinions but then again he might not want to.

updated mono build script: mono 2.2

filed mid-afternoon by Dr. Dirk Husemann in: from the grid,linux

average time to read 0:11 minutes

here is the updated mono-build script for ubuntu hardy (intrepid might work as well).

update: since posting this i’ve been experiencing intermittent (that is, non-deterministic) mono crashes when running OpenSim, and have reverted to mono 2.0.1 which runs much more stable.

comments (13)

all content posted on these pages is an expression of my own mind. my employer is welcome to share these opinions but then again he might not want to.

the mystery of the blank opengl window on screen 2

filed in the early evening by Dr. Dirk Husemann in: hacking,linux,research

average time to read 0:28 minutes

the x200 i’m using has enough omph to power OpenGL on both the builtin display and the 1600×1200 external display. unfortunately whenever i moved that glxgears window on to the second screen it would just show…nothing. googling around for that showed that a number of people had the same problem but nobody had found a solution.

repeated staring at the intel(4) man pages resulted in the hunch to try and switch from EXA acceleration to XAA acceleration

Section "Device"
Identifier      "intel"
Driver          "intel"

Option          "monitor-VGA" "VGA"
Option          "monitor-LVDS" "LVDS"

Option          "AccelMethod" "XAA"
EndSection

an X server restart later, and, voilá!, glxgears shows its gears on screen 2. problem solved.

no comment so far

all content posted on these pages is an expression of my own mind. my employer is welcome to share these opinions but then again he might not want to.

latest tweets...

Blogroll