xyzzy xyzzy...

spamassassin: learning from inbox and junk folders

Dr. Dirk Husemann — Sat, 15 Sep 2012 06:28:35 +0000

while tweaking the spam filters on our mail server i finally took the step of adding a cron job to learn from the inbox and junk folders of each user. as we are using spamassassin as part of our spam defense this basically involves a couple of invocations of sa-learn to

learn the “ham” from each users inbox folder
learn the “spam” from each users junk folder

below is the shell script that gets invoked by cron once a day:

#!/bin/bash
echo "updating spamassassin bayesian spam/ham filter"
echo
for userDir in /home/*; do

    user=$(basename $userDir)
    ham=$userDir/Maildir/{cur,new}
    spam=$userDir/Maildir/.Junk/{cur,new}

    echo "    learning from $user"

    echo "       spam: $spam"
    /usr/bin/sa-learn --no-sync --spam $spam | while read line; do
	echo "            $line"
    done

    echo "        ham: $ham"
    /usr/bin/sa-learn --no-sync --ham $ham | while read line; do
	echo "            $line"
    done

    echo
done
echo "syncing:"
/usr/bin/sa-learn --sync | while read line; do
    echo "    $line"
done
echo
echo "stats:"
sa-learn --dump magic | while read line; do
    echo "    $line"
done

the while read line; do ... done bits are there so that i can nicely indent the output of sa-learn.

works rather nicely.

ajax fun with jquery & python

Dr. Dirk Husemann — Sun, 01 Jul 2012 14:47:18 +0000

the other day i tried to come up with a minimalistic Ajax setup — minimalistic in the sense of minimal coding overhead. my ingredients of choice were jquery and JavaScript on the client side and python on the server side. the goal was to use both GET and POST REST calls for obtaining the dynamic data and utilize JSON as the data format.

the python server code makes use of python’s BaseHTTPServer.HTTPServer and BaseHTTPServer.BaseHTTPRequestHandler classes:

AjaxServer extends BaseHTTPServer.HTTPServer and supplies Handler which in turn extends BaseHTTPServer.BaseHTTPRequestHandler. the Handler class also implements the GET and POST REST interfaces.

the server supports

text/html GETs on file names from the document root, retrieving index.html for GETs on directories, and
application/json GETs on
/ajax/time returning the current time
/ajax/processcount returning the number of currently running processes
application/json POSTs on any URL returning the length of the posted content

the client is a simple HTML5 page that uses jquery’s ajax() support to invoke the GET REST and POST REST calls on our python server.

client code





AJAX sandbox




 





OK, let's go!

current time: waiting for time update...

current process count: waiting for process count update...


waiting for result

the HTML5 code itself doesn’t really do anything. the magic happens in the ajax-datetime.js, ajax-processcount.js, ajax-processinput.js javascripts:

ajax-datetime.js calls the /ajax/time GET REST method every 3 seconds and sets the content of the #ajax-datetime div element to the return value of the GET REST call:

$(document).ready(function() { 

function updateDateTime() {
$.ajax({
cache: false,
url: "/ajax/time",
type: "GET",
contentType: "application/json",
}).done(function(json) {
$("#ajax-datetime").html(json.time);
});
}

setInterval(updateDateTime, (3 * 1000));
});

pretty much the same procedure is used for ajax-processcount.js:

$(document).ready(function() { 

function updateProcessCount() {
$.ajax({
cache: false,
url: "/ajax/processcount",
type: "GET",
contentType: "application/json",
}).done(function(json) {
$("#ajax-processcount").html(json.processcount);
}).error(function(json) {
$("#ajax-processcount").html("connection to server lost...");
});
}

setInterval(updateProcessCount, (5 * 1000));
});

the only difference is that ajax-processcount.js invokes the /ajax/processcount GET REST method and does so every 5 seconds — demonstrating the asynchronous nature of AJAX rather nicely.

while the previous two javascripts used GET to retrieve information from our python server, the next, ajax-processinput.js attaches itself to the #typehere text input field and POSTs the content of the field to /ajax/processinput, returning the result of the POST REST call via the #ajax-processedinput div element:

(document).ready(function() { 

function processInput() {
$.ajax({
cache: false,
url: "/ajax/processinput",
type: "POST",
contentType: "application/json",
dataType: "json",
data: JSON.stringify({ "input": $("#typehere").val() })
}).done(function(json) {
$("#ajax-processedinput").html(json.processed);
}).error(function(json) {
$("#ajax-processedinput").html("connection to server lost...");
});
}

$("#typehere").change(function () {
processInput();
});
});

unfortunately, even though i tell ajax(...) that i’m sending JSON — the contentType: "application/json" line — jquery will not convert the data i’m sending into JSON and i need to explicitly turn it into JSON myself:

...
data: JSON.stringify({ "input": $("#typehere").val() })
...

server code

the server code consists of two modules: ajaxserver and handler.

handler contains the Handler class which extends BaseHTTPServer.BaseHTTPRequestHandler and implements the GET and POST REST calls explained earlier:

#!/usr/bin/env python2.6
# -*- encoding: utf-8 -*- 

'''
handler.Handler is the HTTP handler for ajaxserver
'''

import BaseHTTPServer
import logging as log
import os
import subprocess

try:
import simplejson as json
except ImportError:
import json

from datetime import datetime

class Handler(BaseHTTPServer.BaseHTTPRequestHandler):

def logRequest(s):
log.debug("%s: %s/%s" %(s.client_address, s.command, s.path))

def do_GET(self):
self.logRequest()

path = self.path

if 'Content-type' in self.headers:
contentType = self.headers['Content-type']
else:
contentType = 'text/html'

log.debug("ajaxserver.GET: Path %s" % path)
log.debug("ajaxserver.GET: Content-type %s" % contentType)

if contentType.startswith('text/html'):
(respCode, respContent, respContentType) = self.do_GET_HTML()
elif contentType.startswith('application/json'):
(respCode, respContent, respContentType) = self.do_GET_JSON()

self.send_response(respCode, "thank you")
self.send_header('Content-type', respContentType)
self.send_header('Content-length', len(respContent))
self.end_headers()

self.wfile.write(respContent)

def do_POST(self):
self.logRequest()

path = self.path
contentType = self.headers['Content-type']

log.debug("ajaxserver.POST: Path %s" % path)
log.debug("ajaxserver.POST: Content-type %s" % contentType)

if not contentType.startswith('application/json'):
self.send_response(404, "oh, shiny!")
self.send_header('Content-type', 'application/json')
self.end_headers()
self.wfile.write(json.dumps({ "error": "only accepting application/json" }))
return

contentLength = int(self.headers['Content-length'])
content = self.rfile.read(contentLength)

log.debug("ajaxserver.POST: Content-length %d" % contentLength)
log.debug("ajaxserver.POST: Content: %s" % content)

data = json.loads(content)
if 'input' not in data:
log.debug("ajaxserver.POST: missing input parameter: %s" % content)
self.send_response(404, "oh, shiny!")
self.send_header('Content-type', 'application/json')
self.end_headers()
self.wfile.write(json.dumps({ "error": "missing input parameter" }))
return

self.send_response(200, "oh, shiny!")
self.send_header('Content-type', 'application/json')
self.end_headers()
self.wfile.write(json.dumps({ "processed": len(data['input'])}))

return

def do_GET_HTML(self):

docRoot = self.server.docRoot
docPath = "%s%s" % (docRoot, self.path)
errorPath = "%s/not-found.html" % docRoot

if self.path.startswith('../'):
log.debug('ajaxserver.GET(HTML): "%s" trying to escape from the sandbox' % self.path)
docFile = open(errorPath, 'r')
code = 404
elif docPath.endswith('/') and os.path.exists("%sindex.html" % docPath):
log.debug('ajaxserver.GET(HTML): serving %sindex.html for %s' % (docPath, self.path))
docFile = open("%sindex.html" % docPath, 'r')
code = 200
elif os.path.exists(docPath):
log.debug('ajaxserver.GET(HTML): serving %s for %s' % (docPath, self.path))
docFile = open(docPath, 'r')
code = 200
else:
log.debug('ajaxserver.GET(HTML): no document found for %s' % self.path)
docFile = open(errorPath, 'r')
code = 404

doc = docFile.read()
docFile.close()

return (code, doc, 'text/html')

def utilProcessCount(self):
psCmd = subprocess.Popen(["ps", "ax"], stdout=subprocess.PIPE)
wcCmd = subprocess.Popen(["wc", "-l"], stdin=psCmd.stdout, stdout=subprocess.PIPE)
count = wcCmd.communicate()[0]

return json.dumps({ 'processcount': int(count) - 1 })

def do_GET_JSON(self):

if self.path.startswith('/ajax/time'):
return (200, '{ "time": "%s" }' % datetime.utcnow().strftime('%Y-%m-%d %H:%M:%S UTC'), 'application/json')

if self.path.startswith('/ajax/processcount'):
return (200, self.utilProcessCount(), 'application/json')

return (200, '{ "resp": "bark!" }', 'application/json')

ajaxserver contains the AjaxServer class which extends the BaseHTTPServer.HTTPServer class and is the driver of my simple python web server. it also contains the setup code to instantiate the server and start it:

#!/usr/bin/env python2.6
# -*- encoding: utf-8 -*- 

'''
ajaxServer is a simple proof-of-concept web server for serving AJAX pages and their contents.
'''

import BaseHTTPServer
import logging as log
import logging.handlers
import os
import re
import subprocess
import sys
import urllib2

import handler

from datetime import date

class AjaxServer(BaseHTTPServer.HTTPServer):

def __init__(self, serverAddress, requestHandlerClass, docRoot):
BaseHTTPServer.HTTPServer.__init__(self, serverAddress, requestHandlerClass)
self.docRoot = docRoot

# from python module doc
def run(server_class=BaseHTTPServer.HTTPServer,
handler_class=BaseHTTPServer.BaseHTTPRequestHandler,
root='root',
ip='', port=8000):

server_address = (ip, port)

httpd = server_class(server_address, handler_class, root)
log.info("ajaxserver: waiting for requests")
httpd.serve_forever()

if __name__ == '__main__':

log.basicConfig(level = log.DEBUG,
format = '%(asctime)s %(levelname)-8s %(message)s',
datefmt = '%a, %d %b %Y %H:%M:%S')

ip = '127.0.0.1'
port = 8000
docRoot = '../root'

try:

# start the server
log.info("ajaxserver: starting HTTP server on %s:%d" %(ip, port))
run(server_class=AjaxServer, handler_class=handler.Handler, ip=ip, port=port, root=docRoot)
log.info("ajaxserver: terminating")

except KeyboardInterrupt, k:
print "\r<>"
log.info("aborted by user, terminating")

running it

start the ajax server:

> ./ajaxserver 
Sun, 01 Jul 2012 16:35:12 INFO     ajaxserver: starting HTTP server on 127.0.0.1:8000
Sun, 01 Jul 2012 16:35:12 INFO     ajaxserver: waiting for requests

then point your browser to http://localhost:8000/ — in your terminal running ajaxserver you’ll see a bunch of log lines showing you the requests triggered by loading the index.html page.

the browser will initially show just the static text of index.html —

— after a couple of seconds the current date and time will be displayed and continuously updated, followed by the current process count of your machine. when you enter text into the text input field and tab out, the length of your input will be shown next to it.

all three activities are happening independently from one another, the page itself is never reloaded (as you can verify in the ajaxserver log output).

emacs tip: getting lines numbered

Dr. Dirk Husemann — Thu, 19 Jan 2012 10:09:08 +0000

the other day i was coding a scala apply(array: Array[String}) method to instantiate an object from a CSV file. i ended up with something like this:

apply(uuid = uuid,
source = source,
hostname = rewired(0),
type = rewired(0),
state = rewired(0),
category = rewired(0),
id = rewired(0),
lifecycle = rewired(0),
classification = rewired(0),
...
flag = rewired(0))

there were about a 100 parameters to use. i wasn't really too excited about having to change the rewired(0) to use the proper index by hand. so, here's what i did:

in emacs mark the region
then invoke shell-command-on-region and
use perl -pi -e 's{rewired(0)}{sprintf("rewired(%d)", $. - 1)}e;' as command to invoke

emacs will then show you the result of the command in a temporary buffer, you can either copy and paste from there, or just repeat the shell-command-on-region and prefix it with ctrl-u --- emacs with then replace the region with the output of the command directly.

voila!

sbt10, webplugin, jetty, run-in-place: once more…

Dr. Dirk Husemann — Fri, 22 Jul 2011 14:55:03 +0000

it turns out that the sbt/web-plugin configuration i described in my last post is not quite cutting it. what we observed is that while changes to the contents of the webapp subtree were indeed effective immediately¹ sbt commands like prepare-webapp and jetty-reload would sometimes work and sometimes just ignore us.

so, back to digging around in the web-plugin sources and this is what we are now using:

[scala] temporaryWarPath <<= (sourceDirectory in Runtime) / "webapp", // watch temporaryWarPath / WEB-INF / classes jettyScanDirs <<= (temporaryWarPath) { (target) => Seq(target / "WEB-INF" / "classes") }, [/scala]

this sbt configuration tells jetty to run out of the src/main/webapp directory — any changes you do in there will become effective immediately and prepare-webapp and jetty-reload are working as expected. the drawback is that your generated classes and lib files get copied to src/main/webapp/WEB-INF i know, sucks, but still better than having to restart your webapp everytime you change a {html,css,js} file. if you are using git you might want to add

src/main/webapp/WEB-INF/lib/*
src/main/webapp/WEB-INF/classes/*

to your top-level .gitignore file.

on browser reload, that is. ↩

sbt10, webplugin: running jetty from source tree

Dr. Dirk Husemann — Mon, 04 Jul 2011 14:39:38 +0000

the solution described below turns out to have its flaws and is not really recommended — have a look at the recently posted update for a better solution!

simple build tool 0.7.5 comprised the jetty and webapp functionality — version 0.10.0 (sbt10) no longer does so, instead you have to pull in the sbt webplugin. one feature of sbt7.5′s webapp support that we used heavily was the ability to run jetty out of the source tree instead of the exploded temporary WAR file:

> Another possibility is to directly run the web application out of the the source web application path: > > override def jettyWebappPath = webappPath > override def scanDirectories = mainCompilePath :: testCompilePath :: Nil > > — sbt7.5, continuous redeployment

(we actually used to set scanDirectories to Nil)

achieving the same setup with sbt10 is not as simple or even obvious, the solution reported in the simple build tool newgroups does not really provide the same feature, as it still requires to have a ~ prepare-webapp running in sbt. here’s what seems to work:

// run jetty from source tree
jettyConfiguration <<= (sourceDirectory in Runtime, jettyConfiguration) map {
(sourceDir, jettyConf) => {
val conf = jettyConf.asInstanceOf[DefaultJettyConfiguration]
new DefaultJettyConfiguration {
def classpath = conf.classpath
def jettyClasspath = conf.jettyClasspath
def war = sourceDir / “webapp”
def contextPath = conf.contextPath
def classpathName = conf.classpathName
def parentLoader = conf.parentLoader
def scanDirectories = conf.scanDirectories
def scanInterval = conf.scanInterval
def port = conf.port
def log = conf.log
def jettyEnv = conf.jettyEnv
def webDefaultXml = conf.webDefaultXml
}
}
}

// set jetty scan dirs to empty list jettyScanDirs := Nil

changes in src/main/webapp are immediately effective for the running jetty instance. voila!

scala, emacs, sbt 0.10.0, ensime

Dr. Dirk Husemann — Tue, 28 Jun 2011 09:56:11 +0000

i’m an ardent emacs user. for our current project we chose the scala language as our programming language along with lift as the web application framework. initially we used maven but have switched over to the simple build tool — sbt in short — about half a year ago, as we found that to be faster than maven and also offering more features (such as test-only, test-quick, etc). to get scala support in emacs i’ve been using ensime which cooperated really well with sbt.

the sbt project recently made version 0.10.0 available — aka sbt10. switching from sbt7.5 to sbt10 is either very easy or a bit of a struggle. it’s easy if you are using a plain vanilla setup. it’s a struggle if you’ve created your own tasks — to paraphrase star trek: “it’s tasks, but not as we know it, jim”.

among the changes brought by sbt10 is that manged JARs are no longer copied into the lib_managed tree (instead the version in $HOME/.ivy2/cache is used, avoiding redundant copies) — that unfortunately confuses the heck out of ensime. florian hars, suffering from the same problem, wrote an sbt10 ensime plugin that fixes the issue for the time being (until ensime has proper sbt10 support).

to install the sbt ensime plugin, follow the instructions in its README, then in restart sbt and invoke the ensime task. after a compile of your project, it will generate a new .ensime project file (save an eventually existing old one if you care about it). then it’s just a restart of emacs and the invocation of ensime.

getting thunderbird to use yyyy-MM-dd HH:MM date format

Dr. Dirk Husemann — Wed, 13 Apr 2011 06:40:08 +0000

thunderbird is a terrific mail application (“mail user agent” is the posh name, i believe).

pretty much the only thing i don’t like about it though is its way of formatting the date and time and not providing a way of letting me configure the date format via preferences. it got bad enough that i recently spent a bit of time to figure out how get thunderbird to display date and time in yyyy-MM-dd HH:MM format — that is, 2011-04-13 20:34 instead of 04/13/2011 08:34pm or something similarly silly.

the solution was hiding in a forum post on the ubuntu forum: by switching the locale (well, at least parts of it) to en_DK.utf8 thunderbird would use the yyyy-MM-dd HH:MM format (aka ISO date–time format).

here’s how i did that on ubuntu:

[sourcecode language="bash"] # first: create a diversion of the normal /usr/bin/thunderbird dpkg-divert –divert /usr/bin/thunderbird.ubuntu –rename –local –add /usr/bin/thunderbird # next: create our locale setting wrapper cat </usr/bin/thunderbird #!/bin/bash export LC_TIME=en_DK.utf8 export LC_PAPER=en_DK.utf8 export LC_MEASUREMENT=en_DK.utf8 exec /usr/bin/thunderbird.ubuntu "$@" HERE chmod a+x /usr/bin/thunderbird [/sourcecode]

that should do the trick.

adding a USB monitor

Dr. Dirk Husemann — Wed, 09 Mar 2011 13:30:18 +0000

i’ve long been using an external 22″ monitor with my ubuntu linux powered X200 thinkpad. while ubuntu maverick (10.10) has some issues with attaching and detaching the second monitor and subsequent suspend–resume cycles (the second suspend after a detach would not resume, d’oh), ubuntu lucid (10.04.02) works just fine (as befitting a long-term-support release).

for quite a while i’ve had the old 20″ monitor which i had been using previously sitting on my desktop along with an USB DisplayLink adapter — the idea being to hook the old monitor up as well as a third monitor¹. the displaylink driver provided by ubuntu lucid seemed to work properly — the monitor’s screen would turn green on connecting it and a framebuffer device (/dev/fb1) and so i tried various recipes floating around on ubuntu forums and elsewhere — all promising to achieve a grand unified desktop comprising all three monitors — and they either didn’t work or if they achieved the grand unified desktop it was unusable.

so, after another prolonged period during which the monitor and the displaylink adapter gathered even more dust, i tried a different approach: give up on the grand unified desktop goal and instead just try to make use of the monitor. the idea this time was to start up a VNC server, then use a VNC client to directly render the server into the framebuffer device (/dev/fb1) provided by linux’s displaylink driver. the vncserver bit is actually quite easy:

[sourcecode language="bash"] vncserver -name hidden -geometry 1600×1200 -depth 16 :42 [/sourcecode]

which starts a VNC server for the :42 display.

the VNC client bit turned out to be a bit more difficult. ubuntu lucid does have directvnc client which is “a vnc client for the linux framebuffer device”². that client does seem to require keyboard and mouse access and in some configurations did not work at all or locked up my keyboard (not the mouse, though, funnily enough) or crashed the running X session, so no points on that one. further research luckily turned up vnc2dl by none other than quentin stafford-fraser one of the original VNC developers (and also the inventor of the webcam it seems). vnc2dl seemed a bit more promising and — after slightly modifying³ dldevice.c — did do the job:

[sourcecode language="bash"] sudo vnc2dl :42 [/sourcecode]

next up was fusing display :42 to my main display :0 on a keyboard and mouse level so that i could just move the mouse pointer over to the left and end up on display :42 — here x2x (in the equally named ubuntu package) came into play:

[sourcecode language="bash"] x2x -west -to :42 >/dev/null 2>&1 & [/sourcecode]

and, hey, presto!, both displays are linked mouse and keyboard wise.

only thing still bothering me was that cut and paste was not working. to fix that required adding

[sourcecode language="bash"] vncconfig -nowin & [/sourcecode]

to my .vnc/xstartup file.

to have firefox run on display :42 required creation of a new firefox profile — i added that to .vnc/xstartup as well.⁴

yep, even more screen real estate; can’t have enough of that: the ur-IDE emacs in one screen, instant messaging app pidgin on the other (for communicating with the team), the third screen would be really useful to host a firefox window with the API docs and so forth. ↩
see man directvnc for more information about that one. ↩
vnc2dl in the version on quentin’s github is hard-wired to 1280×1040/24bpp which my old monitor doesn’t quite grok, changing the wiring to 1600×1200/16bpp made it more grokkable for my setup. ↩
to get the new profile synchronized with the default profile i used firefox’s recently added Firefox Sync add-on. ↩

protecting an emacs buffer window from being split or replaced…

Dr. Dirk Husemann — Thu, 03 Mar 2011 10:00:04 +0000

i’m a big fan of IRC and we use it in our distributed team to stay in touch and also coordinate testing and upgrades. as i’m also a big fan of the ur-IDE emacs i’m using emacs’s rcirc mode for my IRC needs. typically i have a small buffer window at the bottom of my emacs frame dedicated to IRC. quite nice…

…until you start sbt or do some git work in egg as then those modes try to acquire some screen real-estate and without scruples either “recycle” my IRC buffer for their content or split it into two

this being emacs there had to be a way to get that IRC buffer window protected. and there is digging into the emacs info docs and looking at the underlying emacs lisp files this is what i came up with:

 [sourcecode language="text"]
 (defun rcirc-toggle-current-window-dedication ()
   (interactive)
   (let* ((window (selected-window))
          (dedicated (window-dedicated-p window)))
     (set-window-dedicated-p window (not dedicated))
     (setq window-size-fixed (not window-size-fixed))
     (message "Window %sdedicated to %s"
              (if dedicated "no longer " "")
              (buffer-name))))

 (global-set-key (kbd "<Scroll_Lock>") 'rcirc-toggle-current-window-dedication)
 [/sourcecode]

this will toggle the window-dedicated-p and window-size-fixed bits of the window in which you press the scroll-lock key — and, hey, presto! — no more pirating of the IRC buffer window.

CSS selector transforms…

Dr. Dirk Husemann — Fri, 18 Feb 2011 16:05:46 +0000

we have been using scala and the lift framework for our project for over a year now. as we are starting on a new subcomponent of it, i thought i’d take a crack at using lift’s new CSS selector transforms.

CSS transforms provide an alternative to the traditional lift templating mechanism:

[sourcecode language="xml"] [/sourcecode]

which requires the following scala code:

[sourcecode language="scala"] class FooBar { def render(in: NodeSeq): NodeSeq = bind("foobar", in, "dosomething" -> dog) } [/sourcecode]

and would result in

[sourcecode language="xml"] dog [/sourcecode]

with the new CSS transforms we get rid of the tags and instead use plain HTML:

[sourcecode language="xml"]

XXX

[/sourcecode]

and then change the binding to:

[sourcecode language="scala"] class FooBar { def render = "#animal *" "dog" } [/sourcecode]

and end up, again, with

[sourcecode language="xml"] dog [/sourcecode]

— just with much less code! and once you start looking into expanding lists, it really becomes obvious that CSS transforms are a huge step forward. add to that that you can now design your web apps using normal HTML design tools (which in my case is emacs, nxhtml mode, and moz-repl) and life has just become a lot easier and more productive…

…except for one thing you see, we are using comets and lift’s rather cool partialUpdate(SetHtml("dog", Text(wuff! wuff!))) construct to dynamically update the “dog” span. for that we were — in the pre-CSS transform age — using calls to chooseTemplate to fetch XML snippets and re-render them. how to do this with CSS transforms?

let’s assume our HTML snippet is a bit more complex and includes an image:

[sourcecode language="xml"]

[/sourcecode]

the CSS selector transforms page lists the following construct:

[sourcecode language="scala"] "animal ^^" #> "ignore" [/sourcecode]

this will select the element with the id “animal”. so far, so good. what’s not so good is that the intuitively next idea

[sourcecode language="scala"] "animal ^^" #> "ignore" & "img [src]" #> "dog.png" [/sourcecode]

doesn’t work (CSS transforms are actually functions, so we can invoke them on NodeSeq in sbt’s console):

scala> val in = 
in: scala.xml.Elem = 

scala> ("#animal ^^" #> "ignore")(in)
res0: scala.xml.NodeSeq = NodeSeq()

scala> ("#animal ^^" #> "ignore" & "img [src]" #> "dog.png")(in) 
res1: scala.xml.NodeSeq = NodeSeq()

what does work is the following code:

[sourcecode language="scala"] val template = ("#animal ^^" #> "ignore")(in) ("img [src]" #> "dog.png")(template) [/sourcecode]

if we try this in console we get:

scala> val template = ("#animal ^^" #> "ignore")(in)
template: scala.xml.NodeSeq = NodeSeq()

scala> ("img [src]" #> "dog.png")(template)
res2: scala.xml.NodeSeq = NodeSeq()

which is what we expect. just a bit cumbersome to write and the contraction

[sourcecode language="scala"] ("img [src]" #> "dog.png")(("#animal ^^" #> "ignore")(in)) [/sourcecode]

while producing the same result, is a bit hard to understand.

ideally, we could write it as:

[sourcecode language="scala"] "#animal ^^" #> "ignore" ~> "img [src]" #> "dog.png" [/sourcecode]

as in: first select the template, then apply the following transform. well, we are in scala land, where (almost) everything is possible! so, why not create that operator? how hard can it be?

“not very” is the answer. first step is to figure out what it is that we want: CssBindFunc can be viewed as (NodeSeq) => NodeSeq functions. so, we really want to concatenate one (NodeSeq) => NodeSeq with another one. writing this as a function that translates into:

[sourcecode language="scala"] def andThenInto(first: (NodeSeq) => NodeSeq, second: (NodeSeq) => NodeSeq): (NodeSeq) => NodeSeq = (ns: NodeSeq) => second(first(ns)) [/sourcecode]

throwing that into the sbt console yields:

scala> def andThenInto(first: (NodeSeq) => NodeSeq, second: (NodeSeq) => NodeSeq): (NodeSeq) => NodeSeq =
    (ns: NodeSeq) => second(first(ns))
andThenInto: (first: (scala.xml.NodeSeq) => scala.xml.NodeSeq,second: (scala.xml.NodeSeq) => scala.xml.NodeSeq)(scala.xml.NodeSeq) => scala.xml.NodeSeq

scala> andThenInto("#animal ^^" #> "ignore", "img [src]" #> "dog.png")
res4: (scala.xml.NodeSeq) => scala.xml.NodeSeq = 

scala> res4(in)
res4(in)
res5: scala.xml.NodeSeq = NodeSeq()

scala>

which is exactly what we are after — well, almost. andThenInto is a bit long, we wanted to use ~>. so, let’s try that:

def ~>(first: (NodeSeq) => NodeSeq, second: (NodeSeq) => NodeSeq): (NodeSeq) => NodeSeq =
    (ns: NodeSeq) => second(first(ns))
$tilde$greater: (first: (scala.xml.NodeSeq) => scala.xml.NodeSeq,second: (scala.xml.NodeSeq) => scala.xml.NodeSeq)(scala.xml.NodeSeq) => scala.xml.NodeSeq

doesn’t look too bad. let’s give it a spin:

~>("#animal ^^" #> "ignore", "img [src]" #> "dog.png")
res6: (scala.xml.NodeSeq) => scala.xml.NodeSeq = 

scala> res6(in)
res6(in)
res7: scala.xml.NodeSeq = NodeSeq()

looking good. next, let’s pimp CssBindFunc with that:

[sourcecode language="scala"] implicit def pimpCssBindFuncWithAndThenInto(first: (NodeSeq) => NodeSeq) = new { def ~>(second: (NodeSeq) => NodeSeq): (NodeSeq) => NodeSeq = (ns: NodeSeq) => second(first(ns)) } [/sourcecode]

and give it a go:

scala> "#animal ^^" #> "ignore" ~> "img [src]" #> "dog.png"
:14: error: type mismatch;
 found   : java.lang.String("img [src]")
 required: (scala.xml.NodeSeq) => scala.xml.NodeSeq
       "#animal ^^" #> "ignore" ~> "img [src]" #> "dog.png"

ooops. what happened? the compiler seems to be doing the grouping in a different way then we expected. “operator precedence” was jumping up and down in my mind in the back row desparate to get called on: looking up that topic in the “scala bible” turned up the explanation: the ~ character is in the highest precedence category together with #. to avoid having to use parenthesis we changed our “andThenInto” operator to &~>:

[sourcecode language="scala"] implicit def pimpCssBindFuncWithAndThenInto(first: (NodeSeq) => NodeSeq) = new { def &~>(second: (NodeSeq) => NodeSeq): (NodeSeq) => NodeSeq = (ns: NodeSeq) => second(first(ns)) } [/sourcecode]

which then yields the desired result:

"#animal ^^" #> "ignore" &~> "img [src]" #> "dog.png"
res10: (scala.xml.NodeSeq) => scala.xml.NodeSeq = 

scala> res10(in)
res10(in)
res11: scala.xml.NodeSeq = NodeSeq()

scala>