for over a year we’ve been meeting more or less once a month or two over lunch to discuss and chat about all things OpenSim — very creatively volker gässler and i called them OpenSim Zurich Lunches back then when. for the past year we’ve been having lunch at the cafeteria of the IBM Research — Zurich lab, this month’s OpenSim Zurich Lunch will be the first time we meet closer to the downtown zurich area:

• monday, january 25th, 12:30h — 14:00h (approx)
• king’s kurry, freyastrasse 3, zürich (close to the train station wiedikon)

so, if you are around, have time to spare, and are interested to meet up with fellow OpenSim-ers, let volker know by dropping him an email — fe11 at gmx.ch — so that we can reserve enough seats at king’s kurry.

the latest rezzme now contains the necessary code to build debian/ubuntu dpkgs — making it as easy to install rezzme packages on the latest debian/ubuntu systems as it already is for windows and mac OSX machines as the dpkg format allows me to explicitly specify the prerequisites (such as PyQt4).

while working on the dpkg support i also found out that debian/ubuntu has the very useful /etc/firefox-3.0, /etc/firefox-3.5, and /etc/thunderbird directories that allow me to provide package specific javascript to, for example, add a protocol handler for the rezzme: URI scheme. that along with gnome’s gconf tool gives us the same seamless rezzme user experience as on windows or mac OSX — unfortunately, it seems that nothing comparable exists for redhat/fedora based systems.¹

also the windows build code has been cleaned up and the no longer required dependency on setuptools removed.

a colleague of mine in japan contacted us asking whether our in-world collaboration tools could be extended to support japanese characters — he had tried our internal, public sametime 3d servers and had noticed that while in-world chat and instant messaging worked just fine, our flipcharts, brain storming boards, calendar, and voting tools did just display rather boring little rectangles

not good. even worse when it means lost leads. so i set out to investigate. since OpenSim is being used in japan, it was a fair assumption that we might have a font problem here. this was confirmed by a twitter from adamfrisby:

@DrScofield so it should load fine.Only issue you need to be aware of is fonts – you need to use a font with the correct charset availible.

so, first port of call: how are we specifying the font to use for text rendering? digging through the OpenSim source i ended up in the VectorRenderModule:

    private void GDIDraw(string data, Graphics graph, char dataDelim)
    {
        Point startPoint = new Point(0, 0);
        Point endPoint = new Point(0, 0);
        Pen drawPen = new Pen(Color.Black, 7);
        string fontName = "Arial";
        float fontSize = 14;
        Font myFont = new Font(fontName, fontSize);
        SolidBrush myBrush = new SolidBrush(Color.Black);
        ...

gulp: we’ve got it hard coded

so, next question: can we change it dynamically? looking through the code, it seems like we might:

    ...
    else if (nextLine.StartsWith("FontName"))
    {
        nextLine = nextLine.Remove(0, 8);
        fontName = nextLine.Trim();
        myFont = new Font(fontName, fontSize);
    }
    ...

except, hmph, there is no osSetFontName() OSSL function

a couple of SMOPs later, we had osSetFontName() and could also configure the default font name in OpenSim.ini, good. now to figure out which linux font would do the trick.

i tried:¹

• Arial — no good: doesn’t have japanese script support
• DejaVu Sans/Serif — again, no good, no japanese script support
• Kochi Mincho — works!

— the configuration for that one is:

[VectorRender]
    font_name = "Kochi Mincho"

my next quest was to see whether i could find a font that would support the whole range: european scripts as well as japanese and also chinese.

…unfortunately, while windows users have Arial Unicode MS there is no such complete beast available to linux users — yes, there are fonts such as bitstream cybit, but their license conditions either don’t allow commercial use or only for a single user. so, if you are running under linux you need to localize your OpenSim installation via OpenSim.ini.

but! at least we figured out how to get japanese character sets supported in OpenSim on linux and can now configure one of our sametime 3d servers to support japanese OpenSim users properly!

a couple of weeks ago i blogged about our rather sobering tests with real avatars and that we run into serious trouble round about the 21 avatar mark. alan webb, arthur valadares, and i started looking at the what could possibly be biting us there.

having profiled opensim (albeit briefly, you end up with lots of data), arthur had found out that a lot of time was spend in the packet handling code, so we concentrated on the packet and textures handling — hypothesis #2 being that the on-rush of avatars trying to log in all at ance (and being logged in almost all at once) was causing a massive spike in textures demand: alan “outsourced” the texture handling to a thread of its own instead of having it part of the normal packet handling. arthur and i were successful in getting rid of a number of short-lived mass object instantiations.

the result: well, with even more vicious bots (20 of those new ones were sufficient to bring opensim to a stand-still now, similar to real avatars) we now managed to go up to 40 avatars! sure the region became laggy, but, and that is important, we could still use the in-world tools, could still move around, chat and work.

so, progress. now to figure out why massive amounts of collision events reported via LSL cause a deep ODE physics engine crash…

this morning i finally could push our VivoxVoiceModule into opensim trunk. we had been using it internally for almost three month as part of the Sametime3D project but only now got the necessary legal “paperwork” done so that we could finally release it into the wild!

VivoxVoiceModule works with the same voice technology which Linden Labs are using for their SecondLife virtual world — and, what’s even nicer, it plays along rather well with recent SecondLife clients (anything from 1.22.9 onwards), meaning you get full spatial sound, speaker indication, direct avatar-to-avatar voice chats, the works, without having to configure anything  client side.  you can even — in contrast to Linden Labs SecondLife grid — switch from spatial voice to conference call style voice channels, or change the range of voice, how quickly it fades, and so forth (all explained in OpenSim.ini.example). voice quality is superb — except for those occasions where we muck around with the grid itself (and thus cannot use in-world voice), we use it instead of conference calls and have been using it for a couple of months now.

VivoxVoiceModule does require a vivox voice server to talk to — which means that you need to obtain a customer admin account (IIRC) from vivox. as i don’t work for vivox (alan webb and i just wrote the module, nothing else), i can’t provide any further information on how to go about obtaining such an account nor can i tell you how it will cost. there are rumours on the opensim-dev mailing list that adam frisby has a bunch of licenses from vivox for OSgrid, again, i know no details.

after our intoxicating stress test ten days ago we repeated the stress test today with a group of volunteers — thus, real SL clients instead of bots.

the result: 21.

somewhere around the 21 avatar mark we are got stuck and the (single) region standalone system started becoming very laggy: avatars could not move anymore (or erratically at best), chat got relayed occasionally. interestingly enough we still saw new users logging in. looking at OpenSim’s log we saw that OpenSim was busy processing new user requests — a lot of new user requests.

so, the current hypothesis is that new users clog the out-pipe with large amount of texture requests, causing the existing users to starve and the system becoming unresponsive.

for stress testing i needed to create 100 avatar accounts quickly. instead of going via opensim’s console i started ipython and entered the following python script:

import xmlrpclib
os = xmlrpclib.Server('http://127.0.0.1:9000/')
for i in range(100):
    os.admin_create_user(dict(password = 'SECRET', 
                              user_firstname = 'Bot%d' % i,
                              user_lastname = 'Dude', 
                              user_password = 'b0tb0tb0t', 
                              start_region_x = 1000, 
                              start_region_y = 1000, 
                              model = 'Bot Dude')) 

a couple of minutes later i had successfully created 100 avatar accounts, ”Bot0 Dude” to ”Bot99′, all wearing the same outfit as ”Bot Dude”, an avatar i had prepared earlier.

voila! avatar machine

it was a dark and stormy night. lightning was ripping the heavens apart, thunder was rolling through the valley. no living soul dared venture outside…¹

…and i wasn’t. i sat in front of my screen, mesmerized by the numbers coming up: 20…27…34…38…40…43…48…52… eventually peaking at 81! we were stress testing our opensim server and our in-world build. the numbers represented rather brutal bots, that, as soon as they were in-world started running around, chatting with one another and grabbing all the textures they could find — we’d lovingly nicknamed them “hooligan bots” as they’d successfully managed to demolish a number of our opensim test instances and reduced them to nothing more than a pile of broken bits and bytes.

after fixing another memory gobbling piece of code and some tetchy ODE physics engine part, we’d gone past the 20 mark that had meant the end of our opensim instance in the past couple of trials and the numbers kept on climbing. during the test we had to create more and more test accounts to be able to keep sending in the bots. the system behaved extraordinarily stable: not only were we able to keep moving around and not only did opensim keep chugging along but also our scripts (themselves not exactly shy when it comes to CPU cycles and memory) did!

our test system, a 4 core intel system, running in 32-bit PAE mode with about 8GB of memory, coped rather well on the memory front: with about 5 regions (script heavy) we clocked at about 1.3 — 1.5 GB memory usage — CPU load, however, was a different story: that went up like crazy: up to 390% CPU utilization, load level of about 9–10. still, the only effect was that things became a bit sluggish.²

note, this was a one off! a pleasant and encouraging one off but nevertheless so far just a one off. we need to carry out further testing to corroborate this.

still, 81 avatars in one region!

recently i was in need of being able to send inventory items (well, a whole folder full of stuff) to the inventory of a logged-in avatar.

just copying the the inventory items to the avatar inventory (for example, using Scene.GiveInventoryItem() or Scene.GiveInventoryFolder()) will not cause your avatar to notice that there is additional junk in her inventory all of a sudden — once the avatar is logged out and logs in again, she will find the stuff in her inventory. kicking the avatar out every time you want to send some inventory her way is a tad bad on the user experience side¹. clearly not something we want to do.

after trying a 101 ways of sending inventory to the avatar and none of them catching, i finally remembered that avatars can give inventory to one another, even whole folders!² taking my favorite tool to the OpenSim source tree³ i found code in the InventoryTransferModule that seemed to take care of this process. the first attempt of just sending a GridInstantMessage via the usual means was not producing the right result: the avatar would get the pop-up that a folder was being given to her and would she like to keep it? looked good but checking the inventory revealed it to be a big blue lie.

here’s what i finally figured out. you need:

• a ScenePresence object representing the avatar
• the UUID of the sending “avatar”
• the name of the sending “avatar”
• the InventoryNodeBase object (actually you would pass in a subclass such as InventoryItemBase or InventoryFolderBase or similar)

here’s the “pseudo-code”:

// pseudo code: send inventory content to a logged in avatar's
// inventory 
//
// avatar: ScenePresence object
// src: struct containing "UUID" and "Name" of the source "avatar"

// get the CachedUserInfo object for the target avatar
CachedUserInfo cuiAvatar = 
    m_commsManager.UserProfileCacheService.GetUserDetails(avatar,UUID);
if (!cuiAvatar.HasReceivedInventory) cuiAvatar.FetchInventory();

// target folder is the Clothing folder (type 5)
InventoryFolderImpl dstFolder = cuiAvatar.FindFolderForType(5);

// copy source folder from src avatar to dstFolder
copyFolder = scene.GiveInventoryFolder(avatar.UUID, src.UUID, 
                                       dstFolder.ID);

// content is in the target folder, ping target avatar and let her
// know about it: we do this through a GridInstantMessage

// GridInstantMessage: prepare bucket byte array first byte is asset
// type, remaining bytes are the UUID of the InventoryItem
byte[] idBytes = item.ID.GetBytes();
byte[] bucket = new byte[1 + idBytes.Length];

if (item is InventoryFolderBase)
    bucket[0] = (byte)AssetType.Folder;
else if (item is InventoryItemBase)
    bucket[0] = (byte)(item as e).AssetType;

Array.Copy(idBytes, 0, bucket, 1, idBytes.Length);

GridInstantMessage im = 
    new GridInstantMessage(scene, src.UUID, src.Name, avatar.UUID,  
                           (byte)InstantMessageDialog.InventoryOffered,
                           false, item.Name, item.ID,
                           Vector3.Zero, bucket);

// send bulk update inventory message and GridInstantMessage
avatar.ControllingClient.SendBulkUpdateInventory(item);
avatar.ControllingClient.SendInstantMessage(im);
[/c-sharp]

a couple of weeks ago i added the regioninfo REST call to OpenSim. here is a little python script that shows you how to use it:

#!/usr/bin/python
# -*- encoding: utf-8 -*-

import sys
import urllib2
import xml.etree.ElementTree as ET

def RegionInfo(url):
    if not url.endswith('/'):
        url = '%s/' % url
    regionInfoUri = '%sadmin/regioninfo/' % url
    print '== OpenSim server %s ==' % url

    riXml = ET.parse(urllib2.urlopen(regionInfoUri)).getroot()
    print 'regions (curr): %d' % int(riXml.attrib['number'])
    print 'regions (max):  %d' % int(riXml.attrib['max'])

    totalAvatars = 0
    totalObjects = 0

    for region in riXml:
        print '- region %-20s: avatars: %d' % (region.attrib['name'], int(region.attrib['avatars']))
        print '         %-20s  objects: %d' % (' ', int(region.attrib['objects']))

        totalAvatars += int(region.attrib['avatars'])
        totalObjects += int(region.attrib['objects'])

    print 'total avatars: %d' % totalAvatars
    print 'total objects: %d\n' % totalObjects


if __name__ == '__main__':
    for opensim in sys.argv[1:]:
        RegionInfo(opensim)

invoke it like this:

% osinfo http://opensim.zurich.ibm.com:9000

and you will get output similar to this:

== OpenSim server http://opensim.zurich.ibm.com:9000/ ==
regions (curr): 6
regions (max):  10
- region ST3D theatre        : avatars: 0
                               objects: 494
- region ST3D collaboration  : avatars: 0
                               objects: 283
- region ST3D boardroom      : avatars: 0
                               objects: 243
- region Zurich ISL          : avatars: 0
                               objects: 0
- region IBM ACVWS 2009      : avatars: 0
                               objects: 283
- region Sametime 3D         : avatars: 0
                               objects: 16
total avatars: 0
total objects: 1319

quite useful.