after our intoxicating stress test ten days ago we repeated the stress test today with a group of volunteers — thus, real SL clients instead of bots.

the result: 21.

somewhere around the 21 avatar mark we are got stuck and the (single) region standalone system started becoming very laggy: avatars could not move anymore (or erratically at best), chat got relayed occasionally. interestingly enough we still saw new users logging in. looking at OpenSim’s log we saw that OpenSim was busy processing new user requests — a lot of new user requests.

so, the current hypothesis is that new users clog the out-pipe with large amount of texture requests, causing the existing users to starve and the system becoming unresponsive.

for stress testing i needed to create 100 avatar accounts quickly. instead of going via opensim’s console i started ipython and entered the following python script:

import xmlrpclib
os = xmlrpclib.Server('http://127.0.0.1:9000/')
for i in range(100):
    os.admin_create_user(dict(password = 'SECRET', 
                              user_firstname = 'Bot%d' % i,
                              user_lastname = 'Dude', 
                              user_password = 'b0tb0tb0t', 
                              start_region_x = 1000, 
                              start_region_y = 1000, 
                              model = 'Bot Dude')) 

a couple of minutes later i had successfully created 100 avatar accounts, ”Bot0 Dude” to ”Bot99′, all wearing the same outfit as ”Bot Dude”, an avatar i had prepared earlier.

voila! avatar machine

it was a dark and stormy night. lightning was ripping the heavens apart, thunder was rolling through the valley. no living soul dared venture outside…¹

…and i wasn’t. i sat in front of my screen, mesmerized by the numbers coming up: 20…27…34…38…40…43…48…52… eventually peaking at 81! we were stress testing our opensim server and our in-world build. the numbers represented rather brutal bots, that, as soon as they were in-world started running around, chatting with one another and grabbing all the textures they could find — we’d lovingly nicknamed them “hooligan bots” as they’d successfully managed to demolish a number of our opensim test instances and reduced them to nothing more than a pile of broken bits and bytes.

after fixing another memory gobbling piece of code and some tetchy ODE physics engine part, we’d gone past the 20 mark that had meant the end of our opensim instance in the past couple of trials and the numbers kept on climbing. during the test we had to create more and more test accounts to be able to keep sending in the bots. the system behaved extraordinarily stable: not only were we able to keep moving around and not only did opensim keep chugging along but also our scripts (themselves not exactly shy when it comes to CPU cycles and memory) did!

our test system, a 4 core intel system, running in 32-bit PAE mode with about 8GB of memory, coped rather well on the memory front: with about 5 regions (script heavy) we clocked at about 1.3 — 1.5 GB memory usage — CPU load, however, was a different story: that went up like crazy: up to 390% CPU utilization, load level of about 9–10. still, the only effect was that things became a bit sluggish.²

note, this was a one off! a pleasant and encouraging one off but nevertheless so far just a one off. we need to carry out further testing to corroborate this.

still, 81 avatars in one region!

i’ve been a big fan of the syntax highlighter aka syntaxhighligher evolved wordpress plugin for quite a while now. after updating to the syntaxhighlighter evolved version, my code sections started looking funny, like this:

a = None
b = 1


if all((a, b)):
    print 'yes!'

instead of like this:

a = None
b = 1

if all((a, b)):
    print 'yes!'

the problem seemed to be that syntaxhighlighter’s shortcodes confuddled markdown (or markdown confuddled syntaxhighlighter?). this morning i realized that the solution might be actually quite simple: instead of not indenting, indent the code by 4 space (as you usually do with markdown to tag code sections, sigh) and all might be well…

…and it was. sigh. quite easy.

recently i was in need of being able to send inventory items (well, a whole folder full of stuff) to the inventory of a logged-in avatar.

just copying the the inventory items to the avatar inventory (for example, using Scene.GiveInventoryItem() or Scene.GiveInventoryFolder()) will not cause your avatar to notice that there is additional junk in her inventory all of a sudden — once the avatar is logged out and logs in again, she will find the stuff in her inventory. kicking the avatar out every time you want to send some inventory her way is a tad bad on the user experience side¹. clearly not something we want to do.

after trying a 101 ways of sending inventory to the avatar and none of them catching, i finally remembered that avatars can give inventory to one another, even whole folders!² taking my favorite tool to the OpenSim source tree³ i found code in the InventoryTransferModule that seemed to take care of this process. the first attempt of just sending a GridInstantMessage via the usual means was not producing the right result: the avatar would get the pop-up that a folder was being given to her and would she like to keep it? looked good but checking the inventory revealed it to be a big blue lie.

here’s what i finally figured out. you need:

• a ScenePresence object representing the avatar
• the UUID of the sending “avatar”
• the name of the sending “avatar”
• the InventoryNodeBase object (actually you would pass in a subclass such as InventoryItemBase or InventoryFolderBase or similar)

here’s the “pseudo-code”:

// pseudo code: send inventory content to a logged in avatar's
// inventory 
//
// avatar: ScenePresence object
// src: struct containing "UUID" and "Name" of the source "avatar"

// get the CachedUserInfo object for the target avatar
CachedUserInfo cuiAvatar = 
    m_commsManager.UserProfileCacheService.GetUserDetails(avatar,UUID);
if (!cuiAvatar.HasReceivedInventory) cuiAvatar.FetchInventory();

// target folder is the Clothing folder (type 5)
InventoryFolderImpl dstFolder = cuiAvatar.FindFolderForType(5);

// copy source folder from src avatar to dstFolder
copyFolder = scene.GiveInventoryFolder(avatar.UUID, src.UUID, 
                                       dstFolder.ID);

// content is in the target folder, ping target avatar and let her
// know about it: we do this through a GridInstantMessage

// GridInstantMessage: prepare bucket byte array first byte is asset
// type, remaining bytes are the UUID of the InventoryItem
byte[] idBytes = item.ID.GetBytes();
byte[] bucket = new byte[1 + idBytes.Length];

if (item is InventoryFolderBase)
    bucket[0] = (byte)AssetType.Folder;
else if (item is InventoryItemBase)
    bucket[0] = (byte)(item as e).AssetType;

Array.Copy(idBytes, 0, bucket, 1, idBytes.Length);

GridInstantMessage im = 
    new GridInstantMessage(scene, src.UUID, src.Name, avatar.UUID,  
                           (byte)InstantMessageDialog.InventoryOffered,
                           false, item.Name, item.ID,
                           Vector3.Zero, bucket);

// send bulk update inventory message and GridInstantMessage
avatar.ControllingClient.SendBulkUpdateInventory(item);
avatar.ControllingClient.SendInstantMessage(im);

while researching the german word “wallen” i bought the universalwörterbuch from duden, a respected and well-known dictionary publisher in german speaking countries. the download page offers a debian–ubuntu package of officebib, the underlying application for a series of dictionaries from duden and other publishers. running kubuntu i naturally chose that one to install — and it worked…

…it worked but in contrast to all other applications was using pixel fonts instead of truetype fonts, rendering all text rather ugly. today i did a bit of searching around whether other folks had encountered this issue and had perhaps found a way around it. eventually i came across a post on the {} blog (sic) which recommended not to use the .deb package but instead to one of the .rpm packages:

% sudo apt-get install fakeroot alien

will install the pre-requisites. then download the SuSE .rpm package and turn it into a debian package like so:

% fakeroot alien officebib-5.0.4-1-suse.rpm

and install it:

% dpkg -i officebib_5.0.4-2_i386.deb

voila! no more ugly fonts.

now, why doesn’t duden do this instead of providing a flawed .deb package? puzzled.

a couple of weeks ago i added the regioninfo REST call to OpenSim. here is a little python script that shows you how to use it:

#!/usr/bin/python
# -*- encoding: utf-8 -*-

import sys
import urllib2
import xml.etree.ElementTree as ET

def RegionInfo(url):
    if not url.endswith('/'):
        url = '%s/' % url
    regionInfoUri = '%sadmin/regioninfo/' % url
    print '== OpenSim server %s ==' % url

    riXml = ET.parse(urllib2.urlopen(regionInfoUri)).getroot()
    print 'regions (curr): %d' % int(riXml.attrib['number'])
    print 'regions (max):  %d' % int(riXml.attrib['max'])

    totalAvatars = 0
    totalObjects = 0

    for region in riXml:
        print '- region %-20s: avatars: %d' % (region.attrib['name'], int(region.attrib['avatars']))
        print '         %-20s  objects: %d' % (' ', int(region.attrib['objects']))

        totalAvatars += int(region.attrib['avatars'])
        totalObjects += int(region.attrib['objects'])

    print 'total avatars: %d' % totalAvatars
    print 'total objects: %d\n' % totalObjects


if __name__ == '__main__':
    for opensim in sys.argv[1:]:
        RegionInfo(opensim)

invoke it like this:

% osinfo http://opensim.zurich.ibm.com:9000

and you will get output similar to this:

== OpenSim server http://opensim.zurich.ibm.com:9000/ ==
regions (curr): 6
regions (max):  10
- region ST3D theatre        : avatars: 0
                               objects: 494
- region ST3D collaboration  : avatars: 0
                               objects: 283
- region ST3D boardroom      : avatars: 0
                               objects: 243
- region Zurich ISL          : avatars: 0
                               objects: 0
- region IBM ACVWS 2009      : avatars: 0
                               objects: 283
- region Sametime 3D         : avatars: 0
                               objects: 16
total avatars: 0
total objects: 1319

quite useful.

followers of the riveting¹ opensim-dev mailing list will have noticed that i “culled” AsteriskVoiceModule and SIPVoiceModule yesterday. the reason is quite simple: with the recent addition of the FreeSwitchVoiceModule we now have a voice module that works! and works without having to replace the SLVoice client that is part of the standard SecondLife® clients²

AsteriskVoiceModule which ansgar schmidt and i started last year — and which, together with work that alan webb and i did for the soon-to-be-released VivoxVoiceModule, provided the basis for the excellent work rob smart did for FreeSwitchVoiceModule — never really led to a complete voice solution as it required replacing SLVoice on the client side. rob figured out that FreeSwitch supported the same voice codecs that SLVoice was using and started experimenting. though FreeSwitchVoiceModule doesn’t (yet?) do spatial voice nor speaker indication it already works together quite nicely with SLVoice, so we decided to drop AsteriskVoice (and its little brother SIPVoice) from the opensim tree.

our Sametime3D test server is running a number of regions, some of them are rather on the monster side of things with gazillion scripts (ah, those brainstorming boards). until recently our opensim startup scripts would just grap a copy of mono and let it loose on OpenSim.exe and things would be working just fine.

sometime over the course of the last 10–14 days opensim must have crossed some (invisible) line drawn in the sandy thread beaches of monomania, as all of a sudden we noticed that three-avatar-meetings were working fine (chat & movement work) but as soon as we were joined by the fourth avatar, things came to a grinding halt: no movement, no chat (voice was still working but that’s to be expected as SL voice is pretty much running independent most of the time from what’s going on inside the region).

after some nose cone polishing, some more befuddled staring at the log files, and not finding anything immediately obvious, i went and discussed this problem with fellow opensimers on our intranet IRC opensim channel — sean had the good idea of mentioning the MONO_THREADS_PER_CPU environment variable. checking our startup script (just to make sure) showed that we were not setting it — and, thus, it was at its default setting of 50…

…so we went ahead and added it, setting it to 500. why 500? good question. and a question to which there is no clear answer: ask two opensimers and you’ll likely get three different recommendations, i’ve heard 75, 500, 1000, and also 2000. ralf haifisch recently tried to obtain clarity on this issue but so far the exact value for MONO_THREADS_PER_CPU remains a bit of magic.

and, guess what? that did the trick, it seems. we got over the three-avatar-limit.

here is the updated mono-build script for ubuntu hardy (intrepid might work as well).

update: since posting this i’ve been experiencing intermittent (that is, non-deterministic) mono crashes when running OpenSim, and have reverted to mono 2.0.1 which runs much more stable.