the result: 21.

somewhere around the 21 avatar mark we are got stuck and the (single) region standalone system started becoming very laggy: avatars could not move anymore (or erratically at best), chat got relayed occasionally. interestingly enough we still saw new users logging in. looking at OpenSim’s log we saw that OpenSim was busy processing new user requests — a lot of new user requests.

so, the current hypothesis is that new users clog the out-pipe with large amount of texture requests, causing the existing users to starve and the system becoming unresponsive.

import xmlrpclib
os = xmlrpclib.Server('http://127.0.0.1:9000/')
for i in range(100):
    os.admin_create_user(dict(password = 'SECRET', 
                              user_firstname = 'Bot%d' % i,
                              user_lastname = 'Dude', 
                              user_password = 'b0tb0tb0t', 
                              start_region_x = 1000, 
                              start_region_y = 1000, 
                              model = 'Bot Dude')) 

a couple of minutes later i had successfully created 100 avatar accounts, ”Bot0 Dude” to ”Bot99′, all wearing the same outfit as ”Bot Dude”, an avatar i had prepared earlier.

voila! avatar machine

…and i wasn’t. i sat in front of my screen, mesmerized by the numbers coming up: 20…27…34…38…40…43…48…52… eventually peaking at 81! we were stress testing our opensim server and our in-world build. the numbers represented rather brutal bots, that, as soon as they were in-world started running around, chatting with one another and grabbing all the textures they could find — we’d lovingly nicknamed them “hooligan bots” as they’d successfully managed to demolish a number of our opensim test instances and reduced them to nothing more than a pile of broken bits and bytes.

after fixing another memory gobbling piece of code and some tetchy ODE physics engine part, we’d gone past the 20 mark that had meant the end of our opensim instance in the past couple of trials and the numbers kept on climbing. during the test we had to create more and more test accounts to be able to keep sending in the bots. the system behaved extraordinarily stable: not only were we able to keep moving around and not only did opensim keep chugging along but also our scripts (themselves not exactly shy when it comes to CPU cycles and memory) did!

our test system, a 4 core intel system, running in 32-bit PAE mode with about 8GB of memory, coped rather well on the memory front: with about 5 regions (script heavy) we clocked at about 1.3 — 1.5 GB memory usage — CPU load, however, was a different story: that went up like crazy: up to 390% CPU utilization, load level of about 9–10. still, the only effect was that things became a bit sluggish.²

note, this was a one off! a pleasant and encouraging one off but nevertheless so far just a one off. we need to carry out further testing to corroborate this.

still, 81 avatars in one region!

just copying the the inventory items to the avatar inventory (for example, using Scene.GiveInventoryItem() or Scene.GiveInventoryFolder()) will not cause your avatar to notice that there is additional junk in her inventory all of a sudden — once the avatar is logged out and logs in again, she will find the stuff in her inventory. kicking the avatar out every time you want to send some inventory her way is a tad bad on the user experience side¹. clearly not something we want to do.

after trying a 101 ways of sending inventory to the avatar and none of them catching, i finally remembered that avatars can give inventory to one another, even whole folders!² taking my favorite tool to the OpenSim source tree³ i found code in the InventoryTransferModule that seemed to take care of this process. the first attempt of just sending a GridInstantMessage via the usual means was not producing the right result: the avatar would get the pop-up that a folder was being given to her and would she like to keep it? looked good but checking the inventory revealed it to be a big blue lie.

here’s what i finally figured out. you need:

• a ScenePresence object representing the avatar
• the UUID of the sending “avatar”
• the name of the sending “avatar”
• the InventoryNodeBase object (actually you would pass in a subclass such as InventoryItemBase or InventoryFolderBase or similar)

here’s the “pseudo-code”:

// pseudo code: send inventory content to a logged in avatar's
// inventory 
//
// avatar: ScenePresence object
// src: struct containing "UUID" and "Name" of the source "avatar"

// get the CachedUserInfo object for the target avatar
CachedUserInfo cuiAvatar = 
    m_commsManager.UserProfileCacheService.GetUserDetails(avatar,UUID);
if (!cuiAvatar.HasReceivedInventory) cuiAvatar.FetchInventory();

// target folder is the Clothing folder (type 5)
InventoryFolderImpl dstFolder = cuiAvatar.FindFolderForType(5);

// copy source folder from src avatar to dstFolder
copyFolder = scene.GiveInventoryFolder(avatar.UUID, src.UUID, 
                                       dstFolder.ID);

// content is in the target folder, ping target avatar and let her
// know about it: we do this through a GridInstantMessage

// GridInstantMessage: prepare bucket byte array first byte is asset
// type, remaining bytes are the UUID of the InventoryItem
byte[] idBytes = item.ID.GetBytes();
byte[] bucket = new byte[1 + idBytes.Length];

if (item is InventoryFolderBase)
    bucket[0] = (byte)AssetType.Folder;
else if (item is InventoryItemBase)
    bucket[0] = (byte)(item as e).AssetType;

Array.Copy(idBytes, 0, bucket, 1, idBytes.Length);

GridInstantMessage im = 
    new GridInstantMessage(scene, src.UUID, src.Name, avatar.UUID,  
                           (byte)InstantMessageDialog.InventoryOffered,
                           false, item.Name, item.ID,
                           Vector3.Zero, bucket);

// send bulk update inventory message and GridInstantMessage
avatar.ControllingClient.SendBulkUpdateInventory(item);
avatar.ControllingClient.SendInstantMessage(im);

#!/usr/bin/python
# -*- encoding: utf-8 -*-

import sys
import urllib2
import xml.etree.ElementTree as ET

def RegionInfo(url):
    if not url.endswith('/'):
        url = '%s/' % url
    regionInfoUri = '%sadmin/regioninfo/' % url
    print '== OpenSim server %s ==' % url

    riXml = ET.parse(urllib2.urlopen(regionInfoUri)).getroot()
    print 'regions (curr): %d' % int(riXml.attrib['number'])
    print 'regions (max):  %d' % int(riXml.attrib['max'])

    totalAvatars = 0
    totalObjects = 0

    for region in riXml:
        print '- region %-20s: avatars: %d' % (region.attrib['name'], int(region.attrib['avatars']))
        print '         %-20s  objects: %d' % (' ', int(region.attrib['objects']))

        totalAvatars += int(region.attrib['avatars'])
        totalObjects += int(region.attrib['objects'])

    print 'total avatars: %d' % totalAvatars
    print 'total objects: %d\n' % totalObjects


if __name__ == '__main__':
    for opensim in sys.argv[1:]:
        RegionInfo(opensim)

invoke it like this:

% osinfo http://opensim.zurich.ibm.com:9000

and you will get output similar to this:

== OpenSim server http://opensim.zurich.ibm.com:9000/ ==
regions (curr): 6
regions (max):  10
- region ST3D theatre        : avatars: 0
                               objects: 494
- region ST3D collaboration  : avatars: 0
                               objects: 283
- region ST3D boardroom      : avatars: 0
                               objects: 243
- region Zurich ISL          : avatars: 0
                               objects: 0
- region IBM ACVWS 2009      : avatars: 0
                               objects: 283
- region Sametime 3D         : avatars: 0
                               objects: 16
total avatars: 0
total objects: 1319

quite useful.

AsteriskVoiceModule which ansgar schmidt and i started last year — and which, together with work that alan webb and i did for the soon-to-be-released VivoxVoiceModule, provided the basis for the excellent work rob smart did for FreeSwitchVoiceModule — never really led to a complete voice solution as it required replacing SLVoice on the client side. rob figured out that FreeSwitch supported the same voice codecs that SLVoice was using and started experimenting. though FreeSwitchVoiceModule doesn’t (yet?) do spatial voice nor speaker indication it already works together quite nicely with SLVoice, so we decided to drop AsteriskVoice (and its little brother SIPVoice) from the opensim tree.

sometime over the course of the last 10–14 days opensim must have crossed some (invisible) line drawn in the sandy thread beaches of monomania, as all of a sudden we noticed that three-avatar-meetings were working fine (chat & movement work) but as soon as we were joined by the fourth avatar, things came to a grinding halt: no movement, no chat (voice was still working but that’s to be expected as SL voice is pretty much running independent most of the time from what’s going on inside the region).

after some nose cone polishing, some more befuddled staring at the log files, and not finding anything immediately obvious, i went and discussed this problem with fellow opensimers on our intranet IRC opensim channel — sean had the good idea of mentioning the MONO_THREADS_PER_CPU environment variable. checking our startup script (just to make sure) showed that we were not setting it — and, thus, it was at its default setting of 50…

…so we went ahead and added it, setting it to 500. why 500? good question. and a question to which there is no clear answer: ask two opensimers and you’ll likely get three different recommendations, i’ve heard 75, 500, 1000, and also 2000. ralf haifisch recently tried to obtain clarity on this issue but so far the exact value for MONO_THREADS_PER_CPU remains a bit of magic.

and, guess what? that did the trick, it seems. we got over the three-avatar-limit.

update: since posting this i’ve been experiencing intermittent (that is, non-deterministic) mono crashes when running OpenSim, and have reverted to mono 2.0.1 which runs much more stable.

Normally, I am one of the most conservative members of OpenSim, but to see a teacher, SnowDrop Short gather 12 students together, form teams of three, exchange e-mails, assign a little independent study is so exhilarating that I needed to describe it tonight.

this is really what 3D platforms are about: collaboration, learning together, accomplishing something together! excellent stuff!

also, considering that OpenSim is just a bit over 2 years old — not a bad accomplishment!

The consensus is that OpenSim was ‘born’ on Jan 29 2007, when Darren Guard (MW) made his prototypical c# 3D world server publicly available.

i became involved with OpenSim almost about a year ago: sean dague asked me whether i could help with trying to figure out why OpenSim on PowerPC was not quite working¹, which i eventually did² — and got hooked…

…since then i’ve been working on voice support, splitting the chat module, adding a couple of REST interface for regions, and have been heavily involved in IBM’s Sametime3D effort which extends IBM’s Sametime instant messaging product with the capability to extend a chat session into a 3D meeting space:

considering that it’s just two years since

…MW came on [#libsl] and said ‘I have a prototype for an c# SL server, but before I publish it I need somebody to log onto it to see if it supports multiple clients’… [OpenSim turns 2]

it is utterly amazing how far OpenSim has come already!

here’s to a brilliant future! to virtual world domination! happy birthday OpenSim!