after our intoxicating stress test ten days ago we repeated the stress test today with a group of volunteers — thus, real SL clients instead of bots.

the result: 21.

somewhere around the 21 avatar mark we are got stuck and the (single) region standalone system started becoming very laggy: avatars could not move anymore (or erratically at best), chat got relayed occasionally. interestingly enough we still saw new users logging in. looking at OpenSim’s log we saw that OpenSim was busy processing new user requests — a lot of new user requests.

so, the current hypothesis is that new users clog the out-pipe with large amount of texture requests, causing the existing users to starve and the system becoming unresponsive.

for stress testing i needed to create 100 avatar accounts quickly. instead of going via opensim’s console i started ipython and entered the following python script:

import xmlrpclib
os = xmlrpclib.Server('http://127.0.0.1:9000/')
for i in range(100):
    os.admin_create_user(dict(password = 'SECRET', 
                              user_firstname = 'Bot%d' % i,
                              user_lastname = 'Dude', 
                              user_password = 'b0tb0tb0t', 
                              start_region_x = 1000, 
                              start_region_y = 1000, 
                              model = 'Bot Dude')) 

a couple of minutes later i had successfully created 100 avatar accounts, ”Bot0 Dude” to ”Bot99′, all wearing the same outfit as ”Bot Dude”, an avatar i had prepared earlier.

voila! avatar machine

it was a dark and stormy night. lightning was ripping the heavens apart, thunder was rolling through the valley. no living soul dared venture outside…¹

…and i wasn’t. i sat in front of my screen, mesmerized by the numbers coming up: 20…27…34…38…40…43…48…52… eventually peaking at 81! we were stress testing our opensim server and our in-world build. the numbers represented rather brutal bots, that, as soon as they were in-world started running around, chatting with one another and grabbing all the textures they could find — we’d lovingly nicknamed them “hooligan bots” as they’d successfully managed to demolish a number of our opensim test instances and reduced them to nothing more than a pile of broken bits and bytes.

after fixing another memory gobbling piece of code and some tetchy ODE physics engine part, we’d gone past the 20 mark that had meant the end of our opensim instance in the past couple of trials and the numbers kept on climbing. during the test we had to create more and more test accounts to be able to keep sending in the bots. the system behaved extraordinarily stable: not only were we able to keep moving around and not only did opensim keep chugging along but also our scripts (themselves not exactly shy when it comes to CPU cycles and memory) did!

our test system, a 4 core intel system, running in 32-bit PAE mode with about 8GB of memory, coped rather well on the memory front: with about 5 regions (script heavy) we clocked at about 1.3 — 1.5 GB memory usage — CPU load, however, was a different story: that went up like crazy: up to 390% CPU utilization, load level of about 9–10. still, the only effect was that things became a bit sluggish.²

note, this was a one off! a pleasant and encouraging one off but nevertheless so far just a one off. we need to carry out further testing to corroborate this.

still, 81 avatars in one region!

i’ve been a big fan of the syntax highlighter aka syntaxhighligher evolved wordpress plugin for quite a while now. after updating to the syntaxhighlighter evolved version, my code sections started looking funny, like this:

a = None
b = 1</p>

<p>if all((a, b)):
    print 'yes!'

instead of like this:

a = None
b = 1

if all((a, b)):
    print 'yes!'

the problem seemed to be that syntaxhighlighter’s shortcodes confuddled markdown (or markdown confuddled syntaxhighlighter?). this morning i realized that the solution might be actually quite simple: instead of not indenting, indent the code by 4 space (as you usually do with markdown to tag code sections, sigh) and all might be well…

…and it was. sigh. quite easy.

recently i was in need of being able to send inventory items (well, a whole folder full of stuff) to the inventory of a logged-in avatar.

just copying the the inventory items to the avatar inventory (for example, using Scene.GiveInventoryItem() or Scene.GiveInventoryFolder()) will not cause your avatar to notice that there is additional junk in her inventory all of a sudden — once the avatar is logged out and logs in again, she will find the stuff in her inventory. kicking the avatar out every time you want to send some inventory her way is a tad bad on the user experience side¹. clearly not something we want to do.

after trying a 101 ways of sending inventory to the avatar and none of them catching, i finally remembered that avatars can give inventory to one another, even whole folders!² taking my favorite tool to the OpenSim source tree³ i found code in the InventoryTransferModule that seemed to take care of this process. the first attempt of just sending a GridInstantMessage via the usual means was not producing the right result: the avatar would get the pop-up that a folder was being given to her and would she like to keep it? looked good but checking the inventory revealed it to be a big blue lie.

here’s what i finally figured out. you need:

• a ScenePresence object representing the avatar
• the UUID of the sending “avatar”
• the name of the sending “avatar”
• the InventoryNodeBase object (actually you would pass in a subclass such as InventoryItemBase or InventoryFolderBase or similar)

here’s the “pseudo-code”:

// pseudo code: send inventory content to a logged in avatar's
// inventory 
//
// avatar: ScenePresence object
// src: struct containing "UUID" and "Name" of the source "avatar"

// get the CachedUserInfo object for the target avatar
CachedUserInfo cuiAvatar = 
    m_commsManager.UserProfileCacheService.GetUserDetails(avatar,UUID);
if (!cuiAvatar.HasReceivedInventory) cuiAvatar.FetchInventory();

// target folder is the Clothing folder (type 5)
InventoryFolderImpl dstFolder = cuiAvatar.FindFolderForType(5);

// copy source folder from src avatar to dstFolder
copyFolder = scene.GiveInventoryFolder(avatar.UUID, src.UUID, 
                                       dstFolder.ID);

// content is in the target folder, ping target avatar and let her
// know about it: we do this through a GridInstantMessage

// GridInstantMessage: prepare bucket byte array first byte is asset
// type, remaining bytes are the UUID of the InventoryItem
byte[] idBytes = item.ID.GetBytes();
byte[] bucket = new byte[1 + idBytes.Length];

if (item is InventoryFolderBase)
    bucket[0] = (byte)AssetType.Folder;
else if (item is InventoryItemBase)
    bucket[0] = (byte)(item as e).AssetType;

Array.Copy(idBytes, 0, bucket, 1, idBytes.Length);

GridInstantMessage im = 
    new GridInstantMessage(scene, src.UUID, src.Name, avatar.UUID,  
                           (byte)InstantMessageDialog.InventoryOffered,
                           false, item.Name, item.ID,
                           Vector3.Zero, bucket);

// send bulk update inventory message and GridInstantMessage
avatar.ControllingClient.SendBulkUpdateInventory(item);
avatar.ControllingClient.SendInstantMessage(im);
[/c-sharp]