SocketAsyncEventArgs Sending Timeout

Microsoft .Net 3.5 introduced a new way to do Async Socket operations the SocketAsyncEventArgs class. It was specifically designed for high performance application and over it works great. For the applications I have used it for it has significantly reduced latency for our operations.


But I do have one pet peeve about it. There is no way (that I have found) to timeout if a send operation is taking to long. The specific case I am running up against is that occasionally my data consumer will encounter an error where it will stop processing data out of the TCP buffer but not disconnect. The server keeps sending and everything looks normal on the server side until the TCP buffers fill up. Once that happens the next call to SendAsync will just end up in the abyss. The completed event of the SocketAsyncEventArgs will never be fired.


This ends up being a problem. I ended up having to add some monitoring logic on top of the socket sending to make sure I know when calls aren't returning. but this seems like something that should be handled by the framework. Hopefully this will get remedied soon.

IL Generation, Optomizations, Debuggers and AccessViolationExceptions

Recently I have been working with some code that dynamically generates IL. Before we get into the guts of the issue I was running into let me say that yes I know that generating IL on the fly is a terrible idea, it not something that I like doing. But in this case it is necessary.

We'll start with the code:

namespace TestApplication1

{

    class Program

    {

        static void Main()

        {

            string methodName = "TestMethod";

 

            AssemblyBuilder asmBuilder = AppDomain.CurrentDomain.DefineDynamicAssembly(

                new AssemblyName("DynamicAssembly"), AssemblyBuilderAccess.Run);

 

            ModuleBuilder moduleBuilder = asmBuilder.DefineDynamicModule("DynamicModule");

            TypeBuilder typeBuilder = moduleBuilder.DefineType("DynamicType");

 

            MethodBuilder method = typeBuilder.DefineMethod(methodName, MethodAttributes.Public);

 

            ILGenerator il = method.GetILGenerator();

 

            il.Emit(OpCodes.Newobj, typeof(TestClass).GetConstructor(Type.EmptyTypes));

 

            il.Emit(OpCodes.Callvirt, typeof(TestClass).GetMethod("StaticFunction", BindingFlags.Public | BindingFlags.Static));

 

            il.Emit(OpCodes.Pop);

 

            il.Emit(OpCodes.Ret);

 

 

            Type dynamicType = typeBuilder.CreateType();

 

            ConstructorInfo constructor = dynamicType.GetConstructor(Type.EmptyTypes);

 

 

            object o = constructor.Invoke(null);

            o.GetType().GetMethod(methodName).Invoke(o,null);

        }

    }

 

    public class TestClass

    {

        public static void StaticFunction()

        {

            Console.WriteLine("Static Hi");

        }

    }

}

Go ahead, paste into Visual Studio and hit F5, runs fine, StaticFunction is called, just like you would expect it to be.

Now instead of hitting F5 run it via Ctrl+F5:

System.Reflection.TargetInvocationException was unhandled
  Message="Exception has been thrown by the target of an invocation."
  Source="mscorlib"
  StackTrace:
       at System.RuntimeMethodHandle._InvokeMethodFast(Object target, Object[] arguments, SignatureStruct& sig, MethodAttributes methodAttributes, RuntimeTypeHandle typeOwner)
       at System.RuntimeMethodHandle.InvokeMethodFast(Object target, Object[] arguments, Signature sig, MethodAttributes methodAttributes, RuntimeTypeHandle typeOwner)
       at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture, Boolean skipVisibilityChecks)
       at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
       at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters)
       at TestApplication1.Program.Main() in C:\Documents and Settings\bwillard\Desktop\TestApplication1\Program.cs:line 38
  InnerException: System.AccessViolationException
       Message="Attempted to read or write protected memory. This is often an indication that other memory is corrupt."
       Source="DynamicAssembly"
       StackTrace:
              at DynamicType.TestMethod()
       InnerException:

What the heck? The same code ran, it should have yielded the same results. Well it turns out that running with a debugger attached causes the code that is generated not to be optimized (this is independent of weather you compiled with the optimize code flag or not). It is the same as if you had put the following code in the previous sample:

Type debuggableAttribute = typeof(DebuggableAttribute);

ConstructorInfo daCtor = debuggableAttribute.GetConstructor(new[] { typeof(DebuggableAttribute.DebuggingModes) });

CustomAttributeBuilder daBuilder = new CustomAttributeBuilder(daCtor, new object[]

                                                                          { DebuggableAttribute.DebuggingModes.DisableOptimizations |

                                                                            DebuggableAttribute.DebuggingModes.Default});

asmBuilder.SetCustomAttribute(daBuilder);

Mystery #1 solved, but why doesn't the code work when not run in a debugger. This is a fairly straight forward bug. You can't Callvirt on a static method, the line should read:

il.Emit(OpCodes.Call, method);

Took me a while to figure those two things out.

I'll leave you with one more bonus tip about emiting code. If you are getting the exception:

System.Reflection.TargetInvocationException was unhandled
  Message="Exception has been thrown by the target of an invocation."
  Source="mscorlib"
  StackTrace:
       at System.RuntimeMethodHandle._InvokeMethodFast(Object target, Object[] arguments, SignatureStruct& sig, MethodAttributes methodAttributes, RuntimeTypeHandle typeOwner)
       at System.RuntimeMethodHandle.InvokeMethodFast(Object target, Object[] arguments, Signature sig, MethodAttributes methodAttributes, RuntimeTypeHandle typeOwner)
       at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture, Boolean skipVisibilityChecks)
       at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
       at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters)
       at TestApplication1.Program.Main() in C:\Documents and Settings\bwillard\Desktop\TestApplication1\Program.cs:line 49
       at System.AppDomain._nExecuteAssembly(Assembly assembly, String[] args)
       at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)
       at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
       at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
       at System.Threading.ThreadHelper.ThreadStart()
  InnerException: System.InvalidProgramException
       Message="JIT Compiler encountered an internal limitation."
       Source="DynamicAssembly"
       StackTrace:
            at DynamicType.TestMethod()
       InnerException: 

What it means in this case is that there is an extra object on the stack when the function returned. That is why I need the last "pop" in my sample. Because I am calling a static function it doesn't consume the TestClass object that I created and that is now at the top of the stack.

Adding Search Paths to a Domain

Recently I was trying to load an assembly, call it assembly A, into my AppDomain and execute code in the assembly. This is usualy pretty easy thing to do, the problem comes from the fact that assembly A depends on another assembly B that is in the same directory as A.


If I was creating a new AppDomain this would be easy I could just add the search path to the AppDomainSetup when I create the domain. But in this case I wanted everything to be in one app domain so I couldn't do that.


The solution (that I found, there might be an easier way in which case please let me know) is to leverage closures and the AssemblyResolve event:

public static void LoadPackage(string dllPath, string className)

{

    string dllFullPath = Path.GetFullPath(dllPath);

    var assembly = Assembly.LoadFile(dllFullPath);

    var package = assembly.CreateInstance(className);

    if (null == package)

    {

        throw new Exception("Couldn't load type " + className + " from " + dllPath);

    }

    var asembliesInDirectory = new Dictionary<string, string>();

    foreach (string file in Directory.GetFiles(Path.GetDirectoryName(dllFullPath), "*.dll"))

    {

        try

        {

            asembliesInDirectory.Add(AssemblyName.GetAssemblyName(file).FullName, file);

        }

        catch{}

    }

    AppDomain.CurrentDomain.AssemblyResolve += (sender, eventArgs) =>

    {

        if (!asembliesInDirectory.ContainsKey(eventArgs.Name))

        {

         return null;

        }

        return Assembly.LoadFile(asembliesInDirectory[eventArgs.Name]);

    };

}

The AssemblyResolve event gets called when the CLR find an assembly it needs to load. Handling this event lets you add your own assembly resolving logic into the process. This code will check the directory you loaded your first dll from any time the CLR can't find a dll. The reason for the dictionary is purely for performance, it is expensive to scan the directory every time you are trying to load something. So this just pre-caches all the assembly names for easy lookup.

Programaticly Configuring NServiceBus

I just started implementing a PubSub system for our company using NServiceBus. Overall if has been super easy to setup. The samples are a great starting point and everything is pretty self explanatory.


However as I was trying to roll it out into production I realized that using the standard Spring configuration wasn't going to work for us. I wanted the name of the Msmq input queue to be specified programatilcy instead of via a config file. I knew it had to be possible but I had a really tough time figuring out how to do it. All the samples I could find were about how to set things up with Spring. So I though I would share how I did it.


The standard example shows:

NServiceBus.Configure.With()

    .XmlSerializer()

    .MsmqTransport()

        .IsTransactional(false)

        .PurgeOnStartup(false)

    .UnicastBus()

        .ImpersonateSender(false)

        .LoadMessageHandlers()

    .CreateBus()

    .Start();

So to dynamically set the MsmqTransoprt values you can write the following:

Configure configure = NServiceBus.Configure.With().SpringBuilder();

MsmqTransport transport = configure.Configurer.ConfigureComponent<MsmqTransport>(ComponentCallModelEnum.Singleton);

transport.InputQueue = "DynamicQueueName";

transport.NumberOfWorkerThreads = 1;

transport.ErrorQueue = "DynamicErrorQueue";

transport.MaxRetries = 5;

configure

    .XmlSerializer()

    .UnicastBus()

        .ImpersonateSender(false)

        .LoadMessageHandlers()

    .CreateBus()

    .Start();

This works for me, I have no idea if that is the best practice or if there is a better way to set these values programaticly. Hopefully this helps you and saves you from some the headache I went through (it always seems pretty simple once you get it figured out). Please let me know if you know a better way to do this.

NBA Data Analysis

Awhile ago Mark Cuban posted an interesting break down of some possible inequalities in the NBA scheduling. Basically some teams had more back-to-back games and more set of 4 games in 5 nights. The assumption was that teams would do worse with less rest time.

I was interested to see if that assumption was true. The best data I could get was from Department of Statistics, University of Munich, and the SFB386 which has the entire 95/96 NBA season broken down game by game.

I wrote a simple app to analyse the data (run the test NBATests.DataSetTests.ParseTest to re-run the analysis). The results of the test are in the spread sheet stats.xlsx in the zip file. In that season there didn't appear to be any 4-in-5 sets. That must be a new invention by the NBA.

The numbers are interesting. Overall the win percentage for the league when playing the second game of a back-to-back set is 44%. When taking into account a team's overall win percentage for the season the average difference in win percentage between games that were back-to-back and those where the team had a rest night before was 7%.

8 teams did better in back-to-back games, Miami led the way, their win percentage was 15% higher in their second game.

Atlanta on the other was abysmal, their win percent dropped 39% when they had to play two games in a row.

I had the thought that better teams might be better because of their bench and thus would fair better in back-to-back games because of their depth but this doesn't appear to be the case. The r-squared value for win percent plotted against win percent difference in back-to-back games was only .015.

So overall while some teams appear to get better in back-to-back games overall I think the assumption that it is a disadvantage to have more back-to-back games is a fair one.

There are all kind of other ways to slice up the data that I didn't do but someone should:

• Exclude games where both teams are playing back-to-back.
• Factor in home and away win percentages and not just overall win percentage.
• Doing some kind of qualitative analysis on the depth of the teams that year to see if that plays a factor.

Also if anyone has a data set(s) for more recent years send them over and I'll plug them in and see if these results hold year over year.