Server-Side Office Document Generation Bug

One of my current projects includes the capability of generating what can be very large Word 2007 documents. The size of the document is mostly driven by embedded Word, PDF and image files, with the business requirement of handling file sizes up to 150MB. We're using the Microsoft SDK for OpenXML Formats for generating these files, and all was well with the world, until we started testing large files on the server. Once we got to file sizes around 10MB, strange exceptions started showing up in the logs and the generation operation began failing when calling OpenXmlPart.FeedData():

System.ObjectDisposedException: Can not access a closed Stream.

This was particularly frustrating since the operation worked fine on local development machines, but failed miserably when deployed to the Windows Server 2003 development environment, and again, only when the file size of the documents being embedded got around 10MB+. After a lot of log4net debugging, strategically placed try/catch blocks, and a whole lot of Reflector spelunking, I was finally able to sort out what was actually going on, and it seems to be a bug in the .NET Framework.

Why me?!

The OpenXML SDK uses System.IO.Packaging for various operations and Packaging then uses the MS.Internal.IO.Packaging.SparseMemoryStream for various stream operations. This class is meant to use MemoryStreams for small files, and then switch to an IsolatedStorageFile stream for large files. I discovered this by putting a try/catch block directly around the call to OpenXmlPart.FeedData(), resulting in this error:

Unable to open the store. (Exception from HRESULT: 0x80131460) System.IO.IsolatedStorage.IsolatedStorageException: Unable to open the store. (Exception from HRESULT: 0x80131460) at System.IO.IsolatedStorage.IsolatedStorageFile.nOpen(String infoFile, String syncName) at System.IO.IsolatedStorage.IsolatedStorageFile.Lock() at System.IO.IsolatedStorage.IsolatedStorageFileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, IsolatedStorageFile isf) at MS.Internal.IO.Packaging.PackagingUtilities.SafeIsolatedStorageFileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, ReliableIsolatedStorageFileFolder folder) at MS.Internal.IO.Packaging.PackagingUtilities. CreateUserScopedIsolatedStorageFileStreamWithRandomName(Int32 retryCount, String& fileName) at MS.Internal.IO.Packaging.SparseMemoryStream.SwitchModeIfNecessary() ...

As we can see from the error, the application was failing when attempting to open the IsolatedStorageFile stream after SparseMemoryStream switched modes. It's important to highlight again that SparseMemoryStream only switches to IsolatedStorageFile-based streams when files being processed are large (~10MB, but I'm not sure on the exact size). This explains why everything worked fine on the server and developer machines when files were small.

Now we've got the issue of why large files worked on developer machines, but not on the server. This seems to be caused by the type of IsolatedStorageFile being instantiated by SparseMemoryStream through the use of the MS.Internal.IO.PackagingUtilities+ReliableIsolatedStorageFileFolder.ctor and it's corresponding call to MS.Internal.IO.Packaging.PackagingUtilities.UserHasProfile() to decide between a user- or machine-scoped IsolatedStorageFile. On a local developer machine, applications are commonly executing under the developer's account, so a user-scoped IsolatedStorageFile is instantiated and everything works fine. On Windows Server, however, ASP.NET applications are typically running as Network Service, and a user-scoped IsolatedStorageFile does not work for this account.

Unfortunately, the MS.Internal.IO.Packaging.PackagingUtilities.UserHasProfile() method checks the registry for the existence of a user profile node in HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\ProfileList and Network Service does, in fact, have a profile in the registry. This means that the .NET Framework is selecting a user-scoped IsolatedStorageFile and failing due to permissions (a machine-scoped IS would work fine).

Having finally understood what exactly was going on, I then set out to discuss the issue with some Microsoft contacts and verify that this is an issue in the framework (it is) and put together a sample application to more fully describe the issue to them. It may be easier to understand by looking at that sample, so you can check that out as well. The last word I've gotten back from Microsoft is that this issue will be resolved in future releases of the framework, but there wouldn't be any hotfixes in the short-term.

Any solutions?

While understanding what exactly was going on was a pain, we still had to find a solution to the problem. Luckily, another developer on the team had the bright idea of simply impersonating a user that doesn't have a profile in the registry at the time of document generation. While this isn't an ideal solution, it is an effective work-around for the problem. Since the impersonated user doesn't have a profile in the registry, the machine-scoped IsolatedStorageFile is instantiated and everything works fine.

Whew . . . I've got to admit that this was one of the most fun issues I've come across in my short career. That said, I'm still glad it's over!

Comments

[Christoffel Klimbie]

Hi Kevin,

Pherhaps my response is a bit late - I work for Aia Software and we are specialized in producing out of the box solutions for document composition.
I keep a blog on the decision whether to make or buy a document composition tool: www.datatextmerge.com.

You may have found a solution for the problem listed above.

Cheers,

Christoffel.

[Simon Corston-Oliver]

Thanks this helped point me in the right direction. In our case, we had code in a COM component trying to access a Package. As you describe, stuff was being streamed to IsolatedStorage by the framework. But because our COM code was running in the DefaultDomain with no evidence, we couldn't read it back.

We didn't have the option of running under a different user account so we had to run in a separate AppDomain that we created with appropriate permission.

http://rekiwi.blogspot.com/2008/12/unable-to-determine-identity-of-domain.html