MSDN Magazine - July 2008 - (Page 91)

TransacTions Build Scalable Systems That Handle Failure Without Losing Data Udi Dahan Designing distributed systems has never been easier. for our WCF-based services. As a part of the message processing, With the increasing power coming with every version of the CLR, the productivity of Visual Studio®, and the fine-grained control found in frameworks like Windows® Communication Foundation (WCF), developers have all the tools they need to build scalable systems. Unfortunately, it is not enough. When working on a large-scale distributed system, my team found out that developing robust systems that handle failure scenarios without losing data is far from trivial. It’s not a failing in the toolset, though; these tools need to be employed in very specific patterns to achieve systems that are both scalable and robust. our services would often write data to the database. Not to belabor the point, but many systems are designed this way. Before taking our system to the staging environment, rigorous stress testing was done—the system was run under heavy load for a period of one week. The result of the stress testing showed that our system lost messages. During the analysis of what had occurred, we identified that the messages lost were primarily those containing order information—not surprisingly since most of the load on the system had to do with order information. However, we were surprised that all of this message loss occurred during one 10-minute interval. After that, the system operated normally. After going through various log files, we discovered that, during that week, a critical Windows patch was issued and the servers in the staging lab restarted themselves after automatically installing the patch. When considering the implications of this revelation— servers do restart every once in a while—we realized that we couldn’t ignore this. The system would need to be operational for several years and needed to weather many server upgrades and restarts. When processing a message over HTTP, a service would open a transaction against the database, attempt to write the data, and commit the transaction. Under regular circumstances this would succeed. If the message-processing server restarted mid-transaction, the database would detect the timeout of the transaction and July 2008 91 HTTP and Message Loss The first challenge the team faced was message loss. When first designing our system, we had decided on HTTP as the transport This article discusses: • Durable messaging • Transactions and system consistency • Handling problems with error queues • Message size and timing Technologies discussed: WcF, MsMQ

Table of Contents Feed for the Digital Edition of MSDN Magazine - July 2008

MSDN Magazine - July 2008 Contents Toolbox CLR Inside Out Flex Your Data Data Points Advanced Basics Office Space Cutting Edge Data Services ADO.NET Data and WPF Transactions WCF P2P Test Run Security Briefs Foundations .NET Matters {End Bracket}

MSDN Magazine - July 2008

For optimal viewing of this digital publication, please enable JavaScript and then refresh the page. If you would like to try to load the digital publication without using Flash Player detection, please click here.