In the Patterns and Practices Group's "Improving .NET Application Performance and Scalability",
which is available in full text online and as a PDF download from the above
link, as well as in softcover through MSPress and major booksellers, there are
over 1000 pages and appendixes of detailed information about how to improve .NET
application performance and scalability, written by the top experts in the
business. One area that is both little understood and potentially confusing is
the tuning of Internet Information Services 6.0.
The skinny about all this is that the PAP group says the default
settings shipped with IIS and the .NET Framework should be changed. They
provide detailed information in pages 332 through 342 in Chapter 6 on ASP.NET,
and they provide even more information in Chapter 17. I'll summarize some of the
more important points here, since I know that, human nature being what it is,
most people running IIS and reading this article probably have not waded through
this lengthy but excellent publication. Once you see the quality of the
information you can get from it, it may encourage you to do so; it is an
investment of your time as a professional ASP.NET developer that I highly
recommend. The fact that the book is made available free online by Microsoft
should not in any way diminish its importance or value to developers who
are interested in achieving the absolute best performance and scalability
from their .NET Applications.
NOTE: This "helper" article focuses almost
totally on the IIS-related issues and settings. However, Chapter 6 and
additional information in the various checklists and in Chapter 17 address many
other issues that are related to, but do not specifically involve IIS 6.0
settings. Some of these can be addressed at the machine.config level, others are
"best practices" coding techniques, and some can be addressed in web.config.
Paragraphs marked "Discussion" are my individual comments. The rest is (mostly)
untouched snippets from the PAP publication itself.
First, lets examine some of the "reduce contention" formula
settings. All this information, and a lot more, is right in the book:
|
Formula for Reducing Contention
The formula for reducing contention can give you a good empirical start for
tuning the ASP.NET thread pool. Consider using the Microsoft product
group-recommended settings that are shown in Table 6.1 if the following
conditions are true:
- You have available CPU.
- Your application performs I/O bound operations such as calling a Web method or accessing the file system.
- The ASP.NET Applications/Requests In Application Queue performance counter indicates that you have queued requests.
Table 6.1: Recommended Threading Settings for Reducing Contention
Configuration setting Default value (.NET Framework 1.1) Recommended value
maxconnection | 2 | 12 * #CPUs |
maxIoThreads | 20 | 100 |
maxWorkerThreads | 20 | 100 |
minFreeThreads | 8 | 88 * #CPUs |
minLocalRequestFreeThreads | 4 | 76 * #CPUs |
To address this issue, you need to configure the following items in the
Machine.config file. Apply the recommended changes that are
described in the following section, across the settings and not in isolation.
For a detailed description of each of these settings, see "Thread Pool
Attributes" in Chapter 17, "Tuning .NET Application Performance."
- Set maxconnection to 12 * # of CPUs . This setting controls the maximum number of outgoing HTTP connections that you can initiate from a client. In this case, ASP.NET is the client. Set maxconnection to 12 * # of CPUs.
- Set maxIoThreads to 100 . This setting controls the maximum number of I/O threads in the .NET thread pool. This number is automatically multiplied by the number of available CPUs. Set maxloThreads to 100.
- Set maxWorkerThreads to 100 . This setting controls the maximum number of worker threads in the thread pool. This number is then automatically multiplied by the number of available CPUs. Set maxWorkerThreads to 100.
- Set minFreeThreads to 88 * # of CPUs . This setting is used by the worker process to queue all the incoming requests if the number of available threads in the thread pool falls below the value for this setting. This setting effectively limits the number of requests that can run concurrently to maxWorkerThreads ? minFreeThreads . Set minFreeThreads to 88 * # of CPUs. This limits the number of concurrent requests to 12 (assuming maxWorkerThreads is 100).
- Set minLocalRequestFreeThreads to 76 * # of CPUs . This setting is used by the worker process to queue requests from localhost (where a Web application sends requests to a local Web service) if the number of available threads in the thread pool falls below this number. This setting is similar to minFreeThreads but it only applies to localhost requests from the local computer. Set minLocalRequestFreeThreads to 76 * # of CPUs.
Discussion: The proviso above indicates that
these settings should be used when your application has I/O bound operations and
the Applications/Requests In Application Queue perfcounter
indicates you have queued requests. However, I have found that settings
approaching those indicated can improve performance on ASP.NET apps that do not
exhibit these conditions. I recommend using the "Homer" web stress tool from at
least one remote machine (and preferably more than one machine, with the
supplied ASP controller page), or the .NET ACT Application Center Test
application, to throw a good solid load at your app and carefully measure the
performance statistics with each set of both the default and the above settings.
In particular, pay close attention to the Requests per second and the time to
last byte readings. This baseline testing scenario should provide the basis for
further tuning if it is necessary, and it doesn't take long at all. You can only
improve something if you have metrics, and the way you get the metrics is to
take the time to get them! You can easily script all kinds of "user paths"
through your ASP.NET application with testing software such as is mentioned
here, and get the important baseline metrics you need. One more thing-- rule
number 1 of software testing and debugging:
"When you are going to change something, ONLY CHANGE ONE THING AT
A TIME!" Test it, get the metrics, and only then, proceed.
Kernel Mode Caching
If you deploy your application on Windows Server 2003, ASP.NET pages
automatically benefit from the IIS 6.0 kernel cache. The kernel cache is managed
by the HTTP.sys kernel-mode device driver. This driver handles all HTTP
requests. Kernel mode caching may produce significant performance gains because
requests for cached responses are served without switching to user mode.
The following default setting in the Machine.config file ensures that
dynamically generated ASP.NET pages can use kernel mode caching, subject to the
requirements listed below.
<httpRunTime enableKernelOutputCache="true" .
. ./>
Dynamically generated ASP.NET pages are automatically cached subject to the
following restrictions:
- Pages must be retrieved by using HTTP GET requests. Responses to HTTP POST requests are not cached in the kernel.
- Query strings are ignored when responses are cached. If you want a request for http://contoso.com/myapp.aspx?id=1234 to be cached in the kernel, all requests for http://contoso.com/myapp.aspx are served from the cache, regardless of the query string.
- Pages must have an expiration policy. In other words, the pages must have an Expires header.
- Pages must not have VaryByParams .
- Pages must not have VaryByHeaders .
- The page must not have security restrictions. In other words, the request must be anonymous and not require authentication. The HTTP.sys driver only caches anonymous responses.
- There must be no filters configured for the W3wp.exe file instance that are unaware of the kernel cache.
Discussion: The "enableKernelOutputCache = "true" setting IS
NOT present in the default machine.config "httpRunTime" element. Since it is not
present, we should be able to expect that the default setting of "true" is
automatic. Personally, I feel better explicitly putting the attribute in there,
and setting it to "true". As an aside, I have found that it is ALWAYS a good
idea to KEEP A BACKUP COPY of your machine.config stored somewhere safe.
Tuning the Thread Pool for Burst Load Scenarios
If your application experiences unusually high loads of users in small bursts
(for example, 1000 clients all logging in at 9 A.M. in the morning), your system
may be unable to handle the burst load. Consider setting
minWorkerThreads and minIOThreads as specified
in Knowledge Base article 810259, "FIX: SetMinThreads and GetMinThreads API
Added to Common Language Runtime ThreadPool Class," at http://support.microsoft.com/default.aspx?scid=kb;en-us;810259.
Discussion: The .NET Threadpool is somewhat limited in its
flexibility and is specifically limited in terms of how many instances you may
have per process, since it is static. If you have ASP.NET applications that
specifically need to run background thread processing, you may wish to
investigate using a custom threadpool class. I have used Ami
Bar's SmartThreadPool with great success, and have even modified it to
provide a ThreadPriority overload. You can have more than one instance of this
pool, and each can be custom configured. This type of approach provides maximum
flexibility while simultaneously permitting individual threadpool tuning of
critical resources.
Tuning the Thread Pool When Calling COM Objects
ASP.NET Web pages that call single-threaded apartment (STA) COM objects
should use the ASPCOMPAT attribute. The use of this attribute
ensures that the call is executed using a thread from the STA thread pool.
However, all calls to an individual COM object must be executed on the same
thread. As a result, the thread count for the process can increases during
periods of high load. You can monitor the number of active threads used in the
ASP.NET worker process by viewing the Process:Thread Count
(aspnet_wp instance) performance counter.
The thread count value is higher for an application when you are using
ASPCOMPAT attribute compared to when you are not using it. When
tuning the thread pool for scenarios where your application extensively uses STA
COM components and the ASPCOMPAT attribute, you should ensure
that the total thread count for the worker process does not exceed the following
value.
75 + ((maxWorkerThread + maxIoThreads) * #CPUs * 2)
Evaluating the Change
To determine whether the formula for reducing contention has worked, look for
improved throughput. Specifically, look for the following improvements:
- CPU utilization increases.
- Throughput increases according to the ASP.NET Applications\Requests/Sec performance counter.
- Requests in the application queue decrease according to the ASP.NET Applications\Requests In Application Queue performance counter.
If this change does not improve your scenario, you may have a CPU-bound
scenario. In a CPU-bound scenario, adding more threads may increase thread
context switching, further degrading performance.
When tuning the thread pool, monitor the Process\Thread Count
(aspnet_wp) performance counter. This value should not be more than the
following.
75 + ((maxWorkerThread + maxIoThreads) * #CPUs)
If you are using AspCompat, then this value should not be more than the
following.
75 + ((maxWorkerThread + maxIoThreads) * #CPUs * 2)
Values beyond this maximum tend to increase processor context switching.
Discussion: There is a long list of attention items that
revolve around and are tightly woven into the IIS tuning issue for ASP.NET
application tuning and scalability. These include, but are not limted to the
following:
- Improving page response times.
- Designing scalable Web applications.
- Using server controls efficiently.
- Using efficient caching strategies.
- Analyzing and applying appropriate state management techniques.
- Minimizing view state impact.
- Improving performance without impacting security.
- Minimizing COM interop scalability issues.
- Optimizing threading.
- Optimizing resource management.
- Avoiding common data binding mistakes.
- Using security settings to reduce server load.
- Avoiding common deployment mistakes.
You can find detailed treatment of most of these issues in Chapter 6 of the
above-captioned publication.
I hope this brief synopsis of IIS tuning parameters is useful to you. Once
again, I strongly recommend reading all this in the bigger context of the book,
and mapping out an optimization plan that includes code review, refactoring, and
optimization tuning both at the ASP.NET application and IIS webserver levels.
One of the great things about the lessons learned from IIS / ASP.NET testing and
tuning optimizations is that they can be carried forward to new applications and
will improve your skills and value as a professional developer. I spent nearly
three weeks at the Microsoft Testing Lab in Charlotte, NC under the tutelage of
Dennis Bass and his fine crew, and the lessons learned there were invaluable. If
this book were avalable then, I may not have needed to spend so many nights in
hotel rooms.