Deep dive into .NET Garbage Collection

Basics of Operating System

Before we start, you should get acquainted with some basic operating system terminology:

• Physical Memory: The actual physical memory chips in a computer. Only the operating system manages physical memory directly.
• Virtual Memory: A logical organization of memory in a given process. Virtual memory size can be larger than physical memory. For example, 32-bit programs have a 4 GB address space, even if the computer itself only has 2 GB of RAM (random-access memory). Windows allows the program to access only 2 GB of that by default, but all 4 GB is possible if the executable has the large address aware bit set. (On 32-bit versions of Windows, large-address aware programs are limited to 3 GB, with 1 GB reserved for the operating system.) As of Windows 8.1 and Server 2012, 64-bit processes have a 128 TB address space, far larger than the 4 TB physical memory limit. Some of the virtual memory may be in RAM while other parts are stored on disk in a paging file.
• Reserved Memory: A region of virtual memory address space that has been reserved for the process and thus will not be allocated to a future requester. Reserved memory cannot be used for memory allocation requests because nothing is backing it — it is just a description of a range of memory addresses.
• Committed Memory: A region of memory that has a physical backing store. This can be RAM or disk.
• Page: An organizational unit of memory. Blocks of memory are allocated in a page, which is usually a few KB in size.
• Paging: The process of transferring pages between regions of virtual memory. The page can move to or from another process (soft paging) or the disk (hard paging). Soft paging can be accomplished very quickly by mapping the existing memory into the current process’s virtual address space. Hard paging involves a relatively slow transfer of data to or from a disk. Your program must avoid this at all costs to maintain good performance.
• Private Bytes: Committed virtual memory private to the specific process (not shared with any other processes).
• Virtual Bytes: Allocated memory in the process’s address space, some of which may be backed by the page file, possibly shared with other processes or private to the process.
• Working Set: The amount of virtual memory currently resident in physical memory (usually RAM).
• Working Set-Private: The number of private bytes currently resident in physical memory.
• Thread Count: The number of threads in the process. This may or may not be equal to the number of .NET threads.

{ 
   "runtimeOptions": { 
       "configProperties": { "System.GC.Server": true } 
   } 
}

string result;
if (GCSettings.IsServerGC == true) result = "server";
else result = "workstation";

Having multiple heaps gives a couple of advantages: Garbage collection happens in parallel. Each GC thread collects one of the heaps. This can make garbage collection significantly faster than in workstation GC. In some cases, allocations can happen faster, especially on the large object portion of the heap, where allocations are spread across all the heaps. There are other internal differences as well such as larger segment sizes, which can mean a longer time between garbage collections.

GCSettings.LargeObjectHeapCompactionMode =
GCLargeObjectHeapCompactionMode.CompactOnce;
GC.Collect();

{ 
   "runtimeOptions": { 
       "configProperties": { "System.GC.LOHThreshold": 120000 } 
   }
}

{ 
   "runtimeOptions": { 
       "configProperties": { "System.GC.Concurrent": false } 
   } 
}

bool success = GC.TryStartNoGCRegion( totalSize: 2000000, lohSize: 1000000, disallowFullBlockingGC: true);
if (success)
{
   try { 
     // do allocations 
   }
   finally {
      if (GCSettings.LatencyMode == GCLatencyMode.NoGCRegion) { 
        GC.EndNoGCRegion(); 
      }
   }
}

Performance Tips

Reduce Allocation Rate

This almost goes without saying, but if you reduce the amount of memory you are allocating, you reduce the pressure on the garbage collector to operate. You can also reduce memory fragmentation and CPU usage. It can take some creativity to achieve this goal and it might conflict with other design goals.

Critically examine each object and ask yourself:

• Do I really need this object at all?
• Does it have fields that I can get rid of?
• Can I reduce the size of arrays?
• Can I reduce the size of primitives (Int64 to Int32, for example)? Are some objects used only in rare circumstances and can, therefore, be initialized only when needed?
• Can I convert some classes to structs so they live on the stack, or as part of another object, and have no per-instance overhead?
• Am I allocating a lot of memory, to use only a small portion of it?
• Can I get this information in some other way?
• Can I allocate memory upfront?

Collect objects in gen 0 or not at all

Put differently, you want objects to have an extremely short life so that the garbage collector will never touch them at all, or, if you cannot do that, they should go to gen 2 as fast as possible and stay there forever, never to be collected. This means that you maintain a reference to long-lived objects forever. Often, this also means pooling reusable objects, especially anything on the large object heap. Garbage collections get more expensive in each generation. You want to ensure there are many gen 0/1 collections and very few gen 2 collections. Even with background GC for gen 2, there is still a CPU cost that you would rather not pay: a processor the rest of your program should be using.

Reduce Object Lifetime

The shorter an object’s lifetime, the less chance it has of being promoted to the next generation when a GC comes along. In general, you should not allocate objects until right before you need them. The exception would be when the cost of object creation is so high it makes sense to create them at an earlier point when it will not interfere with other processing.

Reduce References Between Objects

References between objects of different generations can cause inefficiencies in the garbage collector, specifically references from older objects to newer objects. For example, if an object in generation 2 has a reference to an object in generation 0, then every time a gen 0 GC occurs, a portion of gen 2 objects will also have to be scanned to see if they are still holding onto this reference to a generation 0 object. It is not as expensive as a full GC, but it is still unnecessary work if you can avoid it.

Avoid Pinning

Pinning an object fixes it in place so that the garbage collector cannot move it. Pinning exists so that you can safely pass managed memory references to unmanaged code. It is most commonly used to pass arrays or strings to unmanaged code but is also used to gain direct fixed memory access to data structures or fields. If you are not doing interop with unmanaged code and you do not have any unsafe code, then you haven’t the need to pin at all.

Avoid Finalizers

Never implement a finalizer unless it is required. Finalizers are code, triggered by the garbage collector to clean up unmanaged resources. They are called from a single thread, one after the other, and only after the garbage collector declares the object dead after a collection. This means that if your class implements a finalizer, you are guaranteeing that it will stay in memory even after the collection that should have killed it.

There is also additional bookkeeping to be done on each GC as the finalizer list needs to be continually updated if the object is relocated. All of this combines to decrease overall GC efficiency and ensure that your program will dedicate more CPU resources to cleaning up your object.

If you do implement a finalizer, you must also implement the IDisposable interface to enable explicit cleanup, and call GC.SuppressFinalize(this) in the Dispose method to remove the object from the finalization queue. As long as you call Dispose before the next collection, it will clean up the object properly without the need for the finalizer to run. The following example correctly demonstrates this pattern.

class Foo : IDisposable { 
  private bool disposed = false; 
  private IntPtr handle; 
  private IDisposable managedResource; ~Foo() // Finalizer { Dispose(false); } 
  public void Dispose() { 
    Dispose(true); 
    GC.SuppressFinalize(this); 
  } 
  protected virtual void Dispose(bool disposing) { 
    if (this.disposed) { return; } 
    if (disposing) { 
      // Not safe to do this from finalizer 
      this.managedResource.Dispose(); 
    }

      // Cleanup unmanaged resources that are safe to 
      // do so in a finalizer 
      UnsafeClose(this.handle); 

      // If the base class is IDisposable object 
      // make sure you call 
      base.Dispose(disposing); 
      this.disposed = true; 
  }
}

Dispose methods and finalizers should never throw exceptions. Should an exception occur during a finalizer’s execution, then the process will terminate. Finalizers should also be careful doing any kind of I/O, even as simple as logging.

You may have heard that finalizers are guaranteed to run. This is generally true, but not absolutely so. If a program is force-terminated then no more code runs and the process dies immediately. The finalizer thread is triggered by a garbage collection, so if there are no garbage collections, finalizers will not run. There is also a time limit to how long all of the finalizers are given on process shutdown. If your finalizer is at the end of the list, it may be skipped. Moreover, because finalizers execute sequentially, if another finalizer has an infinite loop bug in it, then no finalizers after it will ever run. This can lead to memory leaks. For all these reasons, you should not rely on finalizers to clean up the state external to your process.

Avoid Large Object Allocations

Not all allocations go to the same heap. Objects over a certain size will go to the large object heap and immediately be in gen 2. You should avoid allocations on the large object heap as much as possible. Not only is collecting garbage from this heap more expensive, but it is also more likely to fragment, causing unbounded memory to increase over time. Continuous allocations to the large object heap send a strong signal to the garbage collector to do continuous garbage collections—not a good place to be in.

Avoid Copying Buffers

You should usually avoid copying data whenever you can. For example, suppose you have read file data into a MemoryStream (preferably a pooled one if you need large buffers). Once you have that memory allocated, treat it as read-only and every component that needs to access it will read from the same copy of the data.

A common requirement, then, is to refer to sub-ranges of a buffer, array, or memory range. .NET provides two ways to accomplish this at present. The first option, available only for arrays, is the ArraySegment struct to represent just a portion of the underlying array. This ArraySegment can be passed around to APIs independent of the original stream, and you can even attach a new MemoryStream to just that segment. Throughout all of this, no copy of the data has been made.

var memoryStream = new MemoryStream(2048); 
var segment = new ArraySegment(memoryStream.GetBuffer(), 100, 1024); 
var blockStream = new MemoryStream(segment.Array, segment.Offset, segment.Count);

The biggest problem with copying memory is not the CPU necessarily, but the GC. If you find yourself needing to copy a buffer, then try to copy it into another pooled or existing buffer to avoid any new memory allocations. A newer option for representing pieces of existing buffers is the Span struct that is allocated on the stack rather than on the managed heap.

A Span<T> represents a contiguous region of arbitrary memory. A Span<T> instance is often used to hold the elements of an array or a portion of an array. Unlike an array, however, a Span<T> instance can point to managed memory, native memory, or memory managed on the stack.

// Create a span over an array. 
var array = new byte[100]; 
var arraySpan = new Span(array); 
byte data = 0;
for (int ctr = 0; ctr < arraySpan.Length; ctr++) arraySpan[ctr] = data++; 
int arraySum = 0; 
foreach (var value in array) arraySum += value; 
Console.WriteLine($"The sum is {arraySum}"); // Output: The sum is 4950

The following example creates a Span<Byte> from 100 bytes of native memory:

// Create a span from native memory. 
var native = Marshal.AllocHGlobal(100); 
Span nativeSpan; 
unsafe { nativeSpan = new Span(native.ToPointer(), 100); } 
byte data = 0; 
for (int ctr = 0; ctr < nativeSpan.Length; ctr++) nativeSpan[ctr] = data++; 
int nativeSum = 0; 
foreach (var value in nativeSpan) nativeSum += value; 
Console.WriteLine($"The sum is {nativeSum}"); 
Marshal.FreeHGlobal(native); // Output: The sum is 4950

There are also utility methods to convert from arrays and ArraySegment structs to Span structs.

Pool Long-Lived and Large Objects

Objects live very briefly or forever. They should either go away in gen 0 collections or last forever in gen 2. Those objects do not obviously need to last forever, but their natural lifetime in the context of your program ensures they will live longer than the period of a gen 0 (and maybe gen 1) garbage collection. These types of objects are candidates for pooling. Another strong candidate for pooling is any object that you allocate on the large object heap, typically collections.

ArrayPool<T> is a high-performance pool of managed arrays. You can find it in System. Buffers package and it’s source code is available on GitHub.

var samePool = ArrayPool.Shared; 
byte[] buffer = samePool.Rent(minLength); 
try { 
  Use(buffer); 
} 
finally { 
  samePool.Return(buffer); // don't use the reference to the buffer after returning it! 
} 
void Use(byte[] buffer) // it's an array

How to use it?

• Recommended: use the ArrayPool<T>. The shared property, which returns a shared pool instance. It’s thread-safe and all you need to remember is that it has a default max array length, equal to 2^20 (1024*1024 = 1 048 576).
• Call the static ArrayPool<T>. Create a method, which creates a thread-safe pool with custom maxArrayLength and maxArraysPerBucket. You might need it if the default max array length is not enough for you. Please be warned, that once you create it, you are responsible for keeping it alive.
• Derive a custom class from abstract ArrayPool<T> and handle everything on your own.

Remember that a pool that never dumps objects is indistinguishable from a memory leak. Your pool should have a bounded size (in either bytes or number of objects), and once that has been exceeded, it should drop objects for the GC to clean up. Ideally, your pool is large enough to handle normal operations without dropping anything and the GC is only needed after brief spikes of unusual activity.

Use Weak References For Caching

Weak references are references to an object that still allow the garbage collector to clean up the object. They are in contrast to the default strong references, which prevent collection completely (for that object). They are mostly useful for caching expensive objects that you would like to keep around but are willing to let go if there is enough memory pressure. Weak references are a core CLR concept that is exposed through a couple of .NET classes:

• WeakReference
• WeakReference<T>

Example of usage:

// The underlying Foo object can be garbage collected at any time! WeakReference 
weakRef = new WeakReference(new Foo()); 
// Create a strong reference to the object, 
// now no longer eligible for GC 
Foo myFoo; 
if (weakRef.TryGetTarget(out myFoo)) { ... }

You can still have other references, both strong and weak, to the same object. The collection will only happen if the only references to it are weak (or nonexistent).

Most applications do not need to use weak references at all, but some criteria may indicate good usage:

• Memory use needs to be tightly restricted (such as on mobile devices).
• Object lifetime is highly variable. If your object lifetime is predictable, you can just use strong references and control their lifetime directly.
• Objects are large, but easy to create. Weak references are ideal for objects that would be nice to have around, but if they are not, you can easily regenerate them or do without them. (Note that this also implies that the object size should be significant compared to the overhead of using the additional WeakReference objects in the first place).

Read more posts on our blog!
Find out the difference between custom vs off-the-shelf software, python or .net

Bonus links:

https://github.com/Microsoft/Microsoft.IO.RecyclableMemoryStreamhttps://github.com/Microsoft/Microsoft.IO.RecyclableMemoryStream

https://www.codeproject.com/Articles/1191534/To-Heap-or-not-to-Heap-That-s-the-Large-Object-Que

A bonus reading: how to hire developers for a startup?