Juliet C# Benchmark and the SecureString case

Juliet C# and the benchmark initiative

As part of a larger initiative to improve the quality of Sonar’s products findings, in 2023 our teams worked on SAST benchmarks coverage. The reasons behind this are explained in a previous Top 3 C# SAST Benchmarks post that we encourage you to read.

Multiple benchmarks have been selected for each of our products’ flagship languages among which Juliet C# 1.3.

Juliet C# is a project from the National Institute of Standards and Technology of the USA, currently in version 1.3. It is known for supporting over a hundred CWEs, combined with a small set of code variations to form more than 28,000 test cases. 

We put a lot of effort into supporting this benchmark, partly due to its size. Especially, building the ground truth, the list of all valid findings on which a SAST engine should raise, took a lot of time. In the following, we want to give you a glimpse of the work we did around Juliet and some of its test cases.

Juliet C# - the SecureString test case

Among all the test cases implemented in the Juliet C# benchmark, a subset proved to be particularly interesting. It can be summarized by the following code sample. It has been adapted from the CWE313_Cleartext_Storage_in_a_File_or_on_Disk__ReadLine_01.cs test case.

using System.Security;

internal class Program
{
    private static void Main(string[] args)
    {
        string data;
        data = ""; /* Initialize data */
        {
            /* read user input from console with ReadLine */
            try
            {
                /* POTENTIAL FLAW: Read data from the console using ReadLine */
                data = Console.ReadLine();
            }
            catch (IOException exceptIO)
            {
                Console.WriteLine("Error with stream reading" + exceptIO);
            }
        }
        using (SecureString secureData = new SecureString())
        {
            for (int i = 0; i < data.Length; i++)
            {
                secureData.AppendChar(data[i]);
            }
            /* POTENTIAL FLAW: Store data directly in a file */
            File.WriteAllText(@"C:\Users\Public\WriteText.txt", secureData.ToString());
        }
    }
}

In essence, with this test case, Juliet C# showcases an issue where sensitive data is written unprotected in an unsafe location. Such kind of issues are difficult to identify with a static code analyzer because it is generally not possible to determine what is a sensitive piece of data solely based on the code semantic.

In that case, however, Juliet uses the SecureString type to store the data that is deemed sensitive. This could have interesting consequences.

SecureStrings

Stepping back to look at Microsoft’s documentation regarding the SecureString type, its general purpose and behavior can be quickly identified.

Represents text that should be kept confidential, such as by deleting it from computer memory when no longer needed. This class cannot be inherited.

The main function of SecureString objects is to store sensitive information that should be kept confidential. It implements security mechanisms to protect this information in multiple ways:

• Using unmanaged memory, the type prevents the data it contains from being moved and copied into memory in an uncontrolled way.
• Likewise, it allows its users to easily zero out and release the sensitive memory segment.
• An encryption wrapping of the sensitive information keeps it safe from reading by unexpected tiers.

SecureString (non-)deprecation

However, while the SecureString API is not deprecated, Microsoft discourages its use in new development.

We recommend that you don't use the SecureString class for new development on .NET (Core) or when you migrate existing code to .NET (Core). For more information, see SecureString shouldn't be used.

There is a lot of information in Microsoft’s documentation about why SecureString should not be used. The reasons can be summarized in a few key points:

• SecureString is unsupported at the Operating System level and by most .NET API functions. They often need to be converted back to an unsafe type before being used.
• The same is also true for SecureString construction. The source of the sensitive data is also often unprotected.
• Depending on the platform, the SecureString implementation might not protect the sensitive data at all.

Platform-specific behavior

This last statement is easily demonstrated by reading the SecureString type source code. The platform-common code calls a ProtectMemory method when initializing a SecureString.

private void Initialize(ReadOnlySpan<char> value)
        {
            _buffer = UnmanagedBuffer.Allocate(GetAlignedByteSize(value.Length));
            _decryptedLength = value.Length;

            SafeBuffer? bufferToRelease = null;
            try
            {
                Span<char> span = AcquireSpan(ref bufferToRelease);
                value.CopyTo(span);
            }
            finally
            {
                ProtectMemory();
                bufferToRelease?.DangerousRelease();
            }
        }

The Windows-specific implementation of this method uses the system-level DPAPI mechanism to efficiently encrypt the sensitive data value.

private void ProtectMemory()
        {
            if (_decryptedLength != 0 &&
                !_encrypted &&
                !Interop.Crypt32.CryptProtectMemory(_buffer, (uint)_buffer.ByteLength, Interop.Crypt32.CRYPTPROTECTMEMORY_SAME_PROCESS))
            {
            _encrypted = true;
        }

On the contrary, the Unix-specific implementation does not perform any encryption at all.

private void ProtectMemory()
        {
            _encrypted = true;
        }

Note that, contrary to Windows ones, Unix systems generally do not provide any system-level encryption mechanism, which prevents the safe implementation of the ProtectMemory function.

The SecureString type existed before .NET started supporting .NET platform. This might explain why the deprecation state is unclear.

Unprotected timespan

Because no operating system secure string structure exists, the .NET API, as well as the user code, constantly needs to protect and unprotect the SecureString-protected data. This means that the confidential data that it contains is available in clear text in the process memory from time to time. The exact frequency and timespan over which it is readable varies depending on the program’s logic.

For example, let’s execute the test program whose code was presented above and look at what the memory looks like when a piece of sensitive data is written to disk.

At that point in the execution, the SecureString value is properly protected. However, the data buffer that was used during the initialization is in clear text and can be read from the process memory. This makes the SecureString protection useless.

Microsoft documentation discourages initializing a SecureString object from a string for this exact reason.

A SecureString object should never be constructed from a String, because the sensitive data is already subject to the memory persistence consequences of the immutable String class. The best way to construct a SecureString object is from a character-at-a-time unmanaged source, such as the Console.ReadKey method.

However, even in that case, the .NET implementation is forced to decrypt the protected memory every time a character is appended to the SecureString buffer. If we go back to the test case execution and inspect the program’s memory during the addition of the last character of the secret value, we can observe that the secret appears in cleartext.

Here again, with sufficient entitlement, it is possible to access the secret value in the process memory. 

SecureString and SAST

The protection offered by SecureString objects might not be perfect or even as good as one can expect. Still, when properly used, they can add some additional security to an application. There is also no real alternative to using them. SecureString is still actively used despite Microsoft’s warning.

Discussing whether or not you should use SecureString is out of the scope of our topic. What is interesting to note is that SecureStrings are meant to store sensitive data. Seeing the type used in a piece of source code can therefore hint a SAST engine, with otherwise no understanding of an application’s business logic, about the sensitivity of a piece of data.

This makes it possible to detect Juliet’s test case with a SAST engine. The idea of tracking sensitive data usage inside a program also sounds promising and could represent a nice addition to Sonar’s engines.

Juliet C# and SecureString: it’s all about running the code

Before adding new rules and capabilities to our products, it is important to fully understand the security vulnerability the benchmark showcases here. We want to be sure to create the most precise detection logic to prevent later discomfort for our users.

However, running the test program we presented earlier leads to unexpected results. As a reminder, the test code tries to write the SecureString value into the C:\Users\Public\WriteText.txt file.

            /* POTENTIAL FLAW: Store data directly in a file */
            File.WriteAllText(@"C:\Users\Public\WriteText.txt", secureData.ToString());

However, the file that is created that way does not contain the expected sensitive data.

PS C:\Users\Public> Get-Content .\WriteText.txt
System.Security.SecureString

Instead, the fully qualified name of the SecureString type is written. This is because the SecureString type does not implement a toString method. The default Object.ToString method is therefore called which behavior is to return the fully qualified name of the type of the object.

There might have been confusion on the benchmark maintainers’ side when writing this test case. There is no sensitive information unsafely written here. Obviously, we do not want to implement such a detection behavior in our product as it would only result in false positives.

This ends our investigations on the SecureString case.

Juliet C# benchmark wrap-up

In the end, all the test cases for CWE313, CWE314, CWE315, and CWE319, which are all about sensitive data storage issues, proved to be wrong. They were all removed from the benchmark ground truth we created and excluded from our precision score computation.

Those are only an extract of all the bad test cases the Juliet C# benchmark proposed. The samples for CWE78 (OS command injection) are other examples of failed test cases. Those make a wrong assumption over the Process.start API function behavior that results in a buggy code that never runs correctly.

Nevertheless, the SecureString case proved to be inspiring. Using hints in the code to identify potentially sensitive pieces of data is a less explored capability in the SAST engines world. You can expect to see more of those confidentiality-related rules appear in the Sonar products in the future.