TLS 1.2, AES-GCM and .NET network trace


The next exercise on our path to better understand TLS will be a decryption of a network trace collected from a .NET console application. In the last post we examined a simple TLS 1.0 session. Today I would like to focus on the latest version (1.2) of the TLS protocol. The changes introduced by this version (defined in RFC 5246) included: support for authenticated encryption, PRF simplification and removal of all hard-coded security primitives.

Preparations

Our sample .NET client looks as follows:

using System;
using System.IO;
using System.Net;
using System.Net.Http;
using System.Net.Security;
using System.Threading.Tasks;

namespace SimpleHttpsClient
{
    class Program
    {
        static void Main(string[] args)
        {
            if (args.Length != 1) {
                Console.WriteLine("Please provide a site url which you would like to get.");
                return;
            }

            LoadPage(args[0]).Wait();

            Console.WriteLine("Press any key to continue...");
            Console.ReadKey();
        }

        static async Task<string> LoadPage(string url)
        {
            var handler = new WebRequestHandler();

            using (var client = new HttpClient(handler)) {
                var resp = await client.GetAsync(url);
                var respText = await resp.Content.ReadAsStringAsync();

                Console.WriteLine(respText);

                return respText;
            }
        }
    }
}

As we will be analysing .NET network trace, we first need to turn tracing on in the application configuration file:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <startup>
    <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.6.1"/>
  </startup>

  <system.diagnostics>
    <trace autoflush="true"/>
    <sharedListeners>
      <add name="file" initializeData="C:\logs\network.log" type="System.Diagnostics.TextWriterTraceListener"/>
    </sharedListeners>
    <sources>
      <source name="System.Net.Http" switchValue="Verbose">
        <listeners>
          <add name="file"/>
        </listeners>
      </source>
      <source name="System.Net" switchValue="Verbose">
        <listeners>
          <add name="file"/>
        </listeners>
      </source>
      <source name="System.Net.Sockets" switchValue="Verbose" maxdatasize="102400">
        <listeners>
          <add name="file"/>
        </listeners>
      </source>
    </sources>
  </system.diagnostics>
</configuration>

Notice I increased the default logged size of the network packet to 100KB (default is 1KB) so all the transmitted data will be saved to the trace file. The last thing to set up is a test OpenSSL server:

> openssl s_server -cert .\certs\localhost.crt -key .\certs\key.pem -www

If you are using a self-signed certificate, either add it to the Trusted People store or turn off certificate validation in the client.

Finally we may run the first request to our test server:

> SimpleHttpsClient https://localhost:4433

...stripped...

---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID:
    Session-ID-ctx: 01000000
    Master-Key: 04A4DFB8197A978019295D61B6C6F6AD45D2E4974E96567BDDA00845E0C28922645AD36AEF52C737E3341E54CF2212E8
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1462735950
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

...stripped...

Press any key to continue...

Analysing the network trace

If everyhing goes as expected we should have a new text file created under the c:\logs folder. In my case the file starts with the following lines:

System.Net.Http Verbose: 0 : [1740] WebRequestHandler#45004109::.ctor()
System.Net.Http Verbose: 0 : [1740] Exiting WebRequestHandler#45004109::.ctor()
System.Net.Http Verbose: 0 : [1740] HttpClient#2383799::.ctor(WebRequestHandler#45004109)
System.Net.Http Information: 0 : [1740] Associating HttpClient#2383799 with WebRequestHandler#45004109
System.Net.Http Verbose: 0 : [1740] Exiting HttpClient#2383799::.ctor()
System.Net.Http Verbose: 0 : [1740] HttpClient#2383799::.ctor(WebRequestHandler#45004109)
System.Net.Http Verbose: 0 : [1740] Exiting HttpClient#2383799::.ctor()
System.Net.Http Verbose: 0 : [1740] HttpRequestMessage#21454193::.ctor(Method: GET, Uri: 'https://localhost:4433/')
System.Net.Http Verbose: 0 : [1740] Exiting HttpRequestMessage#21454193::.ctor()
System.Net.Http Verbose: 0 : [1740] HttpClient#2383799::SendAsync(HttpRequestMessage#21454193: Method: GET, RequestUri: 'https://localhost:4433/', Version: 1.1, Content: <null>, Headers:
{
})
...

The System.Net.Sockets trace source is the most verbose one, logging the content of the network packets sent and received over the wire and in our analysis will be most useful. Other trace sources provide high-level info about what type of network activity our application is doing (such as DNS resolution, initializing security context, sending HTTP requests). As you probably remember TLS starts with a handshake, after which both the server and the client possess all the necessary parameters to communicate securely. I won’t go into details of the handshake phase (you may read the description in RFC or in my previous post), but rather focus on the System.Net logs which mark the important phases. We start with the ClientHello message:

System.Net Information: 0 : [6104] SecureChannel#36963566::.ctor(hostname=localhost, #clientCertificates=0, encryptionPolicy=RequireEncryption)
System.Net Information: 0 : [6104] Enumerating security packages:
System.Net Information: 0 : [6104]     Negotiate
System.Net Information: 0 : [6104]     NegoExtender
System.Net Information: 0 : [6104]     Kerberos
System.Net Information: 0 : [6104]     NTLM
System.Net Information: 0 : [6104]     TSSSP
System.Net Information: 0 : [6104]     pku2u
System.Net Information: 0 : [6104]     WDigest
System.Net Information: 0 : [6104]     Schannel
System.Net Information: 0 : [6104]     Microsoft Unified Security Protocol Provider
System.Net Information: 0 : [6104]     CREDSSP
System.Net Information: 0 : [6104] SecureChannel#36963566 - Left with 0 client certificates to choose from.
System.Net Information: 0 : [6104] AcquireCredentialsHandle(package = Microsoft Unified Security Protocol Provider, intent  = Outbound, scc     = System.Net.SecureCredential)
System.Net Information: 0 : [6104] InitializeSecurityContext(credential = System.Net.SafeFreeCredential_SECURITY, context = (null), targetName = localhost, inFlags = ReplayDetect, SequenceDetect, Confidentiality, AllocateMemory, InitManualCredValidation)
System.Net Information: 0 : [6104] InitializeSecurityContext(In-Buffer length=0, Out-Buffer length=169, returned code=ContinueNeeded).

.NET under the hood is using the Schannel system library. During the handshake phase the InitilizeSecurityContext method is called each time our application receives or sends data from or to the server. The method’s return code indicates whether the handshake requires some more steps or we can start sending application specific data (as you can see in the trace snippet we are not done yet: returned code=ContinueNeeded). At some point we should see in the log something similar to the snippet below:

System.Net Information: 0 : [6104] InitializeSecurityContext(In-Buffers count=2, Out-Buffer length=0, returned code=OK).
System.Net Information: 0 : [6104] Remote certificate: [Version]
...stripped...
System.Net Information: 0 : [6104] EndProcessAuthentication(Protocol=Tls12, Cipher=Aes256 256 bit strength, Hash=32781 0 bit strength, Key Exchange=44550 256 bit strength).

All messages starting from this point should be encrypted application data. Let’s try to decrypt the first one.

Decryption

The message we will try to decrypt is:

System.Net.Sockets Verbose: 0 : [3696] 00000000 : 17 03 03 00 58 00 00 00-00 00 00 00 01 50 C9 11 : ....X........P..
System.Net.Sockets Verbose: 0 : [3696] 00000010 : 63 F3 E9 FC 2B DB FE 4F-FA 96 FF 6A AA 27 65 0F : c...+..O...j.'e.
System.Net.Sockets Verbose: 0 : [3696] 00000020 : 34 54 5F A8 03 82 36 0D-85 8B EB 4A A1 5D 68 5E : 4T_...6....J.]h^
System.Net.Sockets Verbose: 0 : [3696] 00000030 : AF 1A C0 C2 77 06 EA 76-60 D5 4A 54 A2 F8 60 1E : ....w..v`.JT..`.
System.Net.Sockets Verbose: 0 : [3696] 00000040 : A5 4A 3E 0E 14 3F 84 68-0B 63 FC 36 7D 94 DD 5E : .J>..?.h.c.6}..^
System.Net.Sockets Verbose: 0 : [3696] 00000050 : 4C 63 1A F9 29 24 13 22-01 93 69 92 B1          :

First five bytes are the TLS Record header:

  • 0x17 (23) – record type = ApplicationData
  • 0x0303 – record version = TLS1.2
  • 0x0058 – record payload length

From the application output we know that the cipher in use is AES-GCM (ECDHE-RSA-AES256-GCM-SHA384). AES-GCM is one of the authenticated symmetric encryption algorithms added to TLS 1.2. Apart from data encryption, it also ensures data integrity. To successfully decrypt the message we need:

  • Client Write Key
  • Nonce
  • Authenticated data

As we remember client and server keys are derived from the master secret using the PRF function (we know the master secret from the application console output). In TLS 1.0 the PRF function was quite complicated and combined both MD5 and SHA1 hashes outputs to produce the required amount of bytes. TLS 1.2 makes this process simpler and uses output of only one hash function:

      P_hash(secret, seed) = HMAC_hash(secret, A(1) + seed) +
                             HMAC_hash(secret, A(2) + seed) +
                             HMAC_hash(secret, A(3) + seed) + ...

   where + indicates concatenation.

   A() is defined as:

      A(0) = seed
      A(i) = HMAC_hash(secret, A(i-1))

As a result of the PRF function we need 96 bytes of data (32*2 for symmetric keys + 16*2 for IVs):

> PS ssl> .\PRF.exe --tls=TLS1.2 --sha=384 --secret=.\master_secret.bin --label="key expansion" --length=96 --output=.\key_expansion.bin --data=.\random_sc.bin

There is only one element we are still missing to run the above command: random_sc.bin – a concatenation of the server and client randoms. We need to extract them from the server and client hello messages (the randoms are marked in yellow on the attached images).

For client:

System.Net.Sockets Verbose: 0 : [6104] Socket#48209832::BeginSend()
System.Net.Sockets Verbose: 0 : [6104] Exiting Socket#48209832::BeginSend() 	-> OverlappedAsyncResult#25474675
System.Net.Sockets Verbose: 0 : [3696] Data from Socket#48209832::PostCompletion
System.Net.Sockets Verbose: 0 : [3696] 00000000 : 16 03 03 00 A4 01 00 00-A0 03 03 57 2F 94 4E 06 : ...........W/.N.
System.Net.Sockets Verbose: 0 : [3696] 00000010 : 5B 46 15 BD 53 2C F4 40-E4 E2 3A 8D AF 52 DA 81 : [F..S,.@..:..R..
System.Net.Sockets Verbose: 0 : [3696] 00000020 : 3E A5 C5 8B 82 D5 D8 32-DB CA 82 00 00 34 C0 2C : >......2.....4.,
System.Net.Sockets Verbose: 0 : [3696] 00000030 : C0 2B C0 30 C0 2F 00 9F-00 9E C0 24 C0 23 C0 28 : .+.0./.....$.#.(
System.Net.Sockets Verbose: 0 : [3696] 00000040 : C0 27 C0 0A C0 09 C0 14-C0 13 00 9D 00 9C 00 3D : .'.............=

client_random

and server:

System.Net.Sockets Verbose: 0 : [3696] Exiting Socket#48209832::BeginReceive() 	-> OverlappedAsyncResult#62476613
System.Net.Sockets Verbose: 0 : [6104] Data from Socket#48209832::PostCompletion
System.Net.Sockets Verbose: 0 : [6104] 00000000 : 16 03 03 00 3D                                  : ....=
System.Net.Sockets Verbose: 0 : [6104] Socket#48209832::EndReceive(OverlappedAsyncResult#62476613)
System.Net.Sockets Verbose: 0 : [6104] Exiting Socket#48209832::EndReceive() 	-> Int32#5
System.Net.Sockets Verbose: 0 : [6104] Socket#48209832::BeginReceive()
System.Net.Sockets Verbose: 0 : [6104] Exiting Socket#48209832::BeginReceive() 	-> OverlappedAsyncResult#3038911
System.Net.Sockets Verbose: 0 : [3696] Data from Socket#48209832::PostCompletion
System.Net.Sockets Verbose: 0 : [3696] 00000000 : 02 00 00 39 03 03 ED D7-85 57 95 67 AC 3E 82 FB : ...9.....W.g.>..
System.Net.Sockets Verbose: 0 : [3696] 00000010 : C1 F6 17 20 02 9B D8 80-E3 3F 60 71 19 18 0D 98 : ... .....?`q....
System.Net.Sockets Verbose: 0 : [3696] 00000020 : BA 75 72 DF C9 9B 00 C0-30 00 00 11 FF 01 00 01 : .ur.....0.......
System.Net.Sockets Verbose: 0 : [3696] 00000030 : 00 00 0B 00 04 03 00 01-02 00 23 00 00          : ..........#..

server_random

We may now run the PRF command and split the output:

key-expansion

Next required parameter for decryption on our list is nonce. Nonce is twelve bytes long and is composed of two parts:

  • fixed – four-bytes long (marked in red on the image)
  • dynamic (counter) – eight-bytes long, which is sent in the plain form in the message (just after the TLSRecord header), in our case: 00 00 00 00 00 00 00 01

Finally, the additional authenticated data is a concatenation of seq_num, TLSCompressed.type, TLSCompressed.version and TLSCompressed.length (length of the unencrypted message!). In our case:

00 00 00 00 00 00 00 01 17 03 03 00 40

You may be wondering how I guessed the original message length – in AES-GCM it’s simple: you just need to subtract MAC size (0x10) from the encrypted message length (0x50).

We are now ready to decrypt our message 🙂 I wrote a unit test for this purpose (using the AES-GCM implementation from the BouncyCastle library):

[Fact]
public void TestSampleEncryptedMessage()
{
    var cipherBytes = new byte[] {
        0x50, 0xC9, 0x11, 0x63, 0xF3, 0xE9, 0xFC, 0x2B, 0xDB, 0xFE, 0x4F, 0xFA, 0x96, 0xFF, 0x6A, 0xAA,
        0x27, 0x65, 0x0F, 0x34, 0x54, 0x5F, 0xA8, 0x03, 0x82, 0x36, 0x0D, 0x85, 0x8B, 0xEB, 0x4A, 0xA1,
        0x5D, 0x68, 0x5E, 0xAF, 0x1A, 0xC0, 0xC2, 0x77, 0x06, 0xEA, 0x76, 0x60, 0xD5, 0x4A, 0x54, 0xA2,
        0xF8, 0x60, 0x1E, 0xA5, 0x4A, 0x3E, 0x0E, 0x14, 0x3F, 0x84, 0x68, 0x0B, 0x63, 0xFC, 0x36, 0x7D,
        0x94, 0xDD, 0x5E, 0x4C, 0x63, 0x1A, 0xF9, 0x29, 0x24, 0x13, 0x22, 0x01, 0x93, 0x69, 0x92, 0xB1
    };

    var keyBytes = new byte[] {
        0xC5, 0x2F, 0xB2, 0x44, 0xA8, 0xAD, 0x72, 0x1F, 0xB4, 0xDB, 0x9D, 0x4B, 0x2A, 0x97, 0x05, 0xDA,
        0x06, 0x72, 0x3A, 0x7A, 0x74, 0x91, 0x6E, 0xE8, 0x9E, 0x2D, 0x40, 0x2A, 0x02, 0x04, 0x45, 0x9B
    };

    var nonceBytes = new byte[] { 0x29, 0xAB, 0x12, 0x92, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01 };

    var additionalDataBytes = new byte[] {
        0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x17, 0x03, 0x03, 0x00, 0x40
    };

    var aesGcm = new GcmBlockCipher(new AesFastEngine());
    var key = new KeyParameter(keyBytes);
    var aedParameters = new AeadParameters(key, 16 * 8, nonceBytes, additionalDataBytes);
    aesGcm.Init(false, aedParameters);

    var outputBytes = new byte[aesGcm.GetOutputSize(cipherBytes.Length)];
    int len = aesGcm.ProcessBytes(cipherBytes, 0, cipherBytes.Length, outputBytes, 0);

    // final MAC check
    len += aesGcm.DoFinal(outputBytes, len);

    string originalRequest = Encoding.UTF8.GetString(outputBytes);

    Assert.Equal("GET / HTTP/1.1\r\nHost: localhost:4433\r\nConnection: Keep-Alive\r\n\r\n", originalRequest);
}

2 thoughts on “TLS 1.2, AES-GCM and .NET network trace

  1. I’ve read couple of times that mastering system.diagnostics makes any other loggers obsolete, but never seen any real advantage of using it. But from Your use case I am blown away how much informations can be retrieved using it.

    1. I’m glad to hear that 🙂 And yes, those logs are not commonly used but quite often might be a good alternative to Wireshark (or Message Analyzer) if we set the trace listener to something more performant, such as EventProviderTraceListener.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s