Manually decrypting an HTTPS request

Recently I have spent some time on learning the internals of HTTPS. I wanted to know what makes it secure and how the communication actually looks like. Today I would like to show you the steps required to decrypt a sample HTTPS request. Imagine you got a .pcap file recorded by one of your company clients who complains that your application returned 500 HTTP status code with a strange error message. The client forgot to copy the error message but luckily had a Wireshark instance running in the background (I know it’s highly hypothetical, but just close your eyes to that :)) and he/she sent you the collected traces. Let’s then assume that your server has a certificate with a public RSA key and you are in possession of its private key. Finally the client was using a slightly outdated browser which supports TLS 1.0 (though I will inform you what would have been different if it had been TLS 1.2) and does not use ephemeral keys. My main point in writing this post is to present you the steps of the TLS communication. This post is not a guidance on how to create a secure TLS configuration, but a walk-through on how this protocol works and I will purposely use less secure ciphers to make things easier to explain.

Do it quickly with Wireshark

Wireshark has a fantastic feature which can decrypt the .pcap file for you. You just need to go to Edit -> Preferences and in the dialog that appears select SSL protocol as on the image below:

As we have the private RSA key we need to add it to the Wireshark RSA key list. Click on the Edit button marked on the image and you should see a dialog similar to the one below (wit your own key definitions of course):

My sample server had address 172.28.128.3 and my key file was saved in the PEM format. Another interesting option in the SSL preferences dialog is the Wireshark SSL debug file which you may save to a text file. It reveals a lot of information about Wireshark decryption process and helped me several times when I was struggling with my own decryption tools. After closing the window, Wireshark will decrypt the TLS frames and you could happily find out what the client saw. Let’s now forget about this Wireshark feature and decrypt the .pcap file on our own.

UPDATE (2024): When TLS session uses ephemeral keys (the standard behavior), you may create a new environment variable, SSLKEYLOGFILE, and point it to a path in disk (for example, C:\temp\sslkey.log). Then restart the browser and perform the HTTPS request you want to decrypt while recording the traffic in Wireshark. Later, use the configured path as the “(Pre)-Master-Secret log filename” value and Wireshark should be able to decrypt the request. Make sure you remove this environment variable when you no longer need it. This approach will work in Firefox, Chrome, and Edge browsers and is also described in Wireshark wiki.

Manual decryption

Everything starts with a handshake

The TLS handshake is the first part of the communication. It is a process in which the client and the server agree on the encryption protocols and exchange encryption keys. When the handshake is complete client and server should possess:

• Client and Server Initialization Vectors (IV)
• Client and Server Symmetric Encryption Keys
• Client and Server MAC keys

and be ready to securely communicate.

Let’s move back to the data transmitted between our parties. Each TLS message can be represented by the structure (which is a part of the TLS record layer):

struct {
  ContentType type;
  ProtocolVersion version;
  uint16 length;
  opaque fragment[TLSPlaintext.length];
} TLSPlaintext;

where ContentType defines the type of a message (such as handshake, alert, application_data etc.) and ProtocolVersion describes the TLS version in use (where TLS1.0 is represented as 0x0301 and TLS1.2 is 0x0303). TLS is an onion protocol and next layers (handshake, application data etc.) are embedded in the record layer.

ClientHello

The handshake starts with a ClientHello message sent from the client to the server (the output is taken from Wireshark):

Internet Protocol Version 4, Src: 172.28.128.1, Dst: 172.28.128.3
Transmission Control Protocol, Src Port: 53835 (53835), Dst Port: 443 (443), Seq: 1, Ack: 1, Len: 87
Secure Sockets Layer
    TLSv1 Record Layer: Handshake Protocol: Client Hello
        Content Type: Handshake (22)
        Version: TLS 1.0 (0x0301)
        Length: 82
        Handshake Protocol: Client Hello
            Handshake Type: Client Hello (1)
            Length: 78
            Version: TLS 1.0 (0x0301)
            Random
                GMT Unix Time: Mar  4, 2016 13:20:45.000000000 Central European Standard Time
                Random Bytes: bec6ea9223c15fd63dfd22249013f234aa04edcc308af6c5...
            Session ID Length: 0
            Cipher Suites Length: 10
            Cipher Suites (5 suites)
                Cipher Suite: TLS_RSA_WITH_AES_256_CBC_SHA (0x0035)
                Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA (0x002f)
                Cipher Suite: TLS_RSA_WITH_3DES_EDE_CBC_SHA (0x000a)
                Cipher Suite: TLS_RSA_WITH_RC4_128_SHA (0x0005)
                Cipher Suite: TLS_RSA_WITH_RC4_128_MD5 (0x0004)
            Compression Methods Length: 1
            Compression Methods (1 method)
                Compression Method: null (0)
            Extensions Length: 27
            Extension: server_name
            Extension: Extended Master Secret
            Extension: renegotiation_info

Just after the TLS header comes the payload which most important parts are: client random (32 bytes composed of the Unix time and 28 bytes of random bytes) and cipher suites supported by the client and extensions understood by the client. The full client random looks as follows (we will need this value later):

56 d9 7d 9d be c6 ea 92 23 c1 5f d6 3d fd 22 24
90 13 f2 34 aa 04 ed cc 30 8a f6 c5 09 f8 60 a5

TLS also supports compression, but it’s rarely used.

ServerHello, Certificate, ServerHelloDone

In response to the ClientHello server sends ServerHello, Certificate(s) and ServerHelloDone messages:

Internet Protocol Version 4, Src: 172.28.128.3, Dst: 172.28.128.1
Transmission Control Protocol, Src Port: 443 (443), Dst Port: 53835 (53835), Seq: 1, Ack: 88, Len: 950
Secure Sockets Layer
    TLSv1 Record Layer: Handshake Protocol: Server Hello
        Content Type: Handshake (22)
        Version: TLS 1.0 (0x0301)
        Length: 81
        Handshake Protocol: Server Hello
            Handshake Type: Server Hello (2)
            Length: 77
            Version: TLS 1.0 (0x0301)
            Random
                GMT Unix Time: Nov 26, 1981 02:33:35.000000000 Central European Standard Time
                Random Bytes: 2c1406b273a5e376a6254a818606d4759ab7efb797a1e559...
            Session ID Length: 32
            Session ID: f739bcc4a4ef4bb190115cc9705feaa9feaeffd6464b196f...
            Cipher Suite: TLS_RSA_WITH_AES_256_CBC_SHA (0x0035)
            Compression Method: null (0)
            Extensions Length: 5
            Extension: renegotiation_info
    TLSv1 Record Layer: Handshake Protocol: Certificate
        Content Type: Handshake (22)
        Version: TLS 1.0 (0x0301)
        Length: 850
        Handshake Protocol: Certificate
            Handshake Type: Certificate (11)
            Length: 846
            Certificates Length: 843
            Certificates (843 bytes)
                Certificate Length: 840
                Certificate: 308203443082022c020900a7a6bfffdfc4e090300d06092a... (id-at-countryName=PL,id-at-localityName=Warsaw,id-at-organizationName=MyCompany,pkcs-9-at-emailAddress=test@test.com,id-at-commonName=opensslvm)
    TLSv1 Record Layer: Handshake Protocol: Server Hello Done
        Content Type: Handshake (22)
        Version: TLS 1.0 (0x0301)
        Length: 4
        Handshake Protocol: Server Hello Done
            Handshake Type: Server Hello Done (14)
            Length: 0

ServerHello contains server random (also 32 bytes):

16 62 fe 6f 2c 14 06 b2 73 a5 e3 76 a6 25 4a 81
86 06 d4 75 9a b7 ef b7 97 a1 e5 59 f4 dd 04 7f

and the strongest cipher suite supported by both the client and the server – in our case it’s TLS_RSA_WITH_AES_256_CBC_SHA and extensions supported by both the client and the server.

Next server response part of the handshake is the Certificate message. My test server was using a self-signed certificate so only this certificate was sent back in the handshake. But it’s common that servers send a chain of certificates so the client can verify the highest certificate signature against its own cert store and does not need to perform any additional requests to certificate providers.

Finally server sends ServerHelloDone to communicate that it finished sending hello messages.

Client Key Exchange, Change Cipher Spec, Encrypted Handshake Message

Here comes the most important part of the handshake, i.e. the key exchange. The client generates a cryptographically random pre-master key, encrypts it with the public RSA key and sends it to the server. Just after the Client Key Exchange message the client sends the Change Cipher Spec message and the encrypted Client Finished (we will have a look at it in a moment):

Internet Protocol Version 4, Src: 172.28.128.1, Dst: 172.28.128.3
Transmission Control Protocol, Src Port: 53835 (53835), Dst Port: 443 (443), Seq: 88, Ack: 951, Len: 326
Secure Sockets Layer
    TLSv1 Record Layer: Handshake Protocol: Client Key Exchange
        Content Type: Handshake (22)
        Version: TLS 1.0 (0x0301)
        Length: 262
        Handshake Protocol: Client Key Exchange
            Handshake Type: Client Key Exchange (16)
            Length: 258
            RSA Encrypted PreMaster Secret
                Encrypted PreMaster length: 256
                Encrypted PreMaster: 0b3366f120198ce3839e0c5b106462da2b88105f3d9f2fe9...
    TLSv1 Record Layer: Change Cipher Spec Protocol: Change Cipher Spec
        Content Type: Change Cipher Spec (20)
        Version: TLS 1.0 (0x0301)
        Length: 1
        Change Cipher Spec Message
    TLSv1 Record Layer: Handshake Protocol: Encrypted Handshake Message
        Content Type: Handshake (22)
        Version: TLS 1.0 (0x0301)
        Length: 48
        Handshake Protocol: Encrypted Handshake Message

As we are in possession of the RSA private key corresponding to the public key used in this handshake it’s time to use it. First we need to save the Encrypted PreMaster (256 bytes = 2048 bits) to a premaster-encrypted.bin file. Then we may use OpenSSL to decrypt it:

PS> openssl pkeyutl -decrypt -in .\premaster-encrypted.bin -inkey .\key.pem -out premaster-decrypted.bin

PS> Format-Hex .\premaster-decrypted.bin

           00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F

00000000   03 01 82 BB 54 5E 61 A2 7C D2 B0 BD 59 9E E8 6A  ..?»T^ac|O°1Y?cj
00000010   41 9C 63 0C 4B B9 E1 25 DB 9F 9E 08 F4 D4 41 F6  A?c.K1á%U??.ôÔAö
00000020   1F 3B 1E C1 3E 88 BA DD 38 AE E3 73 E7 88 FC AB  .;.Á>?oÝ8Rasç?ü«

Note that if the keys were exchanged using a variant of a Diffie-Hellman algorithm (which is a very common and recommended practice) we won’t be able to generate the master secret without additional information (DH params).

Now, both client and server have material to create a master secret which will be later used in the symmetric encryption of the communication. In TLS 1.0 the master secret is generated using a PRF (Pseudo-Random Function) which uses both MD5 and SHA1. As you would see the same PRF function is also used to derive other communication parameters from the master secret. Note that TLS1.2 PRF function is based on SHA-256 or stronger hash (details can be found in RFC5246), but it is used in the same contexts as in TLS1.0. I created a sample application PRF.exe which you may download here. To create a master secret run:

PS> PRF.exe --label "master secret" --length 48 --secret .\premaster-decrypted.bin --data .\random_cs.bin  --output master-secret.bin
PS> Format-Hex .\master-secret.bin

           00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F

00000000   27 90 3D 7D 3F 40 B9 0E 64 84 40 87 A5 8D FF F0  '?=}?@1.d?@?Y?.?
00000010   1A 1D D7 6C 96 81 8E F9 62 4F 38 AF 38 60 9E 52  ..×l???ubO8─8`?R
00000020   D3 18 0F E8 07 3D 4A 81 EC 28 26 44 22 66 C8 CF  Ó..c.=J?i(&&D&"fEI

where random_cs.bin is a concatenation of the client random and the server random (in this order!). However, this way of calculating the master secret has a pitfall, which allows an active attacker to synchronize two TLS sessions so that they share the same “master_secret”. A remedy to this problem was described in RFC 7627 which recommends using the hash of all the previous handshake messages as a seed to the PRF function (instead of the client and server randoms). It’s more and more common, yet our example client does not use it.

With the master secret generated we may produce the other parameters required for the secure communication. In our case we will need:

• 20 bytes for a client MAC key (SHA1)
• 20 bytes for a server MAC key (SHA1)
• 32 bytes for a client encryption key (AES256)
• 32 bytes for a server encryption key (AES256)
• 16 bytes for a client IV (AES uses 128-bit blocks)
• 16 bytes for a server IV (AES uses 128-bit blocks)

This all sums up to 136 bytes which we will derive from our master secret using the PRF function:

PS> prf --secret .\master-secret.bin --label "key expansion&amp" --data .\random_sc.bin -n 136 -o key-expansion.bin

PS> Format-Hex .\key-expansion.bin

           00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F

00000000   13 E2 39 F0 75 02 5B F9 64 4F 58 27 5A FF 36 D9  .â9?u.[udOX'Z.6U
00000010   BB C1 6C 79 CA B0 4B 1F 91 94 73 F0 BF E9 94 37  »ÁlyE°K.??s??é?7
00000020   99 FE C5 EC FB D0 F8 72 1D 00 7E 17 B3 A4 54 83  ??Aiu?or..~.3¤T?
00000030   AA 92 1A D9 DE 06 CC 6B 47 57 18 69 CF 63 92 19  a?.U?.IkGW.iIc?.
00000040   C9 05 29 70 16 3B B8 A2 11 38 77 BB 10 53 8E 76  É.)p.;¸c.8w».S?v
00000050   B5 C9 EC 7B E6 83 F3 DA 8F 04 48 C2 EF C9 C5 B9  uÉi{a?óÚ?.HÂiÉA1
00000060   86 58 B5 72 1A B8 BA CE 3E 0F F5 34 87 FE FF 6B  ?Xur.¸oÎ>.o4??.k
00000070   73 60 BC FE 7B 08 48 D0 2D 85 BE 50 45 58 51 59  s`1?{.H?-?3PEXQY
00000080   8E 63 33 8A CF 38 A0 B0 2D 85 BE 50 45 58 51 59  ?c3?I8 °-?3PEXQY

Notice that we are now using random_sc.bin which is a concatenation of the server random and the client random (notice the order is reversed compared to the previous call!). As you remember, the next message sent from the client to the server was Changed Cipher Spec which indicates that from now on all the sent data will be encrypted using the negotiated keys. Thus the Client Finished (the last message from the last Wireshark output) is an encrypted message. Let’s decrypt it:

PS>  openssl enc -aes-256-cbc -d -K 1D007E17B3A45483AA921AD9DE06CC6B47571869CF639219C9052970163BB8A2 -iv 3E0FF53487FEFF6B7360BCFE7B0848D0 -in .\handshake-client-finished.bin -out .\handshake-client-finished-decrypted.bin -nopad

PS>; Format-Hex .\handshake-client-finished-decrypted.bin

           00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F

00000000   14 00 00 0C 9E B1 9A 29 27 A2 FA F8 F1 22 4B 46  ....?+?)'cúon"KF
00000010   07 94 C6 EE B0 8A CC E5 C1 AB 88 F4 49 34 19 45  .?Aî°?IaÁ«?ôI4.E
00000020   00 DF 4C 39 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B 0B  .ßL9............

Starting from the end: the last 12 bytes are padding bytes, the next 20 bytes are a message HMAC. The algorithm for calculating message HMAC is quite complicated: you need to first append an 8 byte long message sequence number, then add TLS record header with the content size set to the actual content size (without HMAC size). The message sequence number is a counter (separate for the client and the server) which is incremented after each encrypted message is sent (Change Cipher Spec message sets its value to zero). In our case the HMAC calculation process looks as follows:

PS> Format-Hex .\handshake-client-finished-without-mac.bin

           00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F

00000000   00 00 00 00 00 00 00 00 16 03 01 00 10 14 00 00  ................
00000010   0C 9E B1 9A 29 27 A2 FA F8 F1 22 4B 46 14 00 00  .?+?)'cúon"KF...

PS> Format-Hex .\client-mac-key.bin

           00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F

00000000   13 E2 39 F0 75 02 5B F9 64 4F 58 27 5A FF 36 D9  .â9?u.[udOX'Z.6U
00000010   BB C1 6C 79 75 02 5B F9 64 4F 58 27 5A FF 36 D9  »Ályu.[udOX'Z.6U

PS> openssl dgst -sha1 -hmac $(cat .\client-mac-key.bin) .\handshake-client-finished-without-mac.bin
HMAC-SHA1(.\handshake-client-finished-without-mac.bin)= 0794c6eeb08acce5c1ab88f44934194500df4c39

The next 12 bytes are the PRF output of the MD5 and SHA1 hashes (again for TLS1.2 stronger hash will be used here) of all the previously exchanged handshake messages (without the TLS record layer part, without the Change Cipher Spec message and all eventual alert messages). The hashes are concatenated and used as a seed to the PRF function (the client-data-for-prf.bin file):

PS> openssl dgst -md5 .\client-before-finished-sent.bin
MD5(.\client-before-finished-sent.bin)= b22d98c8c97a236bc83b2eb5880e03bb
PS> openssl dgst -sha1 .\client-before-finished-sent.bin
SHA1(.\client-before-finished-sent.bin)= 83e2d22480e9cb0c4b5e05f882602deed99274b5
PS> prf -l "client finished" -s .\master-secret.bin -d .\client-data-for-prf.bin -o client-prf-result.bin -n 12
PS> Format-Hex .\client-prf-result.bin

           00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F

00000000   9E B1 9A 29 27 A2 FA F8 F1 22 4B 46 00 00 00 00  ?+?)'cúon"KF....

Change Cipher Spec, Encrypted Handshake Message

After decrypting the Client Finished message, the server responds with the Change Cipher Spec message and later the Server Finished message. The algorithm for generating the Server Finished message is actually almost the same as for the Client Finished message. The message to be hashed will contain the Client Finished message (decrypted, without HMAC and without padding) and the whole message will be HMACed and encrypted with the server variants of the keys. Now, the handshake is finally done and we can start sending the actual data.

Gimme some data

In our case the client sent a simple HTTP request:

Internet Protocol Version 4, Src: 172.28.128.1, Dst: 172.28.128.3
Transmission Control Protocol, Src Port: 53835 (53835), Dst Port: 443 (443), Seq: 414, Ack: 1010, Len: 85
Secure Sockets Layer
    TLSv1 Record Layer: Application Data Protocol: http
        Content Type: Application Data (23)
        Version: TLS 1.0 (0x0301)
        Length: 80
        Encrypted Application Data: 751f29ecf5227fbc89b25bb5ccfed9060db5506c0da9e42b...

Let’s decrypt it using the client key (an IV we will be the last block of the last message sent by the client as we use Cipher Block Chaining):

PS> openssl enc -aes-256-cbc -d -K 1D007E17B3A45483AA921AD9DE06CC6B47571869CF639219C9052970163BB8A2 -iv 0E03400DBE4A33B2D591960526AE0395 -in .\client-request.bin -out .\client-request-decrypted.bin

PS> cat .\client-request-decrypted.bin
GET / HTTP/1.1
Host: opensslvm
Connection: close

ýÎ↓¶čU,Ăâ-kťvľúęmĹ%♣

The last unprintable characters are of course HMAC of the message, but as you can see we managed to decrypt the message sent by the client. When the server responds we would use the server key to decrypt the response and so on.

The decryption process will be slightly different if we were using TLS1.2 as in TLS1.2 a unique IV is appended to each message.

Connection close

To shutdown a TLS connection one of the parties needs to send a special shutdown alert (01 00):

Internet Protocol Version 4, Src: 172.28.128.1, Dst: 172.28.128.3
Transmission Control Protocol, Src Port: 53835 (53835), Dst Port: 443 (443), Seq: 499, Ack: 2477, Len: 37
Secure Sockets Layer
    TLSv1 Record Layer: Encrypted Alert
        Content Type: Alert (21)
        Version: TLS 1.0 (0x0301)
        Length: 32
        Alert Message: Encrypted Alert

After decryption:

PS> openssl enc -aes-256-cbc -d -K 1D007E17B3A45483AA921AD9DE06CC6B47571869CF639219C9052970163BB8A2 -iv DAB4C0D0C65CB65DE8D678FD61369811 -in .\client-shutdown.bin -out .\client-shutdown-decrypted.bin -nopad

PS> Format-Hex .\client-shutdown-decrypted.bin

           00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F

00000000   01 00 DF 06 95 88 29 58 C3 7D 78 D5 21 DA 78 7F  ..ß.??)XA}xO!Úx⌂
00000010   A4 CD 25 2B 99 02 09 09 09 09 09 09 09 09 09 09  ¤Í%+?...........

Then the client might close the TCP connection too or continue the communication in plain text.

Miscellaneous

There is a lot more to write about TLS. It is amazing how much you may learn by simply studying the history of this protocol. You may observe how new extensions were and are added, how the protocol evolved and still evolves. I am actually still reading the RFCs. Quite a good description of the protocol (and a lot of links) can be also found on Wikipedia.

Setup your own lab

If you would like to experiment on your own and repeat all the steps described in this post (but with your own keys) I will show you the required steps in this paragraph.

You will need a private RSA key and a certificate:

openssl genpkey -algorithm RSA -out key.pem -pkeyopt rsa_keygen_bits:2048

openssl req -new -x509 -days 365 -key key.pem -out localhost.crt

The second command might require you to answer several questions unless you have the needed sections in the OpenSSL configuration file, for example:

...

[req]
prompt = no
distinguished_name = dn
req_extensions = ext

[dn]
CN = localhost
emailAddress = test@test.com
O = MyCompany
L = Warsaw
C = PL
[ext]
subjectAltName = DNS:localhost

...

Now you are ready to run a sample OpenSSL server:

PS> openssl s_server -cert .\certs\localhost.crt -key .\key.pem -cipher "AES256-SHA" -www

We are limiting the server to use only the RSA handshake and in response always return TLS session parameters. Additionally, with flags: -ssl2, -ssl3, -tls1, -tls1_1, -tls1_2, -no_ssl2, -no_ssl3, -no_tls1, -no_tls1_1, -no_tls1_2 we may configure which version of the TLS our test server will support. Now you may run Wireshark or TCPDump or any other sniffer to register the network traffic between a browser and the test server.