Extracting Service Principal Credentials in VSTS


When we need to deploy an application to Azure from VSTS (Visual Studio Team Services), we use the Azure tasks prepared by Microsoft. These tasks require a contributor account in Azure AD to make changes to your subscription. As this account is not a regular user account but an application account we call it a Service Principal. A very basic build pipeline might look as follows:

vsts-appservice-buildpipeline

The “Azure App Service Deploy” task is an example of a task that will use a Service Principal account to update your App Service in Azure. VSTS makes it easy to create the Service Principal account; it also automatically assigns a contributor role in your subscription to this newly created account. When you want to have full control over your Azure AD you may manually create an App Registration (another name for the Service Principal) in the portal and give it the required rights. You will also need a key to authenticate the service in Azure:

vsts-appregistration

In the next step, you create a new Azure Resource Manager Service Endpoint, providing all the collected information:

azure-appservice-registration

Once you add the Service Endpoint, you won’t be able to modify or retrieve its credentials. This might be problematic if you don’t have access to the Azure AD, and you need to create a Service Endpoint for a different subscription but using the same Service Principal account. In this post, I will present you a way to get hold of the Service Principal credentials using the build pipeline only. To make the things harder, we will use the Hosted Agent – one provided by Microsoft, with no access through RDP.

A Memory Dump in the Build Pipeline

The first place we usually check when looking for process secret data is the process memory. Here it will be no different. As we don’t have direct access to the target system (and the process), we will use the build pipeline to download procdump, collect a dump, and publish it as a built artifact. We also need to have an Azure task in our pipeline (otherwise VSTS won’t use any Service Principal credentials). The Azure Powershell task fulfills all our requirements. It runs in a context of a Service Principal and executes provided PowerShell code. Our PowerShell script is concise:

Invoke-WebRequest -UseBasicParsing `
	-Uri https://live.sysinternals.com/procdump.exe `
	-OutFile $env:BUILD_SOURCESDIRECTORY\procdump.exe
 
& "$env:BUILD_SOURCESDIRECTORY\procdump.exe" -accepteula `
	-o -ma  $pid $env:BUILD_SOURCESDIRECTORY\ps.dmp

The BUILD_SOURCESDIRECTORY is a special environment variable which VSTS sets to the path of the directory containing application source code. Our build pipeline in VSTS might look as follows:

vsts-procdump-pipeline

Below, you may see a VSTS log of the PowerShell task execution. The highlighted line is the one when VSTS creates the Azure context (using the Add-AzureRMAccount cmdlet).

##[section]Starting: Azure PowerShell script: InlineScript 2017-09-21T22:11:20.8230853Z 
==============================================================================
Task         : Azure PowerShell
Description  : Run a PowerShell script within an Azure environment
Version      : 2.0.2
Author       : Microsoft Corporation
Help         : [More Information](https://go.microsoft.com/fwlink/?LinkID=613749)
==============================================================================
##[command]Import-Module -Name C:\Program Files (x86)\Microsoft SDKs\Azure\PowerShell\ResourceManager\AzureResourceManager\AzureRM.Profile\AzureRM.Profile.psd1 -Global
##[command]Import-Module -Name C:\Program Files (x86)\Microsoft SDKs\Azure\PowerShell\Storage\Azure.Storage\Azure.Storage.psd1 -Global
##[command]Add-AzureRMAccount -ServicePrincipal -Tenant ******** -Credential System.Management.Automation.PSCredential -EnvironmentName AzureCloud
##[command]Select-AzureRMSubscription -SubscriptionId <stripped> -TenantId ********
##[command]& 'd:\a\_temp\f3289bb3-6ee3-4f5f-bb32-b5bf080fd932.ps1' 

ProcDump v9.0 - Sysinternals process dump utility
Copyright (C) 2009-2017 Mark Russinovich and Andrew Richards
Sysinternals - www.sysinternals.com

[22:11:37] Dump 1 initiated: d:\a\1\s\ps.dmp
[22:11:37] Dump 1 writing: Estimated dump file size is 380 MB.
[22:11:48] Dump 1 complete: 381 MB written in 11.4 seconds
[22:11:49] Dump count reached.

##[section]Finishing: Azure PowerShell script: InlineScript

Looking for Secrets in the Build Process Memory

After a successfull build, the process memory dump is available as a build artifact. We may now download it and open in WinDbg. Both Login-AzureRMAccount and Add-AzureRMAccount use the PSAzureContext class from the Microsoft.Azure.Commands.Profile assembly. Let’s find it in the memory:

0:000> .loadby sos clr
0:000> !DumpHeap -type Microsoft.Azure.Commands.Profile.Models.PSAzureContext
         Address               MT     Size
000000f166132b80 00007ff7e2100038       56     
000000f166270010 00007ff7e2100038       56     

Statistics:
              MT    Count    TotalSize Class Name
00007ff7e2100038        2          112 Microsoft.Azure.Commands.Profile.Models.PSAzureContext
Total 2 objects
Fragmented blocks larger than 0.5 MB:
            Addr     Size      Followed by
000000f1667592b8   19.6MB 000000f167afb408 System.Byte[]

0:000> !mdt 000000f166132b80
000000f166132b80 (Microsoft.Azure.Commands.Profile.Models.PSAzureContext)
    <Account>k__BackingField:000000f166132bb8 (Microsoft.Azure.Commands.Profile.Models.PSAzureRmAccount)
    <Environment>k__BackingField:000000f166132fb8 (Microsoft.Azure.Commands.Profile.Models.PSAzureEnvironment)
    <Subscription>k__BackingField:000000f166133060 (Microsoft.Azure.Commands.Profile.Models.PSAzureSubscription)
    <Tenant>k__BackingField:000000f166133108 (Microsoft.Azure.Commands.Profile.Models.PSAzureTenant)
    <TokenCache>k__BackingField:000000f1661324a0 (System.Byte[], Elements: 1597)

The TokenCache property seems interesting:

0:000> db 000000f1661324a0+0x10 L100
000000f1`661324b0  02 00 00 00 01 00 00 00-99 01 68 74 74 70 73 3a  ..........https:
000000f1`661324c0  2f 2f 6c 6f 67 69 6e 2e-6d 69 63 72 6f 73 6f 66  //login.microsof
000000f1`661324d0  74 6f 6e 6c 69 6e 65 2e-63 6f 6d 2f 34 38 30 34  tonline.com/4804
...
000000f1`66132500  2f 3a 3a 3a 68 74 74 70-73 3a 2f 2f 6d 61 6e 61  /:::https://mana
000000f1`66132510  67 65 6d 65 6e 74 2e 63-6f 72 65 2e 77 69 6e 64  gement.core.wind
000000f1`66132520  6f 77 73 2e 6e 65 74 2f-3a 3a 3a 34 63 39 31 66  ows.net/:::4c91f
000000f1`66132530  63 63 30 2d 39 35 34 32-2d 34 38 62 61 2d 38 63  cc0-9542-48ba-8c
000000f1`66132540  64 61 2d 31 38 38 33 32-34 63 38 65 62 37 66 3a  da-188324c8eb7f:
000000f1`66132550  3a 3a 31 98 0b 7b 22 41-63 63 65 73 73 54 6f 6b  ::1..{"AccessTok
000000f1`66132560  65 6e 22 3a 22 65 79 4a-30 65 58 41 69 4f 69 4a  en":"eyJ0eXAiOiJ
000000f1`66132570  4b 56 31 51 69 4c 43 4a-68 62 47 63 69 4f 69 4a  KV1QiLCJhbGciOiJ
000000f1`66132580  53 55 7a 49 31 4e 69 49-73 49 6e 67 31 64 43 49  SUzI1NiIsIng1dCI
000000f1`66132590  36 49 6b 68 49 51 6e 6c-4c 56 53 30 77 52 48 46  6IkhIQnlLVS0wRHF
000000f1`661325a0  42 63 55 31 61 61 44 5a-61 52 6c 42 6b 4d 6c 5a  BcU1aaDZaRlBkMlZ

The ASCII column shows that we found the Bearer token used to authenticate requests to the Azure API. But it is not the Service Principal password. After examining few more classes and finding nothing interesting it is time to try a brute force method. I know that Azure generates the Service Principal password in the form of base64 text usually longer than 40 characters. Let’s dump all managed strings longer than 40 characters from the .NET heap (using an excellent !strings command from the SOSEX extension):

0:000> .load sosex
0:000> !strings -n:40
Address            Gen    Length   Value
---------------------------------------
000000f166130328    0         56   Switch.System.Runtime.Serialization.DoNotUseTimeZoneInfo
000000f1661309f0    0       1432   {"AccessToken":"eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6IkhIQnlLVS0wRHFBcU1aaDZaRlBkMlZXYU90ZyIsImtpZCI6IkhIQnlLVS0wRHFBcU1a...
000000f166133c08    0         40   AddAzureRMAccountCommand end processing.
000000f166133db0    0         54   10:11:32 PM - AddAzureRMAccountCommand end processing.
000000f166133e68    0         40   AddAzureRMAccountCommand end processing.
000000f166134010    0         54   10:11:32 PM - AddAzureRMAccountCommand end processing.
...
000000f174b13fb8   LOH     52296   <?xml version="1.0" encoding="utf-8" ?>
<Configuration>
  <ViewDefinitions>
    <View>
      <Name>Microsoft.Azure.Commands....
---------------------------------------
7965 matching strings

7965 lines are too many to analyze manually. Therefore, I will copy the output of the !strings command to a file and will scan it for valid base64 strings using PowerShell:

Get-Content .\ps.dmp.txt | `
	% { $_ -split " " } | `
	? { $_ -and ($_.Length -gt 40) } | `
	% { try { `
		$null = [Convert]::FromBase64String($_); `
		Write-Host $_ `
	} catch {  } }

This command produces much less text, and it is relatively easy to spot the interesting lines (highlighted):

ABvh6MRNCdsJqHJKL3C6/xlwUa8LvDeAXuJW9eAtA3U=
AzureRmSqlServerActiveDirectoryAdministrator
AzureRmSqlServerBackupLongTermRetentionVault
AzureRmSqlServerActiveDirectoryAdministrator
AzureRmSqlServerBackupLongTermRetentionVault
AzureRmSqlServerActiveDirectoryAdministrator
AzureRmSqlServerActiveDirectoryAdministrator
AzureRmSqlServerBackupLongTermRetentionVault
AzureRmSqlDatabaseExecuteIndexRecommendation
AzureRmSqlDatabaseExecuteIndexRecommendation
PropagateExceptionsToEnclosingStatementBlock
NormalizeRelativePathUnauthorizedAccessError
ParameterArgumentValidationErrorEmptyArrayNotAllowed
4ESBcfx2mEo/o3U7QGxoTP2r5HzX+nv8you9ChT7JpI=
AlreadyExistingUserSpecifiedPropertyNoExpand
AzureKeyVaultCertificateAdministratorDetails
AzureKeyVaultCertificateAdministratorDetails
...

We could stop here and test the strings against Azure API to check which of them works. However, I would stay in the WinDbg window a bit longer and extract some more information about the highlighted strings. Their addresses in memory are as follows:

000000f166240168    0         44   ABvh6MRNCdsJqHJKL3C6/xlwUa8LvDeAXuJW9eAtA3U=
000000f165415e38    2         44   4ESBcfx2mEo/o3U7QGxoTP2r5HzX+nv8you9ChT7JpI=

And their GC roots:

0:000> !GCRoot 000000f166240168
Found 0 unique roots (run '!GCRoot -all' to see all roots).
0:000> !GCRoot 000000f165415e38
Finalizer Queue:
    000000f16541c130
    -> 000000f16541c130 System.Management.Automation.CommandProcessor
    -> 000000f1653f9b88 System.Management.Automation.SessionStateScope
    -> 000000f1653f98d8 System.Management.Automation.MutableTuple`32[[System.Object, mscorlib],[System.Object[], mscorlib],[System.Object, mscorlib],[System.Object, mscorlib],[System.Management.Automation.PSScriptCmdlet, System.Management.Automation],[System.Management.Automation.PSBoundParametersDictionary, System.Management.Automation],[System.Management.Automation.InvocationInfo, System.Management.Automation],[System.String, mscorlib],[System.String, mscorlib],[System.Management.Automation.ActionPreference, System.Management.Automation],[System.Management.Automation.ActionPreference, System.Management.Automation],[System.Management.Automation.ActionPreference, System.Management.Automation],[System.Management.Automation.SwitchParameter, System.Management.Automation],[System.Management.Automation.ActionPreference, System.Management.Automation],[System.Management.Automation.ActionPreference, System.Management.Automation],[System.Management.Automation.ConfirmImpact, System.Management.Automation],[System.String, mscorlib],[System.Object, mscorlib],[System.Object, mscorlib],[System.Object, mscorlib],[System.Object, mscorlib],[System.Object, mscorlib],[System.Management.Automation.LanguagePrimitives+Null, System.Management.Automation],[System.Management.Automation.LanguagePrimitives+Null, System.Management.Automation],[System.Management.Automation.LanguagePrimitives+Null, System.Management.Automation],[System.Management.Automation.LanguagePrimitives+Null, System.Management.Automation],[System.Management.Automation.LanguagePrimitives+Null, System.Management.Automation],[System.Management.Automation.LanguagePrimitives+Null, System.Management.Automation],[System.Management.Automation.LanguagePrimitives+Null, System.Management.Automation],[System.Management.Automation.LanguagePrimitives+Null, System.Management.Automation],[System.Management.Automation.LanguagePrimitives+Null, System.Management.Automation],[System.Management.Automation.LanguagePrimitives+Null, System.Management.Automation]]
    -> 000000f1654192e0 System.Management.Automation.PSObject
    -> 000000f165419410 System.Management.Automation.PSMemberInfoInternalCollection`1[[System.Management.Automation.PSMemberInfo, System.Management.Automation]]
    -> 000000f165419430 System.Collections.Specialized.OrderedDictionary
    -> 000000f1654194c0 System.Collections.ArrayList
    -> 000000f165419508 System.Object[]
    -> 000000f165419818 System.Collections.DictionaryEntry
    -> 000000f165419728 System.Management.Automation.PSNoteProperty
    -> 000000f165415fb0 System.Management.Automation.PSObject
    -> 000000f1654166c0 System.Management.Automation.PSMemberInfoInternalCollection`1[[System.Management.Automation.PSMemberInfo, System.Management.Automation]]
    -> 000000f1654166e0 System.Collections.Specialized.OrderedDictionary
    -> 000000f1654167d0 System.Collections.ArrayList
    -> 000000f165416818 System.Object[]
    -> 000000f1654167f8 System.Collections.DictionaryEntry
    -> 000000f165416690 System.Management.Automation.PSNoteProperty
    -> 000000f165416088 System.Management.Automation.PSObject
    -> 000000f1654161a8 System.Management.Automation.PSMemberInfoInternalCollection`1[[System.Management.Automation.PSMemberInfo, System.Management.Automation]]
    -> 000000f1654161c8 System.Collections.Specialized.OrderedDictionary
    -> 000000f165416258 System.Collections.ArrayList
    -> 000000f1654162a0 System.Object[]
    -> 000000f1654163c8 System.Collections.DictionaryEntry
    -> 000000f165416398 System.Management.Automation.PSNoteProperty
    -> 000000f165415e38 System.String

As you can see the GC root to our second string exists only on the finalizer queue, which means that if only the memory pressure on the machine was higher, our precious password would be gone. It is just something you need to keep in mind when playing with the process memory.

2 thoughts on “Extracting Service Principal Credentials in VSTS

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s