Continuous Integration with Windows Azure SDK 1.7, Powershell and TFS Build too!

Environments change and the solutions to support them have to keep up. I was very entertained with my old deployment solution for a good while. However, we eventually moved to Azure, and I needed to scramble to find something new. Tom Hollander’s Automated Build and Deployment with Windows Azure SDK 1.6 filled that void until I upgraded my project type to the 1.7 SDK. At that point, I realized I had to roll up my sleeves and cobble something new together.

From an automated deployment standpoint, the crippling change between the 1.6 to the 1.7 SDK, is the lack of an “ImportAfter” folder, which allowed us to include legacy msbuild files to attach to the build process. This is what Tom used to attach a Powershell deployment script to the SDK’s Publish build target. However, with the Azure 1.7 SDK, I had to figure out how to execute that PowerShell script myself.

Microsoft: Downloads for Managing Services in Windows Azure

GitHub: StanleyGoldman/AzurePublishHelpers

Creating a Management Certificate & Publish Settings file

Visual Studio has a link which allows you to download a publish settings file, without completely explaining what the side effects are. I myself didn’t understand the problem when I encountered the first symptom, you have reached the maximum number of management certificates. I was forced to understand the situation when I tried to get publish settings files for the 2nd and 3rd Azure Subscriptions my account was associated with. The link creates a management certificate, uploads it to your azure account and provides you with a .publishSettings file to install onto your machine. Life is actually easier when we start taking control of our management certificates.

It is easy enough to create a management certificate. Gaurav Mantri’s blog post really helped on this.

makecert -sky exchange -r -n "CN=JustAProgrammer" -pe -a sha1 -len 2048 -ss My "JustAProgrammer.cer"

We can take this certificate and upload it to the Management Certificates console in Azure. Take note of your subscription id and the thumbprint of the certificate. As you will need it to create your publish Settings file.

Using the PublishSettingsCreator utility, we can create a publish settings file to carry our management information.

PublishSettingsCreator.exe "AzureExample" "ecd7cc1d-12ec-8cf6-a60b-0cf14db32020" "99DFFB9D05D0B5B92893FBBA35988DE281E01E9E"

In order to use the Azure Powershell Cmdlets we have to import the Azure Modules into our Powershell session.

PS C:UsersAdministratorDesktop> Import-Module 'C:Program Files (x86)Microsoft SDKsWindows AzurePowerShellAzureAzure.psd1'

And now we can import the publish settings file.

PS C:UsersAdministratorDesktop> Import-AzurePublishSettingsFile .ecd7cc1d-12ec-8cf6-a60b-0cf14db32020.publishsettings
Setting: AzureExample as the default and current subscription. To view other subscriptions use Get-AzureSubscription

Importing the settings file sets the subscription as default. We can get the default subscription as follows.

PS C:UsersAdministratorDesktop> Get-AzureSubscription

SubscriptionName        : AzureExample
SubscriptionId          : ecd7cc1d-12ec-8cf6-a60b-0cf14db32020
Certificate             : [Subject]
                            CN=JustAProgrammer

                          [Issuer]
                            CN=JustAProgrammer

                          [Serial Number]
                            9157E76A714509B54F8B853A1B80555B

                          [Not Before]
                            10/18/2012 8:21:06 AM

                          [Not After]
                            12/31/2039 6:59:59 PM

                          [Thumbprint]
                            99DFFB9D05D0B5B92893FBBA35988DE281E01E9E

ServiceEndpoint         : https://management.core.windows.net/
SqlAzureServiceEndpoint :
CurrentStorageAccount   :
IsDefault               : True

Before we can use this subscription to deploy, we have to set the storage account for the subscription. You can CurrentStorageAccount is not set in the subscription definition above.

PS C:UsersAdministratorDesktop> Set-AzureSubscription -SubscriptionName "AzureExample" -CurrentStorageAccount "azureexample"

With the publish settings file and Import-AzurePublishSettingsFile, Set-AzureSubscription commands we can allow any machine to deploy using said Management Certificate. The certificate and publish settings file should be guarded well, either of these files allow access to your azure subscription.

Customizing the Build

Create a new build definition and configure it to build the solution. Be sure to add the “Publish” target to your build. The argument will cause the Azure 1.7 SDK to create the deployment package during build.

If you are using TFSBuild, you can do this while configuring your build, look for the field ‘MSBuild Arguments’, add set the value ‘/t:Publish’.

Finish configuring the build definition and queue the build. We are going to use the output from the build to test the powershell script.

Testing the Powershell script

Using the output from the build, we should be able to execute the powershell script. Copy the script over to your build server. Execute it with the path to the publish file, location where the output Package and Configuration file can be found and the name of the package file in that location. A tag can optionally be specified to help identify the build. I usually use the version number of the binary here, but the build label works just as fine.

PS C:UsersAdministratorDesktop> C:BuildsAzureDeploy.ps1 
    'C:Builds1AzureExampleAzureExample DeploymentSourcesAzureExample.AzureProfilesAzureExampleProduction.azurePubxml'
    'C:Builds1AzureExampleAzureExample DeploymentSourcesAzureExample.AzurebinDebugapp.publish' 
    'AzureExample.Azure.cspkg' 
    'Test LabelTag'

Note: I’ve had some problems with the Azure Powershell Cmdlets and relative paths.

If your machine is configured correctly, this should deploy without any problems. If you are using TFS the next section is useful for you to wrap this all together. If you are using a build system other than TFS, you already have what you need to continue. Enjoy!

TFS Build Process Template

The Build Process Template included in the package takes a few arguments and handles the execution of the Powrshell script rather nicely. After choosing it as the template for your build, you just have to specify a few arguments.

Deployment Configuration: The build configuration of the Azure project, “Debug” or “Release”
Deployment Profile: The name of the profile file to be used
Deployment Project Name: The name of the project Azure project
Deployment Script: (Optional) In case you dont keep your deployment script in the same spot as mine

The build process template uses the Build Label as a tag for the build. It might be a bit more useful to use something like github: martinbuberl/VersionTasks to tag the builds.

Enjoy Continuous Integration

It never seems worth the effort until after you are done with it.

Using MongoDB in Powershell

Readers of this blog know that I’ve been using MongoDB for a while, and I’ve recently become very excited about Powershell. Well recently I’ve been able to combine the two together for pure dynamically typed, schema-less, non-relational awesomeness. Such awesomeness is begging to be shared.

To get started you will need version 1.1 of the official 10gen CSharp driver for Mongodb. At the time of this writing 1.1 has not been released, so your going to have to pull the latest version from github and compile it yourself. The minimal revision of the trunk you will need is this one. First download and install the MSI (or download the source, build the MSI and install it). Then open up your favorite text editor or Powershell IDE. Personally I am a fan of Powershell ISE or plain old vim, with some configuration modifications. Now we will load the bson and driver DLLs via the Add-Type cmdlet.

# Check to see if we are running the 64 bit version of Powershell.
# See http://stackoverflow.com/questions/2897569/visual-studio-deployment-project-error-when-writing-to-registry
if ([intptr]::size -eq 8) {
    $mongoDriverPath = (Get-ItemProperty "HKLM:SOFTWAREWow6432NodeMicrosoft.NETFrameworkv3.5AssemblyFoldersExMongoDB CSharpDriver 1.0").'(default)';
}
else {
    $mongoDriverPath = (Get-ItemProperty "HKLM:SOFTWAREMicrosoft.NETFrameworkv3.5AssemblyFoldersExMongoDB CSharpDriver 1.0").'(default)';
}
Add-Type -Path "$($mongoDriverPath)MongoDB.Bson.dll";
Add-Type -Path "$($mongoDriverPath)MongoDB.Driver.dll";

Since the Csharp Driver MSI is 32 bits, it creates the registry entries in the Wow6432Node. Therefore, we have to check to see if we are running in the 32 or 64 bit version of Powershell . Credit to an anonymous commenter on the msgoodies blog for providing this size of a pointer trick to determine if you are running a 32 or 64 bit system.

The next thing we want to do is to create a BSON document. This is surprisingly easy.

[MongoDB.Bson.BsonDocument] $doc = @{
    "_id"= [MongoDB.Bson.ObjectId]::GenerateNewId();
    "FirstName"= "Justin";
    "LastName"= "Dearing";
    "PhoneNumbers"= [MongoDB.Bson.BsonDocument] @{
        'Home'= '718-555-1212';
        'Mobile'= '646-555-1212';
    };
};

As you can see Powershell can convert a HashTable to a BsonDocument. This is because of the public constructor BsonDocument(IDictionary hashTable). Powershell can use these one parameter constructors to cast an object. You can use the same Hashtable trick for the QueryDocument and UpdateDocument classes.

Now that we have our BsonDocument, its time to perform basic crud operations.

$db = [MongoDB.Driver.MongoDatabase]::Create('mongodb://localhost/powershell');
$collection = $db['example1'];
Write-Host "Insert";
$collection.Insert($doc);
$collection.FindOneById($doc['_id']);

$updates = @{'$set' = @{'email'= 'justin@mongodb.org'}};
$query = @{"_id"= $doc['_id']}

Write-Host "Update";
$collection.Update([MongoDB.Driver.QueryDocument]$query, [MongoDb.Driver.UpdateDocument]$updates);
$collection.FindOneById($doc['_id']);

Write-Host "Delete";
$collection.Remove([MongoDB.Driver.QueryDocument]$query);
$collection.FindOneById($doc['_id']);

This gives us the output below.

_id                                                         4dc5e5a43b42bd41d0154679
LastName                                                    Dearing
PhoneNumbers                                                {Mobile=646-555-1212, Home=718-555-1212}
FirstName                                                   Justin
Delete

PS E:srcgistgist-854911> powershell.exe -file .MongoTest.ps1

Insert

Name                                                        Value
----                                                        -----
_id                                                         4dc636c33b42bd3d7c8dc3a4
LastName                                                    Dearing
PhoneNumbers                                                {Mobile=646-555-1212, Home=718-555-1212}
FirstName                                                   Justin
Update
FirstName                                                   Justin
LastName                                                    Dearing
PhoneNumbers                                                {Mobile=646-555-1212, Home=718-555-1212}
_id                                                         4dc636c33b42bd3d7c8dc3a4
email                                                       justin@mongodb.org
Delete

As you can see, its not very hard to use the 10Gen MongoDB Csharp driver from within Powershell. Using Powershell with the MongoDb C-Sharp driver has many possibilities. First of all, adhoc mongodb queries from inside of powershell. Secondly, The code for this example is available in its entirety here.

Visual Studio, Line Numbers and Ankhsvn

Although I am a git and DVCS true believer, I still deal with SVN. In Visual Studio this usually means I use AnkhSVN. For anyone paying for the VisualSVN client, I really think you should evaluate this tool. Its Visual Studio Integration is superb and I’ve had no usability problems in recent versions.

One thing about it annoyed me was caused by it integrating too well with Visual Studio. That problem was it is the display of line numbers on the editor screen. I setup Visual Studio to display line numbers in editor windows, and this has the side effect of displaying line numbers in the log message editor of the commit screen. I find this annoying. I don’t need line numbers for my commit log files and they take up precious screen real estate.

Commit to Subversion With Line Numbers

Luckily I found a solution. Apparently, in the current version of AnkhSVN, the commit editor has its own section in the Visual Studio Text editor options. So turning off the line numbers on the commit screen, but not the other editor screens can be done from the Tools | Options menu.

Options|TextEditor|General
Options|TextEditor|General
Options|TextEditor|Log Messages (AnkhSVN)
Options|TextEditor|Log Messages (AnkhSVN)

After that, the line numbers disappear in the commit dialog.

Commit to Subversion Sans Line Numbers

Its the little things like this that make a software experience truly great.

IIS7 Web Application Enabled Protocols: Yet Another WCF Gotcha!

One of the cool things about IIS 7.0  and WCF is the ability to serve WCF endpoints with non-http bindings. Naturally, this new feature presents new opportunities for the developer to get frustrated by WCF configuration headaches. This blog post is about one of them.

I was writing a WCF web service that had three endpoints, Json, Soap 1.1, and net.tcp. This services primary purpose was to be the middleware for the mongo database where my applications data was held. In the end, I didn’t need the net.tcp endpoint, making this exercise a complete waste of time.  The reasons for my architectural decisions for this app are the subject of another blog post. For now, lets just say if you are supposed to learn more from your failures than your successes, I should get an honorary doctorate for this app.

In the past I’ve mixed soap and net.tcp in EXE hosted WCF services with great success. However, since I’ve yet to find a JSONP endpoint solution for WCF that allows request parameters, I needed to host this in IIS on the same site as my project. So I made a .svc file in my website, added the service dll as a reference, and ran it. I promptly got the the following error:

Could not find a base address that matches scheme

It was a bit frustrating to find the solution, but I did eventually. In IIS manager you have to select Advanced Settings in the applications Application folder, or website if the app is running at the root. In the advanced settings dialog is an option called Enabled Protocols. It probably contains the value http or http,https. You simple have to append ,net.tcp to the current value.

IIS Manager

IIS Application Advanced Settings

After that, everything works.

WCF Service

As an epilogue to this adventure, since I forgot to take all the screenshots needed for this blog article at work, I ended up having to make an example project to reproduce the error at home. As such I took an older example WCF project I wrote called EchoService and adding a website host to it in addition to the exe host. This improved version of EchoService can be found at the justaprogrammer github org. Feel free to use this service s the basis for any WCF related instructional materials. The code is licensed under the very permissive MIT license.

Using the registry to resolve Visual Studio reference paths.

Update 2012-12-24: Updated to reflect the MSBuild 4.0 syntax as I describe here.
Note: To skip the long journey of what lead me to figuring this out, click here to go to the howto.

Recently I was asked to look at a fiddler plugin Stan, the founder of this blog, was developing. He gave me a SVN path and asked me to build it and test it.

So I checked out the source code and hit F5. I got a bunch of compiler errors relating to the fact that I didn’t have fiddler installed. I rectified that matter and still got errors. The problem was that the hintpath of fiddler.exe was wrong. On my machine, Fiddler is installed in ‘C:Program FilesFiddler2’, while on Stan’s machine it is installed to ‘C:Program Files (x86)Fiddler2′. I consulted the mighty google, which led me to a StackOverflow question. The question pointed out that you can have multiple hintpaths to an assembly. However, I wanted a better solution. What if someone installed Fiddler to a custom location?

I got the idea of using the registry. Fiddler has an installer. Surely the installer records its install location to the registry. It does in ‘HKEY_LOCAL_MACHINESOFTWAREMicrosoftFiddler2’ (Apparently fiddler is written by a Microsoft Employee). So the question is how to get MSBuild, the tool that visual studio uses to parse project files, to read a value from the registry.

The answer was found in a post on the MSBuild team blog. However, for Visual Studio 2010 and later, I recommend the syntax described here. The new syntax takes into account RegistryView and the Wow6432Node. The old syntax happens to work just fine in Visual Studio because Fiddler is compiled to 32 bits explicitly and Visual Studio is a 32 bit app. However, if you were to compile the app on the command line on a 64 bit system, the reference would not be resolved.

How To

Unfortunately, Visual Studio does not allow you to edit hintpaths to referenced assemblies. So you’re going to have to edit your vcproj or vbproj file in notepad or some other text editor. Here are the steps:

  1. Open the project file in your text editor.
  2. Look for the element for fiddler.exe. It should look similar to this:
    <Reference Include=”Fiddler, Version=2.2.7.5, Culture=neutral, processorArchitecture=MSIL”>
      <SpecificVersion>False</SpecificVersion>
      <HintPath>C:\Program Files\Fiddler.exe</HintPath>
      <Private>False</Private>
      </Reference>
  3. Change the hintpath as follows (Visual Studio 2010 and later):
    <Reference Include="Fiddler, Version=2.2.7.5, Culture=neutral, processorArchitecture=MSIL">
    <SpecificVersion>False</SpecificVersion>
      <HintPath>$(Registry:HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Fiddler2@InstallPath)Fiddler.exe</HintPath>
      <Private>False</Private>
    </Reference>
  4. On Visual Studio 2008, you must use the old syntax:
    <Reference Include="Fiddler, Version=2.2.7.5, Culture=neutral, processorArchitecture=MSIL">
      <SpecificVersion>False</SpecificVersion>
      <HintPath>$([MSBuild]::GetRegistryValueFromView('HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Fiddler2', 'InstallPath', null, RegistryView.Registry32))Fiddler.exe</HintPath>
    <Private>False</Private>
    </Reference>
  5. Save the file
  6. Visual studio will detect the file change and ask you to reload the file. If you are using SharpDevelop as you’re IDE, you will have to close and reopen the solution.

Thats all there is to it. Happy coding!