Windows Internals Study Group Proposal

UPDATE: I’ve made a google group to move the discussion to.

While most of the code I’ve written to put food on the table has been for application development, I’ve always had a true passion for system development. Its a bit meta, sort of like being the mechanical engineer that just wants to make a better ratchet wrench. However, I think its important to understand, and if given the opportunity, help write the code that runs and supports the code that makes the end users productive.

As a result of this passion, I’ve been reading the 5th edition of Windows Internals by Windows Hacker extraordinaires Mark E. Russinovich and David A. Solomon. This edition covers Windows Vista and Windows Server 2008. While its been enlightening, and the exercises have helped reinforce the concepts, the book is still somewhat academic. It’s also not a book about programming or a book about system administration. Its a book that teaches concepts. Some of these concepts mesh well with my previous knowledge and are directly applicable to things I’ve done or want to do. I retain these concepts well. Other concepts, such as the finer points of windows objects, are not things I can relate to other concepts (yes I see the unix everything is a file parallel, but I don’t see the dd if=/dev/foo of=/tmp/foo.img parallel that makes it useful).

So I decided to form a study group. I’ve done this before for the ZCE in PHP5, with the Long Island PHP user group. Here is what I am proposing. All interested parties contact me via the comments or at @zippy1981 on twitter. We will meet once a week, each week covering a different chapter.  The expectation as participants would be as follows:

Before the meeting

  • Do the assigned homework for the previous chapter
  • Read the chapter. Take notes of what you did not understand. Make a list of hyperlinks and deadtree references that you used to supplement the chapter, if any.
  • Do all the exercises with the prescribed  Windows System Internals tools and Windows Debugger (these are all free downloads).
  • Do the exercises with equivalent third party tools including:

During the Meeting

  • Take a turn in leading on of the meetings. I suggest picking the chapter you are most intimidated by, not the one you are most confident in. This is a reinforcement tool.
  • Present your homework from the previous chapter. We will go around the room. Depending on the size of the group and scope of the assignment we might only have a subset of the group volunteer to present or break into small groups.
  • Discuss the chapter of the week. Share what you didn’t understand, help others who didn’t understand things that you did understand. (the whole point of a study group)
  • Last weeks discussion leader will demonstrate this weeks exercises using both the sysinternals tools and the third party ones we agree upon.
  • The group leader will assign us a homework for next week,
  • Next weeks discussion leader will act as secretary and do the following:
    • Collect everyone’s links to post to a wiki we will maintain.
    • Take minutes of what we discussed in the meetings for the wiki.
    • Where appropriate, filing bugs and feature requests for third party tools that lack the functionality to do the exercises.

Grading

Now, this isn’t high school, and we’re not getting graded. No one will chastise you if you don’t go to all the sessions or do all the homework. We all have jobs, friends, families, and sometimes a study group isn’t the most important thing. I expect the class to consist of mostly adults, but welcome any teenagers that think they can handle the material. Therefore, I will treat you all like adults.

That being said, I do want to run this as a pass fail course and give out certificates. Unlike the PHP ZCE study class I ran, there is no clear external goal. There is no certification on Windows Internals, except perhaps as parts of instructor lead courses. Passing will consist of doing all the homework, and actively participating in all the classes. If you miss a class, you can make it up my meeting later in the week with at least one other member of the study group. While a certificate seems kind of corny, especially coming from as unaccredited and aprestigious a body as the one we will be forming, I feel this  small carrot will help with commitment.

Location

There is no reason this can’t work remotely. I will not turn anyone down because they can only attend via skype. However, I’d like at least some of us to meet in person.

I live in Jersey City and work in Hoboken. I’ll travel to Manhattan, Brooklyn, and Hudson, Bergen and Essex counties, or perhaps somewhere a little farther if I can find a really convenient train. We can certainly have multiple physical meeting locations (e.g. a group of people from Chicago meet there and I meet with some people in Hoboken).

Ideally, I’d like a meeting facility with a projector for the group location((s). We’d probably need to use webex if we are all not in the same room.

What edition?

I’m reading the 5th edition (because I happen to own it). The 6th edition covers Windows 7 and Windows Server 2008R2. Its also a two volume edition. so its more expensive. I need to research if there will be an addition for Windows 8. I’m  proposing at this point the 6th Edition, because part 2 of the 6th edition was just released.

Flexibility

My proposal is mostly based on what worked for the PHP cert. This will be quite different though. We might want to break some chapters in half, or dedicate two meetings to a chapter.

I hope to get enough interest to make this happen. I think this could work out really well and prove to be a great learning experience for everyone.

Excel Link Dump

I was recently told I had to train one of my companies clients to access our API’s using Excel. This ended up being a miscommunication, and the client wanted to use C#. However, I spent a day re-learning Excel VBA before I got the correct information. Relearning involved writing a simple app with out API and doing a lot of googling to fill in the large gaps in my knowledge.

I decided to post the list of useful links I found here, mostly for my own reference. If I ever have to touch Excel again, I’ll add to this list, and perhaps curate it a little better.

Perhaps I might be inspired to rewrite the Excel Reverse DNS macro I wrote many years ago.

Continuous Integration with Windows Azure SDK 1.7, Powershell and TFS Build too!

Environments change and the solutions to support them have to keep up. I was very entertained with my old deployment solution for a good while. However, we eventually moved to Azure, and I needed to scramble to find something new. Tom Hollander’s Automated Build and Deployment with Windows Azure SDK 1.6 filled that void until I upgraded my project type to the 1.7 SDK. At that point, I realized I had to roll up my sleeves and cobble something new together.

From an automated deployment standpoint, the crippling change between the 1.6 to the 1.7 SDK, is the lack of an “ImportAfter” folder, which allowed us to include legacy msbuild files to attach to the build process. This is what Tom used to attach a Powershell deployment script to the SDK’s Publish build target. However, with the Azure 1.7 SDK, I had to figure out how to execute that PowerShell script myself.

Microsoft: Downloads for Managing Services in Windows Azure

GitHub: StanleyGoldman/AzurePublishHelpers

Creating a Management Certificate & Publish Settings file

Visual Studio has a link which allows you to download a publish settings file, without completely explaining what the side effects are. I myself didn’t understand the problem when I encountered the first symptom, you have reached the maximum number of management certificates. I was forced to understand the situation when I tried to get publish settings files for the 2nd and 3rd Azure Subscriptions my account was associated with. The link creates a management certificate, uploads it to your azure account and provides you with a .publishSettings file to install onto your machine. Life is actually easier when we start taking control of our management certificates.

It is easy enough to create a management certificate. Gaurav Mantri’s blog post really helped on this.

makecert -sky exchange -r -n "CN=JustAProgrammer" -pe -a sha1 -len 2048 -ss My "JustAProgrammer.cer"

We can take this certificate and upload it to the Management Certificates console in Azure. Take note of your subscription id and the thumbprint of the certificate. As you will need it to create your publish Settings file.

Using the PublishSettingsCreator utility, we can create a publish settings file to carry our management information.

PublishSettingsCreator.exe "AzureExample" "ecd7cc1d-12ec-8cf6-a60b-0cf14db32020" "99DFFB9D05D0B5B92893FBBA35988DE281E01E9E"

In order to use the Azure Powershell Cmdlets we have to import the Azure Modules into our Powershell session.

PS C:UsersAdministratorDesktop> Import-Module 'C:Program Files (x86)Microsoft SDKsWindows AzurePowerShellAzureAzure.psd1'

And now we can import the publish settings file.

PS C:UsersAdministratorDesktop> Import-AzurePublishSettingsFile .ecd7cc1d-12ec-8cf6-a60b-0cf14db32020.publishsettings
Setting: AzureExample as the default and current subscription. To view other subscriptions use Get-AzureSubscription

Importing the settings file sets the subscription as default. We can get the default subscription as follows.

PS C:UsersAdministratorDesktop> Get-AzureSubscription

SubscriptionName        : AzureExample
SubscriptionId          : ecd7cc1d-12ec-8cf6-a60b-0cf14db32020
Certificate             : [Subject]
                            CN=JustAProgrammer

                          [Issuer]
                            CN=JustAProgrammer

                          [Serial Number]
                            9157E76A714509B54F8B853A1B80555B

                          [Not Before]
                            10/18/2012 8:21:06 AM

                          [Not After]
                            12/31/2039 6:59:59 PM

                          [Thumbprint]
                            99DFFB9D05D0B5B92893FBBA35988DE281E01E9E

ServiceEndpoint         : https://management.core.windows.net/
SqlAzureServiceEndpoint :
CurrentStorageAccount   :
IsDefault               : True

Before we can use this subscription to deploy, we have to set the storage account for the subscription. You can CurrentStorageAccount is not set in the subscription definition above.

PS C:UsersAdministratorDesktop> Set-AzureSubscription -SubscriptionName "AzureExample" -CurrentStorageAccount "azureexample"

With the publish settings file and Import-AzurePublishSettingsFile, Set-AzureSubscription commands we can allow any machine to deploy using said Management Certificate. The certificate and publish settings file should be guarded well, either of these files allow access to your azure subscription.

Customizing the Build

Create a new build definition and configure it to build the solution. Be sure to add the “Publish” target to your build. The argument will cause the Azure 1.7 SDK to create the deployment package during build.

If you are using TFSBuild, you can do this while configuring your build, look for the field ‘MSBuild Arguments’, add set the value ‘/t:Publish’.

Finish configuring the build definition and queue the build. We are going to use the output from the build to test the powershell script.

Testing the Powershell script

Using the output from the build, we should be able to execute the powershell script. Copy the script over to your build server. Execute it with the path to the publish file, location where the output Package and Configuration file can be found and the name of the package file in that location. A tag can optionally be specified to help identify the build. I usually use the version number of the binary here, but the build label works just as fine.

PS C:UsersAdministratorDesktop> C:BuildsAzureDeploy.ps1 
    'C:Builds1AzureExampleAzureExample DeploymentSourcesAzureExample.AzureProfilesAzureExampleProduction.azurePubxml'
    'C:Builds1AzureExampleAzureExample DeploymentSourcesAzureExample.AzurebinDebugapp.publish' 
    'AzureExample.Azure.cspkg' 
    'Test LabelTag'

Note: I’ve had some problems with the Azure Powershell Cmdlets and relative paths.

If your machine is configured correctly, this should deploy without any problems. If you are using TFS the next section is useful for you to wrap this all together. If you are using a build system other than TFS, you already have what you need to continue. Enjoy!

TFS Build Process Template

The Build Process Template included in the package takes a few arguments and handles the execution of the Powrshell script rather nicely. After choosing it as the template for your build, you just have to specify a few arguments.

Deployment Configuration: The build configuration of the Azure project, “Debug” or “Release”
Deployment Profile: The name of the profile file to be used
Deployment Project Name: The name of the project Azure project
Deployment Script: (Optional) In case you dont keep your deployment script in the same spot as mine

The build process template uses the Build Label as a tag for the build. It might be a bit more useful to use something like github: martinbuberl/VersionTasks to tag the builds.

Enjoy Continuous Integration

It never seems worth the effort until after you are done with it.

SQLCLR wrapper for RAISERROR()

Recently I was writing a SQLCLR stored procedure. I made several calls to RAISERROR() in the procedure. While this meant that my stored procedure would be easier to debug in the future because of useful error messages, there was a lot of ceremony involved in these RAISERROR() calls. Therefore I decided to encapsulate all this ceremony into a simple static method.

RAISERROR() presents a small challenge in wrapping in a C# function. That challenge is the variable parameter length due to the printf() like parameter substitution. The first step to overcoming this is to use the params keyword for the substitution parameters. The second step is avoiding the possibility of SQL Injection.

Many built in CLR functions use the params keyword. For example, if you use the String.Format(), Console.WriteLine() or Debug.Print(), with substitution parameters you have used overloads with the params keyword. For example, lets look at these two calls to String.Format();

String.Format("Name: {0}", "Justin");
String.Format("Name: {0} Occupation: {1}", "Justin", "Developer");

It may look like I’m using two overloads, but I am not. One might also conclude that String.Format() takes one string and eleventy billion object parameters all of which have a default of null. However, parameter defaults are new in .NET 3.5, and even the long forgotten .NET Framework 1.0 had a fully functional String.Format(). In actuality, the signature for String.Format() is as follows:

public static string Format(string format, params Object[] args)

Basically, the params keyword lets the user pass a variable size list of parameters, and populates them in a singe argument as an array. So now that we know about params, it seems like it shouldn’t be too hard to write our function lets give it a go.

private static void RaisError(SqlConnection cn, string message, short severity, short state, params object[] args) { . . . }

Now of course you want to protect against SQL injection. The best way to do that is to use the SqlCommand.Parameters collection to scrub the parameters. While you may be very clever, the ADO.NET code is very well battle tested. The only problem is if you don’t know how many parameters you are passing, how do you know how to make the string. The answer, is to generate SqlCommand.CommandText dynamically.

This may seem like you are opening yourself up for SQL injection. However, you aren’t. You simple have to generate a string that contains @arg1, @arg2, @arg3 . . . based on the length of the object array that gets passed to params. You then populate the SqlCommand.Parameters collection with the values of that array. If you don’t feel comfortable with this, then perhaps a stackoverflow answer with 268 votes at the time I wrote this article advocating this practice would convince you.

So putting it all together here is our function:

The above function is copyright 2012 myself (Justin Dearing). However, I release it under the MIT license, so feel free to use it in any of your code.