Announcing the JustAProgrammer github organization

This is a programming blog. Sometimes the authors of this site write code specifically for articles contained here. For all intents and purposes, said projects home pages are the blog articles in which they appear. Naturally anything of consequence gets stored in version control. In the past,, the has meant my personal github repo.

Not anymore! Recently I have made a github organization for these projects. This means that justaprogrammer has its own little place in github, and the authors can make shared repositories. There is already on repository with one powershell script. Expect an article on that soon.

MongoDB as a Windows Service

Update: an older blog article exists on Chris Conway’s blog. The directions are out of date, but its an interesting read to get a historical perspective of the improvements mongod’s windows support.

Unix is an OS built around a worse is better philosophy. Part of that philosophy is defining things through convention. This has many advantages. One is thats its really easy to write a program that can run in both the console and as a daemon.

The mongo server, mongod, is a perfect example of this. If you run mongod in the console, it spews all its output to stdout. This is great for development and testing. However, if you want to run mongod all the time, its very simple to run it from an init script.

On the windows side of things, its more complicated. In windows, daemons run as services, except apparently if you are using an Azure instance. Services operate separately from interactive processes. Actually, thats not entirely true. You can have a service that interacts with the windows gui if you want to. As I said, its complicated.

In order for a windows program to operate as a service, you have to make certain API calls. Its actually not that hard in its most basic form, and there are well established patterns for doing it.

Furthermore, there is actually a wrapper program in the windows resource kit tool srvany.exe. It will allow you to turn almost any console program into a windows service. However, it is not an ideal solution. Luckily a programmer by the name of Alan Wright added proper windows service support to mongod. It was a well implemented service wrapper and I have made good use of it. I have also contributed some modifications to it that 10gen graciously accepted into their repo. The result is a really clean, but powerful service implementation built into mongod. I shall now demonstrate the power of this fully armed and operational death. . . I mean demonstrate the power of mongod’s windows service support.

Before we Begin

First, You are going to want to download the latest stable version of mongo. As of this writing that is mongo 1.6.3. Since mongo is evolving so rapidly, some of the more advanced features related to windows service support, not covered in this article, are only available in the unstable 1.7 series. Things move fast on the bleeding edge.

Second, you want to make sure you have mongod installed in a sane location. My definition of sane location is pretty much anyplace outside of C:Documents and Settings or C:Users. This also means on a hard drive permanently attached to your system. Theres nothing wrong with running mongod off an external hard drive if its always plugged in. Just keep in mind you won’t be able to unplug it while mongo is running, and the service will fail to start up if you boot your system without the drive plugged in.

Third, you want to be able to run mongod from a command prompt using the same switches as you wish the service to use. Please note that you are required to use logging when running mongod as a windows service. In my case I will run mongod like this:

mongod --auth --logpath c:datalogmongo.log --logappend --bind_ip localhost

What I am doing here is not overwriting the log every time I start mongo, and only listening for local connections. If mongod is running on the same machine as your web server, this is a good idea. I am also running with authentication.

Finally you want to make sure your command prompt has administrative access to your computer. Mongod will not raise a UAC prompt and elevate its own privileges. However, there’s a ticket for that. So make sure your do all the following steps from a command prompt running as an administrator. Also, note that in a future article I will talk about running mongod as a service using an unprivileged user.

And now we install the service

So you’ve worked out your particular command line options, made your data andlog folders, etc, etc. Double check your mongo log to make sure there are no errors. Now we are ready to install mongod as a service. To do this we simply append –install to the command prompt. So our install command looks like:

mongod --auth --logpath c:datalogmongo.log --logappend --bind_ip localhost --install

That should output the following:

C:\Program Files\Microsoft SDKs\Windowsv7.1>mongod --auth --logpath c:datalogmongo.log --logappend --bind_ip localhost --install
all output going to: c:datalogmongo.log
Creating service MongoDB.
Service creation successful.
Service can be started from the command line via 'net start "MongoDB"'.

Now lets say to want to change the parameters. For example, you decide to run without authentication. If mongod is already installed as a service, and you want to change the command line parameters, then you have to use –reinstall instead of –install. So lets try that now:

C:\Program Files\Microsoft SDKs\Windowsv7.1>mongod --logpath c:datalogmongo.log --logappend --bind_ip localhost --reinstall
all output going to: c:datalogmongo.log
Deleting service MongoDB.
Service deleted successfully.
Creating service MongoDB.
Service creation successful.
Service can be started from the command line via 'net start "MongoDB"'.

So as you can see –reinstall removes the service and then installs it again. Pretty self explanatory.

Ok and finally we want to cleanup. To remove mongod as a service, we will use –remove. The output:

C:\Program Files\Microsoft SDKs\Windowsv7.1>mongod --remove
Deleting service MongoDB.
Service deleted successfully.
Thu Sep 30 19:32:56  dbexit:

Thu Sep 30 19:32:56      shutdown: going to close listening sockets...
Thu Sep 30 19:32:56      shutdown: going to flush oplog...
Thu Sep 30 19:32:56      shutdown: going to close sockets...
Thu Sep 30 19:32:56      shutdown: waiting for fs preallocator...
Thu Sep 30 19:32:56      shutdown: closing all files...
Thu Sep 30 19:32:56      closeAllFiles() finished

Thu Sep 30 19:32:56  dbexit: really exiting now

You will note that this command seems extra verbose. This is because messages that would normally be sent to the log are being sent to stdout.

Starting and Stopping the Service

There are several ways to start and stop a windows service. You can use the Service Control Manager or SCM of course. However, since we are on the command line already, we might as well use that. The command to start our mongo service is “net start mongodb.” Likewise, “net stop mingodb” stops our service. The service is configured to automatically startup at boot time, which is probably what you want. If not you can tune this behavior in the SCM.

Further Directions

This is only the tip of the mongo as a windows service iceberg. More options are available. I will be discussing them in depth in future articles.

MongoDB The Definitive Guide: The Definitive Review

One of the advantages of being a programmer in New York City, the greatest city in the world, is there are so many companies and individuals contributing to open source. One of these companies is 10gen, the creators of MongoDB, and two individuals that work as developers for 10gen are Kristina Chodorow and Mike Dirolf. I met Kristina at her NYPHP talk where I was introduced to MongoDB. I eventually became a user of and contributor to MongoDB. As a result I met Mike, and other members of the 10gen team.

Due to this convenience of geography I was  able to get my copy of MongoDB: The Definitive Guide signed by its co-authors, the aforementioned Kristina and Mike.

My autographed copy of MongoDB: The Definitive Guide

So right now you’re probably saying, “wait this will probably be the most biased review ever!” Well, I’ll seriously consider re titling this post MongoDB The Definitive Guide: The Definitive Fanboy Review. Until then, on with the review.

When my book first arrived from bn.com, my first reaction was, “this is kinda thin for an O’Reilly book.” However, 192 pages makes sense for a book about MongoDB. MongoDB is a young small codebase. If you compare it to MySQL, or a more full featured database like Postgres, there are many full chapter topics, like triggers, that simple don’t have an equivalent in MongoDB. Finally, some topics are just plain simpler in MongoDB. For example, even though mongo has DBRefs (actually the drivers support this through convention), there are just fewer caveats about them than database joins. Therefore, this book is short for the same reason that The C Programming Language by K&R is short, there’s not a lot to talk about.

Now on to the content. Few people will read this book cover to cover, but its written in such a way that you can do so. However, if you want to go from mongo n00b to seasoned novice reading through the book is the way to go. Chapter 1 is the introductory chapter that explains how Mongo is so different. 2-7 covers the topic most programmers would be interested in, basic and advanced CRUD (Create, Insert, Update, and Delete) operations. Chapter 8, Administration, deals with the typical sysadmin things like starting, stopping, backing up, and monitoring mongo. 9 and10 deal with system architect topics, sharding and replication respectively. Finally chapter 11 gives you some example applications, and you have 3 appendices of reference information.

The book is really thorough about the material it covers. The syntax for CRUD operations is very thoroughly described, and performance implications  of various operations are discussed. I noted the fact that the section on the “$or” operator did not mention it was new in 1.6. However, this book will be used long after “$or” or the 1.6 series is new, so my belief that this was an oversite will soon become obsolete.

The sharding and replication chapters cover real world implications and best practices. Of course one would expect this considering these are the two showcase features of MongoDB. Kristina and Mike do not disappoint here.

My only comment on the example chapter is that I wish the same example was used for all the languages. It would be much easier to compare and contrast features that way. Then again others might have found the repetition a little boring.

All in all I have to say I definitely recommend this book.

Azure+MongoDb: Having my mind blown at Mongo Boston

Update: Originally the Mongo Boston talk on Azure was supposed to be given by David Makogon (blog) (twitter). He talks about the session and some upcoming articles he will write in this post.

I was at the thoroughly awesome Mongo Boston conference at the Micrsoft NERD Center this weekend. I had a great time at the conference, as well as the surrounding activities. Of all the talks during the conference, one stood out. It was given by Mark Eisenberg who does sales for Microsoft Azure. That talk was on running MongoDB in Azure.

For those who do not know, Azure is Microsoft’s cloud offering. While they offer you virtual hosts, its not a traditional VM slice offering. You don’t get to run processes with administrative access, and you don’t get RDP access.

A few things stood out about this talk. First, it was a well executed “initial conversation” sales pitch. However, it was aimed perfectly at the audience in the room: programmers, architects, and technical decision makers. Mark knew his stuff. I asked some pretty deep technical questions, and got actual answers. It was also refreshing to hear, “its probably better to use Azure to write new apps than to port existing apps.” Having a salesman set expectations so frankly is unfortunately unusual in the IT industry.

The second thing that stood out was how you run a standalone exe like mongod.exe on Azure. Since you do not have administrative access to the machine, you cannot deploy via MSI, and you cannot run in the context of a windows service. The mongod process is basically running in a command prompt on a console you don’t have access to. Also, you cannot create different users to run different processes. The assumption, since this is the cloud, is that you spin up a new instance for each process. To be quite frank, I found this quite appalling at first. I spent a good chunk of my early career doing helpdesk and later system administration for a small ISP. Although titularly I was in charge of Unix and iSeries machines, I helped out on the windows side of the shop as well. Sometimes we had to run windows apps on the console of a server, requiring that a user was always logged into the console of that server. Knowing first hand the problems this caused, I declared a crusade against such programs. Also, this means all the windows improvements I contributed to mongo, windows service related improvements, served no purpose in the cloud.

Now, I’ve always been very dogmatic about my development and operations practices, so I’m still adjusting to what a more cool headed developer would accept instantly. However, my emotions will eventually come to accept what my intellect knows to be true. The cloud is here, I can get on it, or become the best buggy whip manufacturer there ever was.

The third thing that stood out had very little to do with the specifics of mongod and Azure. While its quite obvious Microsoft wants you to develop for Azure using .NET, they care about properly supporting all the third party technologies that Azure supports. I felt more like I was being sold by Lou Gerstner era IBM, than modern day Balmer lead Microsoft. Azure is being sold as a service. While Microsoft naturally wants to supplement that with the sale of their software products, they mainly want you to run your software on their platform. The threat of vendor lock-in is still there, but it always is on any platform.

Two Powershell One-Liners: Appending to my %PATH% and Whats in my %PATH%

To say I’ve fallen in love with PowerShell is an understatement. PowerShell is what perl would be; if perl was object based instead of stream based, and lacked all the “culture” of perl. I use PowerShell for a lot of things lately. Recently, I’ve been using it to manage my path, thanks to this PowerShell tip of the week. This has inspired me to share two one-liners related to path management.

The first allows you to add an item to your path. Few things annoy me more than windows programs that don’t include installers, and few installers annoy me more than those that install console programs and don’t offer to update your system path. Correcting this used to be a matter of going into nested levels of dialog windows. However, I can now do it from the PowerShell console. The command to append to your path is as follows:

[Environment]::SetEnvironmentVariable("PATH", [Environment]::GetEnvironmentVariable("PATH", 'Machine') + ';c:Program Filefoobin',"Machine")

A brief dissection:

  • [Environment]::SetEnvironmentVariable calls the static .NET method Environment.SetEnvironmentVariable()
  • “PATH” is the variable we are setting
  • $ENV: is a powershell drive for environment variables.
  • $ENV:PATH + ‘;c:Program Filefoobin’ concats your current path with a string.
  • “Machine” makes this environment variable apply to the entire machine, as opposed to the current user.

So there you have it, instead of digging three levels deep into menus to edit your path, you simply dig up this blog article and copy and paste that one line into PowerShell 🙂

The next one-liner is to determine what’s in your path. Simply typing $Env:PATH will print out your path in the same manner as the cmd.exe command path does. However, you probably want something more human readable. How about if each folder on your path was on a new line and the paths were sorted? Well thats quite easy:

$ENV:Path.Replace('"', '').Split(';') | Sort-Object -unique

On one of my machines this outputs:

%SystemRoot%system32WindowsPowerShellv1.0
C:\Program Files\Common Files\DivX Shared
C:\Program Files\Common Files\Microsoft Shared\Windows Live
C:\Program Files\Cppcheck
C:\Program Files\Gitcmd
C:\Program Files\GNU\GnuPG\pub
C:\Program Files\GnuWin\32\bin
C:\Program Files\Javajdk1.6.0_18bin
C:\Program Files\KDiff3
C:\Program Files\Microsoft SQL Server100\DTS\Binn
C:\Program Files\Microsoft SQL Server100\Tools\Binn
C:\Program Files\Microsoft SQL Server100\Tools\Binn\VSShellCommon7\IDE
C:\Program Files\Microsoft SQL Server80\Tools\Binn
C:\Program Files\Microsoft SQL Server90\DTS\Binn
C:\Program Files\Microsoft SQL Server90\Tools\binn
C:\Program Files\mongodb-win32-i386-1.6.1\bin
C:\Program Files\mtasc-1.14
C:\Program Files\Subversion\bin
C:\Program Files\TortoiseGit\bin
C:\Program Files\TortoiseHg
C:\Program Files\TortoiseSVN\bin
C:\Program Files\Vim\vim72
C:\Program Files\WinMerge
C:\Program Files\Zend\MySQL51\bin
C:\Program Files\Zend\ZendServer\bin
C:\Program Files\Zend\ZendServer\share\ZendFramework\bin
C:\Python26\Scripts
C:\Windows
C:\Windows\system32
C:\Windows\System32\Wbem
C:\Windows\System32\WindowsPowerShell\v1.0
D:\apps\apache-ant-1.8.0bin
D:\apps\jboss-4.2.2.GAbin

So lets break this down:

  • $ENV:Path is our path as before.
  • .Replace() is string.Replace(). Some of your path folders might be quoted, removing the quotes will cause them to be properly sorted.
  • .Split(‘;’) is String.Split(). We want to turn our path into an array of strings, one folder per string.
  • | Is a pipe. This works like cmd.exe and unix piping, excet its object based, not stream based.
  • Sort-Object -unique does exactly what you think it does. It sorts the folders in your path and removed duplicates. Sometimes paths contain a folder multiple times. Cleaning up the duplicates will be addressed in a future blog post.

This one liner is good to run when you inherit someone else’s workstation, or a server setup by someone else. One limitation it has is it does not expand environment variables. Ideally I’d like it to expand %SystemRoot% to C:Windows. However, Sort-Object is case insensitive by default, like the NTFS file system, so casing issues are not a problem.

Thats all for now folks. Happy scripting!