the code monkey

Upload a directory to S3 with C#

2009-03-31T12:54:00.003-04:00

   1:  using System;

   2:  using System.IO;

   3:  using System.Threading;

   4:  using Affirma.ThreeSharp;

   5:  using Affirma.ThreeSharp.Model;

6:

   7:  namespace upload

   8:  {

   9:      class Program

  10:      {

  11:          static Semaphore fileCounter;

  12:          static IThreeSharp service;

13:

  14:          static const int MaxUploads = 6;

15:

  16:          static void Main(string[] args)

  17:          {

  18:              if (args.Length < 4)

  19:              {

  20:                  Console.Error.WriteLine("syntax: upload <aws-key> <aws-secretkey> <bucketname> <directory>");

  21:                  Environment.Exit(1);

  22:              }

23:

  24:              ThreeSharpConfig config = new ThreeSharpConfig();

  25:              config.AwsAccessKeyID = args[0];

  26:              config.AwsSecretAccessKey = args[1];

  27:              config.ServiceType = ThreeSharpServiceType.S3;

  28:              config.IsSecure = false;

  29:              config.ConnectionLimit = MaxUploads * 2;

  30:              ThreadPool.SetMaxThreads(MaxUploads, MaxUploads);

31:

  32:              service = new Affirma.ThreeSharp.Query.ThreeSharpQuery(config);

33:

  34:              string bucketName = args[2];

  35:              string directoryName = args[3].Trim();

36:

  37:              var files = Directory.GetFiles(directoryName);

  38:              fileCounter = new Semaphore(0, files.Length);

  39:              ThreadPool.QueueUserWorkItem(ReportStatus);

  40:              foreach (string file in files)

  41:              {

  42:                  BucketAddRequest add = new BucketAddRequest(bucketName);

  43:                  add.Key = Path.GetFileName(file);

  44:                  add.LoadStreamWithFile(file);

  45:                  ThreadPool.QueueUserWorkItem(StartTransfer, add);

  46:              }

  47:              for(int i=0; i<files.Length; ++i)

  48:              {

  49:                  fileCounter.WaitOne();

  50:              }

51:

  52:          }

53:

  54:          static void ReportStatus(object state)

  55:          {

  56:              while (true)

  57:              {

  58:                  Console.SetCursorPosition(0, 0);

59:

  60:                  foreach (var info in service.GetTransferInfos())

  61:                  {

  62:                      Console.WriteLine("{0} {1}/{2}", info.Key, info.BytesTransferred, info.BytesTotal);

  63:                  }

  64:                  Thread.Sleep(TimeSpan.FromSeconds(2));

  65:              }

  66:          }

67:

  68:          static void StartTransfer(object state)

  69:          {

  70:              BucketAddRequest add = (BucketAddRequest)state;

  71:              var response = service.BucketAdd(add);

  72:              fileCounter.Release();

  73:          }

  74:      }

  75:  }

DataReader Care & Safety

2007-12-13T00:49:00.001-05:00

As any good ADO developer will tell you "Make sure you close your connections!" or more plainly "Clean up after yourself". .NET has made this even easier with things like:

using (IDataReader r = cmd.ExecuteReader())
{
// logic here
}

Recently though while doing some ASP.NET control development, I was stuck on how exactly the ASP.NET library team managed to close connections in certain scenarios. Specifically with the SqlDataSource control and how was it making sure to clean up after the DataReader it uses for databinding.

Let me first illustrate a scenario though in ASp.NET 1.0 data binding:

    Repeater r = GetYourRepeater();
    SqlConnection cn = GetYourSqlConnection();
    SqlCommand cmd = new SqlCommand("SELECT * FROM Table", cn);
    cn.Open();
    r = cmd.ExecuteReader(CommandBehaviour.CloseConnection);
    r.DataSource = r;
    r.DataBind();

This code will actually clean up and close sql connection, but how? There's no "Close" method called or "Dispose" and no using statement. The fact that the connection is closed means SqlDataSource doesn't have to do anything special to make sure the DataReader is cleaned up. When I first started this investigation I didn't realise the above code would actually clean up the connection.

When I started to investigate I went right to SqlDataSourceView.ExecuteSelect which does the actual work of the data binding select. With some reflector work I saw that the DataReader was being executed with the CommandBehaviour.CloseConnection. I knew though that all this meant was that if you closed the DataReader you're underlying connection would be closed.

ExecuteSelect though just returns an IEnumerable. So I looked through various databinding controls looking for a cast to IDataReader and Close method being called, but I didn't find any. So I thought perhaps it's something with the DataReader itself and CommandBehaviour.CloseConnection. I suspected there was some code in DataReader that if you had that flag set and enumerated through the whole connection it would close the reader. The documentation for CommandBehaviour didn't say anything about this, but wierder things have happened.

So I wrote a simple program to test this:

    SqlConnection cn = new SqlConnection("Data Source=HERC;Initial Catalog=MAIN_Development;UserId=WebClientRW;Password=smooth");
    SqlCommand cmd = new SqlCommand("SELECT * FROM _EventRegPeriod", cn);
    cn.Open();
    IDataReader r = cmd.ExecuteReader();
    while (r.Read())
{
    Console.WriteLine(r["ID"]);
    }
    Console.WriteLine(r.IsClosed);
    Console.WriteLine(cn.State);

But the output of this program clearly showed both the reader and the connection were still open. What gives?? I was sure though I was on the right track. I checked again how Repeater used the data it got from the DataSource. It was using the IEnumerable interface though. Was this it? Was there some magic when you accessed a DataReader as an IEnumerable?

Yes, Virginia, there is.

When I replaced the while loop above with the following foreach loop
    foreach (IDataRecord rec in r)
    {
        Console.WriteLine(rec["ID"]);
    }

I found that both the reader and connection were closed. What was happening in GetEnumerator?? SqlDataReader.GetEnumerator was returning a DbEnumerator. what's more, DbEnumerator's constructor takes an argument called... "closeReader"! This is True if the CommandBehaviour.CloseConnection is true. DbEnumerator is the magic that closes the connection if you fully enumerate a DataReader with CommandBehaviour.CloseConnection set.

Mystery solved, and one more place I can stop worrying about closing connections. Thanks .NET!! Sometimes I think I worry too much.

MSBuild Bonanza (Part I)

2007-02-07T23:45:00.000-05:00

I have to confession to make. I am a build configuration / source control junkie. I don't know why or how it happened, but it all started long ago when I was at U of W and had coerced my way into getting access to code on some ROM 2.4 MUD. (MUD = Mult-User Dungeon, and were text based precursors to today's MMORPG ). I somehow got started on RCS, I think this was from school. This lead to me implementing RCS and then CVS on our linux server for our MUD code. Suddenly coordination was so much easier! (Well of course). This then led to setting up cvsweb, and then figuring "make" to the n-th degree so that I had automated builds for both a staging and production environments and automated backups etc etc.

Fast forward to the relative present at my current company I've lead us through three successive version control systems: from CVS to Subversion to now TFS. Hopefully we can stick with TFS for the foreseeable future. TFS certainly has enough lever's and knobs to last a configuration junkie like me a good long while.

So over the past two weeks I've attacked the TFS team build configuration for our web application. Previously we had setup a bare bones Team Build type that relied on a heavily extended Web Deployment Project to do all our custom work. Here's what I wanted:

Build compiles all class libraries and runs code analysis on them
Build pre-compiles website
Turns debugging off in the web.config (somewhat negated by (re)discovering this ASP.NET feature)
Creates install package

The install is an automated process that

installs all required prerequsites (.NET Framework 2.0, ReportViewer, other 3rd party components etc)
restores database(s)
installs other machine wide configuration files
creates website(s), including deleting old one, making sure correct asp.net version is assigned, setting correct permissions
creates shortcut(s)

To create the install involves

backing up all required databases
updating the web.config for install specific settings
packaging all the websites and prerequiste projects together in a self-extracting zip

Deploy the project to the staging environment

I've now refactored almost all of this functionality into a set of .targets files that I've deployed to the $(MSBuildExtensionsPath). This would then let me generate a Team Build Type for any of our website projects where the TFSBuild.proj file would look like the following:

<?xml version="1.0" encoding="utf-8"?>

<Reference>ProjectA</Reference>

</PropertyGroup>

</Project>

I would only have to change the $(Reference) property and all the other Team Build properties could be derived from this. Great! I've got DRY down, it looks well factored. I was happy. But then I started getting messages like these:

Drop location is not defined in the Buld Type X

And I knew the drop location was defined as when I did a Desktop Build I could get the value just fine. What was TFS's problem? Well I discovered that Team Build does NOT run your TFSBuild.proj through MSBuild. In fact, it uses XmlSerializer to parse the file. Which means the DropLocation and BuildDirectoryPath are not in fact evaluated as properties. Given this I don't understand why Microsoft did not just have them as other elements under the ProjectExtensions element. This would have been less misleading.

Well that's the negative side of things. On the positive I wrote a SqlExecute MSBuild task I'm pretty happy with that I'll share here in another post. Also upcoming I should post about installations, and how we wrote our install script which is basically MSBuild and some ideas I have on that.

UPDATE: Buck Hodges replied to my MSDN forum post and confirmed my findings. Also pointed at the Orcas Team Build plans.

iPhone - The Honeymoon's over... already

2007-01-12T12:59:00.000-05:00

When I first saw the iPhone , I was really impressed, or perhaps a more accurate term in hindsight would be infatuated. It was very cool to get all these things into one device. I like having a mini-browser/email/phone all together in my blackberry. Merging in the iPod was great news. But then I started getting the bad news:

Strike #1

6 - 8 GB of memory. Plenty for anything except for a media library, especially if you intend to store video. I hate having to choose what I want, which is why I have the 60 GB iPod video. Now this isn't such a big deal as 8GB still leaves a lot of flexibility and it seems to work for all those Nano users.

Strike #2

Price - $600 USD, which translates to 706 CAD. Can I trade in my iPod Video.. oh wait, not nearly the same memory, and likely not anyways.

Strike #3

Found this out today. I can't write software against it. I'll have this really cool, but closed device. Drive's me crazy. So time to go look some more at the other options.

More coverage on Apple crappiness here.

ASP.NET Custom Control Properties

2007-01-11T16:21:00.000-05:00

When authoring a custom ASP.NET control, I believe it a best practice to store public properties in the control's viewstate. This applies to both controls inheriting from WebControl and to user controls.

In my many, many journeys with Refelector I've noticed a common pattern in the BCL which I've been now using for my public control properties:

[System.ComponentModel.DefaultValue("MyDefault")] public string MyProperty { get { return ViewState["MyProperty"] as string ?? "MyDefault"; } set { if (MyProperty != value) ViewState["MyProperty"] = value; } }

I've put together a snippet for this which makes adding these properties (something I find myself frequently doing) much quicker.

The only thing to remember when using the snippet is if ou use a value type (int, Decimal, etc) you need to change the inner cast to use Nullable, which is really easy in C#. Just type a "?" mark as the snippet places the cursor right where you need it when you finish.

[System.ComponentModel.DefaultValue(42)] public int TheMeaningOfLife { get { return ViewState["TheMeaningOfLife"] as int? ?? 42; } set { if (TheMeaningOfLife != value) ViewState["TheMeaningOfLife"] = value; } }

You can download the snippet here.

Database Maintenance Bible : The Book of Genesis

2007-01-11T14:34:00.000-05:00

In the beginning there was darkness in the database server and no backups to speak of therein.
And the great DBA spoke into the server and said "Let there be an Administration database" and so an Administration database was created.
Then the DBA spoke an said, "let the Administration database hold stored procedure code of varying types" and the stored procedures were created. The DBA looked at the stored procedures and said they were good.
Then the DBA said, "Let the stored procedures be called at distinct intervals and run so that knowledge is maintained". So 3 jobs were created and the DBA named them according to the knowledge they maintained, calling them System Backup, User DB Backup and Tlog Backups. And the DBA said it was good.
Then the DBA breathed life into the jobs and said "be fruitful and don't break dammit". And so they went and ensured the integrity of the server.
Then the DBA said, "let us keep several copies of the knowledge spanning 7 spins of the celestial globe known as "earth" in case the whole server goes to the shits". And so it was... and the DBA said it was very good.
- Sean Walker

I asked graciously for a maintenance plan on our companies TFS server and this is what I received in response.

WorkItemTrackingCacheRoot

2006-11-29T15:37:00.000-05:00

In writing some web applications that access Team Foundation Server's work item tracking API I've run into permission issues as exceptions are thrown when trying to access the work item cache directory. This defaults to a directory in c:\documents and settings\\local settings\ . Some pages I've found recommend that you use a user with appropriate permissions. I would like to use the ASPNET user. And after some Reflector investigation discovered there is an appSetting called "WorkItemTrackingCacheRoot" that the TFS client object model uses to determine this directory. So you can override it in your application and TFS will use this.

<add key="WorkItemTrackingCacheRoot" value="your-absolute-directory-path" />

Use at your own risk!

Brain Explosions

2006-08-02T23:09:00.000-04:00

So am working on putting together the process for the big project. Amalgamating experience and a dozen books and tens of articles on design, methodology, architecture and requirements is making my grey matter hurt... and occasiionally go on side tangents. I'm also faced reently with a plethora of interviewing to do at work. Now here's a skill I need some more work on. Two non-verbal people who don't know each other are not prone to entering an easy dialogue. And technical assessment is hard of marginal cases. I think we just need better candidates. But on to the crazy ideaof this post. I thought I might apply to some technical jobs to just see their interviewing style. Of course who's to say this would be of any value to me, as I've never been through what I thought was a particularly brilliant interview. Not to mention I'd be wasting said Acme Incorporated's time. Yes, not only crazy, but a stupid idea. No easy path to easy interviewing skills. Back to book's and the school of hard knocks.

Rocketboomed

2006-07-13T05:45:00.000-04:00

While trolling procrastination central I noticed a wired news article on rocketboom. Intrigued I finally discovered this amazing vlog. Though the feeling I have now is much akin to discovering an amazing book series, only to discover the author dead with no hope of finishing. Ahhh.. so now I'm satiating myself on the archives.

Billy Talent II

2006-07-11T00:14:00.000-04:00

So I'm browsing around iTunes, because I don't want to/can't seem to focus on work this evening.. and happen upon Billy Talent II in the top 10 of today's albums.

It quickly brought to mind Jamal's rant on Billy Talent, and I do wish I could post it verbatim as a "user review", because well, it was just so damn funny. Jamal is like having our own personal Dennis Miller. All I can recall was it involved him taking the album out and kicking it around the driveway for awhile. Looses something I guess in the translation.

Personally, I liked the album, but yes, the band does look like a complete collection of dorks. But I can not recall one time where i based my taste of music on the appearance of the musician.

SchemaImporterExtension = Giant Disaster

2006-06-23T12:03:00.000-04:00

And I wondered why more people hadn't done this. SchemaImporterExtension sounds like a great idea at first. Let me define custom types for the xml deserialization process of a web service.
I even managed to get it running without installing the library in the GAC and within the ASP.NET compilation model. Your milage may vary in a true environment as this was only on my development machine running Windows XP and IIS 5.1. A key requirement seemed to be having the assembly containing the SIE signed and the reference to it in web.config using the full assembly name reference (with publicKey etc.)
So this was all working great, but I had the expectation that I could use the types I used on the server side on the client-side. No such luck, the whims of the xml deserializer foiled me at every turn. Types completely disappeared, especially the root type of return values. Types containing collections of types disappeared becoming ArrayOfBlahType. Argh. I'm sure there was some XmlAttribute magic and some type structure manipulation I could do to get everything working but it was well on it's way to being very clunky.
Several hours later, I'm back to the much cleaner method of just letting wsdl proxy generator make up it's own types so I'm only "sharing schema, not types" across the service boundary. As both the service producer and consumer this is mildly frustrating.. but I hear Don Box should be happy.

Fickle Flickr

2006-03-30T23:55:00.000-05:00

So, went over to the dark side today. That is, I signed up for a flickr pro account to commemorate the Developer Challenge we had. Not abandoning smugmug, but flickr has a very different community to it. So we'll see how this goes.

Their uploadr tool though doesn't like the JPEG images from my Pentax Option SV. Ergh. Strike one against Flickr.

Writing Documentation is no fun

2006-03-22T01:46:00.000-05:00

Perhaps I am spoiled in that I can call the majority of what I do for a living "fun". I throughly enjoy the work I do and find it challenging. I count myself lucky every day for this. But writing documentation is not something I think I'm good at, and I think from this stems the non-funness factor. In fact when faced with such a task, every procrastination tool at my disposal comes out, hence.. blogging a last resort. Well I should stop procrastinating and get back to specification writing.

A new beginning (again!)

2006-03-18T17:51:00.000-05:00

Well here's a temporary start to a blog... for now anyways. If I ever get my page up and running again I guess I'll have to move back there. Having some trouble with Community Server, need to recompile it I guess...