Fixing Missing NuGet Packages

I have encountered this problem more than one time. Somehow, maybe when merging changes between branches, there are references in .NET projects to assemblies in NuGet packages that are not present in packages.config, which in turn can result in issues when updating packages to later versions. I put together a PowerShell script to detect this issues. It searches all .csproj files in subfolders to the current one.

# Get reference hintpath to packages:
$pattern = "<HintPath>.*Packages\\(?<id>(?:[a-zA-Z]+\.?)+)\.(?<version>(?:\d+\.)+\d+)\\.*</HintPath>"

Get-ChildItem -Path . -Filter "*.csproj" -File -Recurse | foreach {
    $projFilename = $_.FullName
    Write-Output "`r`n*** $projFilename ***"
    $packageFilename = "$($_.DirectoryName)\packages.config"
    if ([System.IO.File]::Exists($packageFilename))
    {
        $projtext = Get-Content -Path $projFilename
        $projtext | foreach {
            if ($_ -imatch $pattern)
            {
                $id = $Matches.Item("id")
                $version = $Matches.Item("version")
                Write-Output "$($Matches[0])"
                $foundInPackages = Select-String -Pattern "id=""$id"" version=""$version""" -Path $packageFilename -Quiet
                if ($foundInPackages) {
                    Write-Output (Select-String -Pattern "id=""$id"" version=""$version""" -Path $packageFilename).Line
                } else {
                    Write-Warning "Not found in package file!"
                }
            }
        }
    } else {
        Write-Warning "There is no package file at $packageFilename!"
    }
}

Uninstalling NuGet Packages the Hard Way

We were recently in a situation where TeamCity had “cleaned” some NuGet packages that some of our projects depended on. Now they didn’t build in TeamCity. I thought this would be easy to fix: Just upgrade them to the latest version, or uninstall the old and install the new one. But bot of these methods failed, because the package manager failed to build the dependency graph, because it didn’t find the packages. So this was a catch 22!

It turned out that it was possible to manually remove references to these packages by editing the packages.config files, and then run Install-Package. I spent time to automate this, and came up with the following PowerShell script:

$pattern = 'id=”Neonstingray.Nettv4.DataTransfer” | id="Neonstingray.Nettv4.Domain" | id="Neonstingray.Nettv4.Configuration"’
Get-ChildItem -Filter packages.config -Recurse | Select-String -Pattern $pattern | Group Path | ForEach-Object { (Get-Content -Path $_.Name) | where { $_ -NotMatch $pattern } | Set-Content -Encoding UTF8 -Path $_.Name }
$source = "https://<myhost>/httpAuth/app/nuget/v1/FeedService.svc"
install-package Neonstingray.Nettv4.DataTransfer -projectname Valtech.Ready4Air.Ingest.UnitTests -source $source
install-package Neonstingray.Nettv4.DataTransfer -projectname Valtech.Ready4Air.Ingest.Ooyala -source $source
install-package Neonstingray.Nettv4.DataTransfer -projectname Valtech.Ready4Air.Ingest.Ooyala.IntegrationsTests -source $source
install-package Neonstingray.Nettv4.DataTransfer -projectname Valtech.Ready4Air.Ingest.Ooyala.UnitTests -source $source
install-package Neonstingray.Nettv4.DataTransfer -projectname Valtech.Ready4Air.Ingest.PartnerApi -source $source
install-package Neonstingray.Nettv4.DataTransfer -projectname Valtech.Ready4Air.Ingest.PartnerApi.IntegrationTests -source $source
install-package Neonstingray.Nettv4.DataTransfer -projectname Valtech.Ready4Air.Ingest.PartnerApi.UnitTests -source $source

Troubleshooting Memory Leaks in .NET

I recently had problems with a memory leak in an ASP.NET application. I learned some useful tools thanks to Fredrik Haglund.

The first one is “handle”, which is part of the Sysinternals suite. With

handle -s -p <pid>

you can display the number of handles for a given process. This could be useful for applications that e.g. open files but do not close them.

The second one is “procdump”, which is also part of the Sysinternals suite. This will dump the entire process by first creating a copy:

procdump64 -ma -r <pid>

This dump can then be analyzed using the Debug Diagnostics Tool from Microsoft.

Processing Requests and Responses with HttpClient

In a recent project, I wanted to log outgoing requests and incoming responses using my own (JSON) format. This can be accomplished using a custom handler derived from DelegatingHandler. Here is an implementation that can be used in e.g. a console program:

        internal class LoggingHandler : DelegatingHandler
        {
            private static int _requestNumber = 0;
            private static readonly string LogFolder;

            static LoggingHandler()
            {
                LogFolder = Path.Combine(Path.GetTempPath(), "Log");
                if (Directory.Exists(LogFolder))
                {
                    foreach (var file in Directory.GetFiles(LogFolder))
                    {
                        File.Delete(file);
                    }
                }
                else
                    Directory.CreateDirectory(LogFolder);
            }

            protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
            {
                try
                {
                    var response = await base.SendAsync(request, cancellationToken);

                    if (request.Method == HttpMethod.Post)
                    {
                        var logEntry = new LogEntry
                        {
                            Request = new Request
                            {
                                Method = request.Method,
                                Uri = request.RequestUri,
                                Headers = request.Headers,
                                Content = await GetContent(request.Content)
                            },
                            Response = new Response
                            {
                                Headers = response.Headers,
                                StatusCode = response.StatusCode,
                                Content = await GetContent(response.Content)
                            }
                        };

                        var logEntryString = JsonConvert.SerializeObject(logEntry, Formatting.Indented);
                        var parts = request.RequestUri.AbsolutePath.Split('/');
                        string controller = parts.Length >= 6 ? parts[5] : "";
                        string path = Path.Combine(LogFolder, string.Format("{0:0000}0_{1}_{2}_{3}.json", ++_requestNumber, logEntry.Request.Method, controller, logEntry.Response.StatusCode));
                        File.WriteAllText(path, logEntryString, Encoding.UTF8);
                        System.Diagnostics.Debug.WriteLine("Wrote {0:N0} characters to {1}", logEntryString.Length, path);
                    }

                    return response;
                }
                catch (Exception ex)
                {
                    logger.Fatal(ex);
                    throw;
                }
            }

            private static async Task<object> GetContent(HttpContent content)
            {
                try
                {
                    if (content == null)
                        return null;
                    string s = await content.ReadAsStringAsync();
                    s = s.Trim();
                    if (s.StartsWith("["))
                        return JArray.Parse(s);
                    else if (s.StartsWith("{"))
                        return JObject.Parse(s);
                    else
                        return s;
                }
                catch (Exception ex)
                {
                    return ex.Message;
                }
            }
        }

Usage is simple:

var httpClient = new HttpClient(new LoggingHandler());

However, the above implementation didn’t work in a web application. The call to base.SendAsync never returns for some reason. I suppose it has to do with a different threading model. Luckily, I found the solution, to use Task.ContinueWith, in the following blog post: http://byterot.blogspot.se/2012/05/aspnet-web-api-series-messagehandler.html. Here is the alternative implementation.

        protected override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
        {
            try
            {
                _logger.Debug($"{GetType().Name} sending request  '{request.RequestUri}'...");
                return base.SendAsync(request, cancellationToken).ContinueWith(task =>
                {
                    _logger.Debug($"Task status is {task.Status}");
                    var response = task.Result;
                    _logger.Debug($"{GetType().Name} got response from'{request.RequestUri}'.");

                    if (request.Method == HttpMethod.Post)
                    {
                        var logEntry = new LogEntry
                        {
                            Request = new Request
                            {
                                Method = request.Method,
                                Uri = request.RequestUri,
                                Headers = request.Headers,
                                Content = GetContent(request.Content)
                            },
                            Response = new Response
                            {
                                Headers = response.Headers,
                                StatusCode = response.StatusCode,
                                Content = GetContent(response.Content)
                            }
                        };

                        var logEntryString = JsonConvert.SerializeObject(logEntry, Formatting.Indented);
                        var controller = GetController(request);
                        string path = Path.Combine(LogFolder, $"{++_requestNumber:0000}0_{logEntry.Request.Method}_{controller}_{logEntry.Response.StatusCode}.json");
                        _logger.Debug($"Writing {logEntryString.Length:N0} characters to {path}...");
                        File.WriteAllText(path, logEntryString, Encoding.UTF8);
                        _logger.Debug($"Wrote {logEntryString.Length:N0} characters to {path}.");
                    }

                    return response;
                }, cancellationToken);
            }
            catch (Exception ex)
            {
                _logger.Fatal(ex.Message, ex);
                throw;
            }
        }

        private static object GetContent(HttpContent content)
        {
            try
            {
                if (content == null)
                    return null;
                string s = content.ReadAsStringAsync().Result;
                s = s.Trim();
                if (s.StartsWith("["))
                    return JArray.Parse(s);
                else if (s.StartsWith("{"))
                    return JObject.Parse(s);
                else
                    return s;
            }
            catch (Exception ex)
            {
                return ex.Message;
            }
        }

Entity Framework Wonderland: Migrations and Seeding

This post is a practical dive into Entity Framework 6 migrations and seeding in an ASP.NET MVC 5 web application.

Initial Setup

Suppose we have the following model:

    public class Entity1
    {
        public int Id { get; set; }
        public DateTime Created { get; set; }
        public string Value { get; set; }
    }

    public class MyContext : DbContext
    {
        public MyContext() : base("MyDb")
        {
        }

        public virtual DbSet<Entity1> Entity1 { get; set; }
    }

Add a new MVC 5 controller with views, using Entity Framework  and call it Entity1Controller.

Now, if you run the web application and go to http://localhost:xxxxx/Entity1, the database should be automagically created.

Making a Model Change

Suppose I add another string property (Value2) to Entity1 and hit F5 to run the web application again, I get an exception:

An exception of type ‘System.InvalidOperationException’ occurred in EntityFramework.dll but was not handled in user code

Additional information: The model backing the ‘MyContext’ context has changed since the database was created. Consider using Code First Migrations to update the database (http://go.microsoft.com/fwlink/?LinkId=238269).

OK, let’s fix that. Open Package Manager Console and type

Enable-Migrations

This will add two files to the Migrations folder:

  • <date and time>_IniticalCreate.cs
  • Configuration.cs

Configuration looks like this:

    internal sealed class Configuration : DbMigrationsConfiguration<Models.MyContext>
    {
        public Configuration()
        {
            AutomaticMigrationsEnabled = false;
            ContextKey = "EFMigration1.Models.MyContext";
        }

        protected override void Seed(Models.MyContext context)
        {
            //  This method will be called after migrating to the latest version.

            //  You can use the DbSet<T>.AddOrUpdate() helper extension method 
            //  to avoid creating duplicate seed data. E.g.
            //
            //    context.People.AddOrUpdate(
            //      p => p.FullName,
            //      new Person { FullName = "Andrew Peters" },
            //      new Person { FullName = "Brice Lambson" },
            //      new Person { FullName = "Rowan Miller" }
            //    );
            //
        }
    }

Let’s add a seed:

            context.Entity1.Add(new Models.Entity1
            {
                Created = DateTime.Now,
                Value = "foo",
                Value2 = "bar"
            });

Now hit F5 again. The same exception again! Well, looking at the initial migration, it is understandable, because it doesn’t contain the newly added property. Let’s create a new migration:

Add-Migration Value2

Hit F5 again. Boom! The same exception again. A simple solution now would be to use the Update-Database command. If we do that, the Value2 column is added and the seed data is inserted. Now F5 works.

This is fine for a development environment, but for testing and production it is IMHO not suitable. We have two options:

  1. Configure the application to migrate automatically as outlined in Code First Migrations on MSDN.
  2. Use Migrate.exe

Configuration

Migrate to Latest Version

Enabling automatic migration involves one line of code:

Database.SetInitializer(new MigrateDatabaseToLatestVersion<MyContext, Configuration>());

In this case, I choose to Create a new class EfConfig in the App_Start folder, and call it from Global.asax.cs. This will run migrations if necessary, and seed every time.

Note: Don’t confuse this with the AutomaticMigrationsEnabled property, which will effectively disable versioning of your database.

Create Database If Not Exists

As an aside note, the default initializer is CreateDatabaseIfNotExists, which means that the database is created if it does not exists, but not upgraded when the model changes. You can also set this explicitly, but there is no configuration parameter. So the question is: How do you seed in this case? The solution is to create a custom initializer that derives from CreateDatabaseIfNotExists and move the seeding to that:

    internal class MyInitializer : CreateDatabaseIfNotExists<Models.MyContext>
    {
        protected override void Seed(Models.MyContext context)
        {
            context.Entity1.Add(new Models.Entity1
            {
                Created = DateTime.Now,
                Value = "Initializer seed",
                Value2 = "baz"
            });
        }
    }

Then, use this initializer instead:

Database.SetInitializer(new MyInitializer());

Since migration is only done once, seed is only called once.

Note however that if you do Update-Database, it is the Seed method in Configuration that is used!

Use Migrate.exe

If you want more control of the migration process, you could manually, or as part of the deployment process, call Migrate.exe, which is located in the tools subfolder of the Entity Framework package:

pushd
copy ..\packages\EntityFramework.6.1.3\tools\migrate.exe bin
cd bin
migrate.exe EFMigration1.dll /startupConfigurationFile=..\Web.config /verbose
del migrate.exe
popd

This will run migrations if necessary, and seed.

Integration Testing a Web App with Forms Authentication

This turned out to be harder than I thought. I had to not only figure out how to post form fields using .Net code but also handle anti-forgery cookies and hidden fields.

When you request a protected URL, the browser is redirected to the login page. The header of this response contains a Set-Cookie header and the body a hidden input. When the login form is posted, these must be sent back and match.

Here is some example code that does all this:

        private string LogIn(string startUrl)
        {
            try
            {
                //var proxy = new WebProxy("127.0.0.1", 8888); // Fiddler
                IWebProxy proxy = null;

                // Make the request. This will result in the login page being returned.
                var request = WebRequest.CreateHttp(startUrl);
                request.Proxy = proxy;
                var response = (HttpWebResponse)request.GetResponse();
                Assert.That(response.StatusCode, Is.EqualTo(HttpStatusCode.OK));
                
                // Get request verification token cookie:
                string setCookie = response.Headers["Set-Cookie"];
                Assert.That(setCookie, Is.Not.Null);
                var requestVerificationTokenCookie = setCookie.Split(';')[0];

                // Get form action to be posted to later:
                string responseBody = new StreamReader(response.GetResponseStream()).ReadToEnd();
                var regex = new Regex("form action=\"([^\"]*)\"");
                Assert.That(regex.IsMatch(responseBody));
                string formAction = regex.Match(responseBody).Groups[1].Value;
                Assert.That(formAction, Is.Not.Null);

                // Get request verification token form field:
                regex = new Regex("<input name=\"__RequestVerificationToken\" .* value=\"([^\"]*)\"");
                Assert.That(regex.IsMatch(responseBody));
                string requestVerificaitonToken = regex.Match(responseBody).Groups[1].Value;
                
                // Post the login form:
                var baseUri = new Uri(Properties.Settings.Default.AppUrl);
                var localUri = new Uri(formAction, UriKind.Relative);
                var loginUri = new Uri(baseUri, localUri);
                request = WebRequest.CreateHttp(loginUri);
                request.Proxy = proxy;
                request.Method = HttpMethod.Post.Method;
                request.Headers.Add(HttpRequestHeader.Cookie, requestVerificationTokenCookie);
                string postData = $"__RequestVerificationToken={requestVerificaitonToken}&UserId={Properties.Settings.Default.UserId}&Password={Properties.Settings.Default.Password}";
                byte[] byteArray = Encoding.UTF8.GetBytes(postData);
                request.ContentType = "application/x-www-form-urlencoded";
                request.ContentLength = byteArray.Length;
                Stream dataStream = request.GetRequestStream();
                dataStream.Write(byteArray, 0, byteArray.Length);
                dataStream.Close();
                request.AllowAutoRedirect = false; // Prevents redirecting back to returnUrl.
                response = (HttpWebResponse) request.GetResponse();
                Assert.That(response.StatusCode, Is.EqualTo(HttpStatusCode.Found)); // Redirect to original url
                setCookie = response.Headers["Set-Cookie"];
                Assert.That(setCookie, Is.Not.Null);
                var sessionCookie = setCookie.Split(';')[0];
                return sessionCookie;
            }
            catch (WebException ex)
            {
                Assert.Fail(ex.ToString());
                return null;
            }
        }

Now, this can be used in a test like this:

        [Test]
        public void Test1()
        {
            string url = "http://localhost/...";
            string sessionCookie = LogIn(url);
            var response = Request(sessionCookie, url);
            Assert.That(response, Contains.Substring("(Some string that is always in correct response)"));
        }

        private static string Request(string sessionCookie, string url)
        {
            var webClient = new WebClient();
            webClient.Headers.Add(HttpRequestHeader.Cookie, sessionCookie);
            string response = webClient.DownloadString(url);
            return response;
        }