.NET
GDI+ Image.FromFile has a problem – here’s how to fix it
Jul 30th
In GDI+ you can call Image.FromFile to load an image from a file. BUT there are several issues with this call, the biggest being that GDI+ will keep the file open long after you are done with it. Here is an image loader that gets around this issue.
If you are running a high volume web site, and your images are on a SAN you’ll find this technique necessary to prevent an eventual exhaustion of filehandles.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Drawing;
using System.IO;
using System.Data;
namespace Utility
{
public static class ImageLoader
{
// This isn’t going to help much – you’ll run out of memory anyway on very large images – but if you are keeping several in memory it might …
public const int MaximumImageDimension = 10000;
///
/// Method to safely load an image from a file without leaving the file open,
/// also gets the size down to a manageable size in the case of HUGE images
///
/// An Image – don’t forget to dispose of it later
public static Image LoadImage (string filePath)
{
try
{
FileInfo fi = new FileInfo(filePath);
if (!fi.Exists) throw new FileNotFoundException(“Cannot find image”);
if (fi.Length == 0) throw new FileNotFoundException(“Zero length image file “);
// Image.FromFile is known to leave files open, so we use a stream instead to read it
using (FileStream fs = new FileStream(filePath, FileMode.Open, FileAccess.Read))
{
if (!fs.CanRead) throw new FileLoadException (“Cannot read file stream”);
if (fs.Length == 0) throw new FileLoadException(“File stream zero length”);
using (Image original = Image.FromStream(fs))
{
// Make a copy of the file in memory, then release the one GDI+ gave us
// thus ensuring that all file handles are closed properly (which GDI+ doesn’t do for us in a timely fashion)
int width = original.Width;
int height = original.Height;
if (width == 0) throw new DataException(“Bad image dimension width=0″);
if (height == 0) throw new DataException(“Bad image dimension height=0″);
// Now shrink it to Max size to control memory consumption
if (width > MaximumImageDimension)
{
height = height * MaximumImageDimension / width;
width = MaximumImageDimension;
}
if (height > MaximumImageDimension)
{
width = width * MaximumImageDimension / height;
height = MaximumImageDimension;
}
Bitmap copy = new Bitmap(width, height);
using (Graphics graphics = Graphics.FromImage(copy))
{
graphics.DrawImage(original, 0, 0, copy.Width, copy.Height);
}
return copy;
}
}
}
catch (Exception ex)
{
ex.Data.Add(“FileName”, filePath);
throw;
}
}
}
Weather Forecasting for Home Automation
Jul 25th
In the USA we are lucky to have the NOAA and their excellent web service that can provide a detailed weather forecast for any location (specified by latitude and longitude). Using this service my home automation system maintains a detailed, regularly updated local weather forecast object which can be queried easily by any other object in the home.
Click to enlarge
On the weather page you can view all of the forecast information it collects. Of interest the graph in the lower right compares the forecast from NOAA with the actual recorded temperature. In this case you can see just how accurate the forecast has been for the last 24 hours. Normally it runs almost this close but with occasionally there is a significant discrepancy caused by the bizarre ‘convergence zone’ we live in where Pacific weather patterns split around the Olympic mountains and then recombine over Seattle in somewhat unpredictable ways.
Unlike most home automation systems that have fairly limited if-the-else or table-driven approaches to defining the home logic (often limited to only the current value of any variable), my system has control structures that include statistical functions over temporal data allowing analysis of past data (e.g. average temperature in the last day), or in the case of the weather forecast, future values, e.g. expected temperature one hour from now found using interpolation.
var oneHourForNow = DateTime.Now.AddHours(1);
double outsideForecastOneHourFromNow = weatherService.ApparentTemperatureHourly.ValueAtTimeWithLinearInterpolation(oneHourForNow);
This forward looking view at the weather allows for features like garden sprinklers that don’t turn on when it’s going to rain or HVAC that skips heating cycles when the forecast predicts warm weather later today.
In addition the house is able to issue a detailed local forecast over the speakers when it wakes you up in the morning. Somewhat uniquely the forecast it generates is a relative weather forecast comparing yesterday with today, or today with tomorrow. It might for example say “Today will be much warmer that yesterday” which is a whole lot less words than a normal weather forecast!
The house also has Natural Language Generation (NLG) features which are able to summarize a group of temporal series into distinct ranges allowing it for instance to highlight which the best times of the day are to be outside:-
Monday : excellent from dawn at 5:34 AM until 11:36 AM; hot from 11:36 AM to 2:00 PM; too hot from 2:00 PM to 7:00 PM; hot until sunset at 8:53 PM.
ASP.NET Charts
Incidentally, all of the graphs are rendered using the .NET Charting Control. The graph object is instantiated on the server, all the lines and axes are added to it, then it is serialized and sent over TCP to the web server using WCF. On the web server it is rendered as a PNG file using an Action method that takes size parameters allowing any size graph to be shown on any page. Here’s the MVC code that takes the stream from WCF, loads the Chart and then delivers it as a PNG.
Chart chart = new Chart();
chart.Serializer.Load(ms);
MemoryStream ms2 = new MemoryStream();
chart.SaveImage(ms2);
return File(ms2.GetBuffer(), @"image/png");
Sequential Logic Blocks – compared to the Reactive Framework
Jul 19th
One of the features of my home automation system is extensions to the C# language that make it easy to define complex logical and temporal behaviors. These behave somewhat like the new Reactive Extensions in .NET but with some key differences which I will explain below. I developed these extensions before Reactive Framework was released and have recently been looking at Rx to see if I could combine my ideas with Rx but because of the differences explained below I haven’t been able to do that.
But, before explaining why, let’s first look at an example:-
FirstFloor.Kitchen.GoesOff.Provided((dt) => Entrance.VisitorCountThisEvening.Count < 2).TurnOff(Aquarium.Light);
This means that if the kitchen goes not occupied (off) provided we have less than two visitors this evening, then turn off the aquarium lights.
Here’s a more complex example where we create an intermediate logic element called ‘activityInKitchen’ using the .Or method, and then based on that activity we decide whether to announce that the fish have not been fed. The decision uses a combination of pulse stretching, repeats (Every) and Then which fires only if a particular sequence is followed within a given time window. Finally it uses the Do method which fires off an Action.
SensorDevice activityInKitchen =
Kitchen.KitchenFloor.Or(Kitchen.MotionSensor, Kitchen.BackDoorToGarage, Kitchen.BreakfastBarFloor, Kitchen.Phone);
activityInKitchen
.ProvidedNot(Home.DinnerGuests)
.PulseStretch(16 * 60) // Make it continuous
.Every(60 * 60) // Once every 1 hour
.Then(activityInKitchen, 15*60)
.Do("Announce fish are hungry", (sender) =>
{
Kitchen.Aquarium.AnnounceFishHungry(Kitchen.MediaZone);
});
As you can see the use of a fluent syntax allows for a very natural, almost english-like definition of complex temporal expressions. But that’s not all it allows…
The sequential logic blocks created when you call one of the many methods like .Then, .Or, .Every, … are actual objects created within the hierarchical structure of the house and the objects used to represent it. This means that they take part (automatically) in all of the features provided by a base house object including logging, browsing, and most importantly persistence.
Because these sequential logic blocks are persistent you can have very long running events like .EveryDays(2) and even if the entire system is shut down and restarted it will come back in the correct state and all the necessary delays and timers will still be working.
Another key benefit of these sequential logic blocks compared to, say, the .NET Reactive Framwork is that they form a chain that can be traversed in either direction: forward as an event propagates through, or in reverse to determine why a particular action was taken. This means that my home automation system is able to explain why it turned the lights on or off. Currently all of these decisions are logged on the per-object logging system which means you can inspect any room, appliance, light, etc. and see everything that happened to it and the reasons why.
Hallway became occupied, caused by Motion sensor
Set brightness on Aisle lights in Kitchen to 20% because Leaving office late at night
Compare this to other home automation systems from major vendors. They contain fairly simple logic, often expressed in a tabular or grid format with limited if-the-else type logic and when something happens there is often no record as to what happened and, worse, there is absolutely no record as to why it happened.
Any sufficiently complex system will have unexpected behavior, the difference here is that when it does something odd I can easily look at the logging and see the explanation as to why it happened.
Lengthening short Urls in C#
Jul 5th
If you deal with Twitter APIs, sooner or later, you’ll probably want to lengthen those short URLs back to long URLs. There’s a couple of services you can use to do this but it’s really quite easy so you might as well do it yourself. The code below follows each redirect until it reaches a real page (or an error), it then returns the last URL it finds.
private string UrlLengthen(string url)
{
string newurl = url;
bool redirecting = true;
while (redirecting)
{
try
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(newurl);
request.AllowAutoRedirect = false;
request.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3 (.NET CLR 4.0.20506)";
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
if ((int)response.StatusCode == 301 || (int)response.StatusCode == 302)
{
string uriString = response.Headers["Location"];
Log.Debug("Redirecting " + newurl + " to " + uriString + " because " + response.StatusCode);
newurl = uriString;
// and keep going
}
else
{
Log.Debug("Not redirecting " + url + " because " + response.StatusCode);
redirecting = false;
}
}
catch (Exception ex)
{
ex.Data.Add("url", newurl);
Exceptions.ExceptionRecord.ReportCritical(ex);
redirecting = false;
}
}
return newurl;
}
Recent reading material on .NET and software development in general
May 22nd
Links I’m reading up on at the moment: Memory Mapped Files in .NET 4.0
An ontology triple (quad) store for RDF/OWL using Entity Framework 4
May 12th
This weeks side-project was the creation of an ontology store using Entity Framework 4. An ontology store stores axioms consisting of Subject, Predicate, Object which are usually serialized as RDF, OWL, N3, … Whereas there’s lots of details about these serialization formats, the actual mechanics of how to store and manipulate them was somewhat harder to come by. Nevertheless, after much experimentation I came up with an Entity Model that can store Quads (Subject, Predicate, Object and Meta) or Quins (Subject, Predicate, Object, Meta, Graph). The addition of Meta allows one Axiom to reference another. The addition of Graph allows the store to be segmented making it easy to import some N3 or RDF into a graph, then flush that graph if it is no longer needed or if a newer version becomes available.
The store is currently hooked up to an Euler reasoner that can reason against it, lazily fetching just the necessary records from the SQL database that backs the Entity Model.
Here’s the EDMX showing how I modeled the Ontology Store:
The relationship could not be changed because one or more of the foreign-key properties is non-nullable
May 9th
Today’s cryptic Entity Framework message …
The operation failed: The relationship could not be changed because one or more of the foreign-key properties is non-nullable. When a change is made to a relationship, the related foreign-key property is set to a null value. If the foreign-key does not support null values, a new relationship must be defined, the foreign-key property must be assigned another non-null value, or the unrelated object must be deleted.
The reason for this was a simple override to GetHashCode(). Entity Framework seems somewhat pedantic about how hash codes work.
A simple redirect route handler for ASP.NET 3.5 routing
Apr 20th
ASP.NET 3.5 Routing is a very powerful tool not just for registering routes for newer ASP.NET MVC applications but also for adding SEO friendly routes to older Webforms (ASPX) applications, or for routing multiple URLs to a single page. But that’s not all it can do. You can create your own IRouteHandler and then have complete control over what to do with any incoming HttpRequest.
Here for example is a way to do a permanent redirect when a given route is matched. To use it you might, for example, do:-
routes.Add(new Route("sample.aspx", new RedirectRouteHandler("/home/start")));
Here is the RedirectRouteHandler that can turn any request into a 301 redirect for you:-
/// <summary>
/// Redirect Route Handler
/// </summary>
public class RedirectRouteHandler : IRouteHandler
{
private string newUrl;
public RedirectRouteHandler(string newUrl)
{
this.newUrl = newUrl;
}
public IHttpHandler GetHttpHandler(RequestContext requestContext)
{
return new RedirectHandler(newUrl);
}
}
/// <summary>
/// <para>Redirecting MVC handler</para>
/// </summary>
public class RedirectHandler : IHttpHandler
{
private string newUrl;
public RedirectHandler(string newUrl)
{
this.newUrl = newUrl;
}
public bool IsReusable
{
get { return true; }
}
public void ProcessRequest(HttpContext httpContext)
{
httpContext.Response.Status = "301 Moved Permanently";
httpContext.Response.StatusCode = 301;
httpContext.Response.AppendHeader("Location", newUrl);
return;
}
}
Note: I’m not saying this is the best or only way to handle this. You’ll want to look at Url Rewriting and the Application and Request Routing module for IIS7 in particular.
How to stop IIS7 from handling 404 errors so you can handle them in ASP.NET
Apr 17th
IIS7 has lots of places you could look to make this change: you might start off looking to see if it’s an advanced option on your application pool, no, so then you try looking at the web site itself and the option .NET Error Pages. That has to be it, surely! So you try every option there Mode=On, Mode=Off, Mode=Remote Only. Nothing works so you consult the help for those items only learn that “Mode” is to “Select a mode for the error pages: On, Off, or Remote Only.” You can see now why help writers at Microsoft are so well paid – who would have guessed that Mode = Remote Only sets the Mode to Remote Only!
Now you are really frustrated but luckily you landed on my blog post here where you learned that the true path to 404 happiness is a simple change to your web.config:
<system.webServer> <httpErrors errorMode="Detailed" />
Why functional programming and LINQ is often better than procedural code
Apr 15th
Functional programming is a relatively new component in the C# language. It can potentially replace for-loops in many situations with simpler code, but the question remains ‘what’s wrong with a good old for loop?’
Here are some of the reasons I think functional programming is important and in particular how LINQ can improve the readability, maintainability, and parallelizability (if there were such a word) of your code:
- Functional approaches are potentially easier to parallelize either manually using PLINQ or by the compiler. As CPUs move to even more cores this may become more important.
- Functional approaches make it easier to achieve lazy evaluation in multi-step processes because you can pass the intermediate results to the next step as a simple variable which hasn’t been evaluated fully yet rather than evaluating the first step entirely and then passing a collection to the next step (or without using a separate method and a yield statement to achieve the same procedurally).
- Functional approaches are often shorter and easier to read.
- Functional approaches often eliminate complex conditional bodies within for loops (e.g. if statements and ‘continue’ statements) because you can break the for loop down into logical steps – selecting all the elements that match, doing an operation on them, …
These days I opt for the functional syntax more often than not and fall back to for-loops when:-
A. The body of the loop contains complex logic that cannot be disentangled into a cleaner sequential application of functions and it simply easier to just write a for-loop with the complex conditional code in it.
B. The task is inherently not functional, i.e. has side effects
C. The task needs exception handling in it. Sure you can write big lambda blocks with try catch in them but at some point it becomes easier and cleaner just to use a for-loop.