21 April 2010

What I Would Like in a Programming Language (Part 1)

Compiler Intrinsics

Intrinsic functions and properties are wonderful things. All the decent C family languages have intrinsic functions like sizeof and alignof, but sometimes you need support for more. While a language designer can try to think of every possible need and expose a good intrinsic for all potential future requirements, this is ultimately a losing battle for pretty obvious reasons. It would be really great if a user could extend the intrinsic properties of the compiler with their own domain-specific needs. I am imagining the compiler gets something like an extension sheet along with the source code so that these properties can be added to the system quite trivially. It would have make it so that people could extend the compiler without actually having to recompile the compiler -- it is the culture of extension that you really care about. Of course, I have no idea how the implementation would work, but it would be nice to have.

Ultimately, this could give a system like C++ type traits that do not feel like a complete hack. Of course, C++ type traits are extremely powerful, but frankly, they were never meant to do what they now do and, of course, just don't feel right. If concepts were not dropped from the C++0x standard, we could be about halfway to a cleaner solution; as it stands, we are stuck with using type traits.

To run completely away with the idea, something on the order of having a Lisp-like macro system where you have an extra program which spits out some abstract syntax tree from the input would be totally awesome. Okay, so this feature already exists in perfect form in Lisp, but I would love to see it in other languages as well.

Unit Testing

QuickCheck

Haskell is a wonderful place to draw examples from. A framework like QuickCheck is just awesome. Because let's face it: Nobody likes writing unit tests. Now, I'm not saying that they are completely unnecessary, just that they are are pain in the ass to write. Say you have a function with the signature sqrt(x : real) : real.

If you wanted to write some unit tests for this function, you would pound away at some known values of various square roots. For brevity, I'll eliminate specifying some range of results that we consider "valid."

assert_equal(sqrt(4), 2)
assert_equal(sqrt(100), 10)
assert_equal(sqrt(2), 1.4142135)
assert_failure(sqrt(-1))

Okay, that's halfway decent and pretty clear what I mean. But let's face it: out of the limited representational power of a real (whatever that may be), I am testing a very pathetic subset of all the possibilities. There might be negative values that somehow work or positive ones that do not - which is especially probable for very small or very large numbers. What would be nice is something with a signature like this:

sqrt(x : real @AcceptableRange(0 .. INFINITY))
: real @ResultCheck(r => (r * r) == x)

So the syntax is not completely readable, but the idea is that we are attaching some annotations to the parameter and the result of the function. These can be processed by whoever might need them, similar to the use of .NET attributes and Java annotations. However, I have stretched the allowable syntax to whatever you want -- in this case, a ranged primitive and a lambda function. The possibilities are endless! Compliers for the language doing static checking could find failures before they happen and IDEs could assist people with their problems.

Better yet, we could use...

Static Type Checking

I am a huge believer in static type checking. From the example above, things like just having a type modifier called unsigned makes exceptional conditions of the sqrt function impossible, which is really nice, since this is what you actually mean. Potential errors due to negative numbers are eliminated at compile time, because you simply cannot compile when there is a chance for an error. Compiler-enforced consistent behavior is an awesome thing.

User-Definable Primitive-looking Types

Let's say you are writing a math function that has the signature rotate(thing : Shape, angle : Real). This function rotates the Shape called thing by angle radians. Oh, you could not tell that I use radians by the method signature? That's a problem...

If we were doing some C++, we might have lines like:
typedef float Radians;
typedef float Degrees;

So the signature would look like: void rotate(Shape* thing, Radians angle). Now the method signature tells you which kind of unit you are using. Of course, the problem here is obvious: since Radians and Degrees are actually the same type, we are free to convert between the two and the compiler will not actually care that there is a difference (because, as far as it is concerned, there is not a difference).

So how can you make the compiler care? In C++, this is notoriously difficult (although possible). Once again, let us pretend that there is something perfect out there for me that looks like this:
type Radians is real range 0 .. 2 * PI
type Degrees is real range 0 .. 360

And then one could specify that there exists a scalar conversion between the two units:
conversion Radians <=> Degrees is implicit direct

That looks a little funny, but stay with me for a second. The <=> token means that the conversion is two-way. After that, I added some words just to show that my dream is really powerful. implicit means that there is an implicit conversion (as opposed to an explicit one) - the compiler is allowed to freely convert the units between each other (assuming it follows the rules of conversion). Then, direct is just a way to specify that there are no fancy conversion rules: the compiler just figures out that 180/PI is a good conversion factor for the number specified.

var angle = 180 degrees
rotate(thing, angle)

This is kind of along the lines of Ada with a little bit of extra conversion logic. A convenient system like this would be great for certain NASA contractors.



I've decided to split this incredibly long post up...

15 April 2010

Implications of Apple's iPhone Lockdown

If you have been living under a rock this week and have not heard about Apple's wonderful new iPhone Developer License Agreement, here is the part that is bothering people:

3.3.1--Applications may only use Documented APIs in the manner prescribed by Apple and must not use or call any private APIs. Applications must be originally written in Objective-C, C, C++, or JavaScript as executed by the iPhone OS WebKit engine, and only code written in C, C++, and Objective-C may compile and directly link against the Documented APIs (e.g., Applications that link to Documented APIs through an intermediary translation or compatibility layer or tool are prohibited).

I am not exactly a businessman, but this does seem like a rather bad move on Apple's part. Apple seems to lack an official stance of the matter, but the response of Steve Jobs to John Gruber's blog is the closest I have seen. I think the John summarized Apple's reasoning in the 4th paragraph (emphasis mine):

So what Apple does not want is for some other company to establish a de facto standard software platform on top of Cocoa Touch. Not Adobe’s Flash. Not .NET (through MonoTouch). If that were to happen, there's no lock-in advantage.

It is Apple's platform which they used to take over the "hip" smartphone market with elegant simplicity, solid design and good marketing. Somehow, despite the incredible lockdown on distribution (to the point where you can develop an application and just have it rejected), people are climbing over each other to develop exclusive applications for it. It seems like everyone and their dog is making an iPhone application and it is the de facto "easy mobile platform," despite the fact that other SDKs exist and look awesome.

John then goes on to say this:

Then Apple releases major new features to iPhone OS, and that other company’s toolkit is slow to adopt them. At that point, it’s the other company that controls when third-party apps can make use of these features.

How is that relevant? As an application developer, layers of abstraction are a good thing. In the simplest case, consider a library like SDL. It allows me to develop an application concurrently for Windows, Linux, OSX and whatever else SDL, despite the drastic differences in windowing systems and system libraries. For the majority of the graphics things I need, SDL provides a convenient way to develop it for all target platforms simultaneously. In the corner cases where I need functionality that SDL does not provide, I can work down to system-specific OpenGL calls. The mark of the good library is that it abstracts the most common functionality and lets you get more specific when you need to. If the abstraction layer does not provide the functionality you need it, make the system call yourself. If the abstraction layer does not allow you to do this, get a new library.

I can understand that Apple does not want a bad standard taking over their magical platform, but unlike a web standard, when you compile a language, assemble the generated code and run it on a processor, the processor is completely agnostic to what the original input language was. I'm willing to bet that when Adobe's software is transforming your ActionScript code to run on the iPhone, they are making the appropriate system calls. There is also a good chance that the code that Adobe emits automatically is more efficient than the average iPhone developer's. If application developers need some functionality provided by Apple's SDK but not yet wrapped in a high-level call, they can just make the SDK call. If the system does not allow this, it is time to get a new one.

All Apple is doing is locking down their system so that people have to use their tools to develop for them. While a great money-grabbing move: Does it help the customers? While John provides the common assertion that cross-platform apps are bad on Apple platforms, I would argue that has less to do with the fact that the application is cross-platform and more to do with developer time. Many Windows+OSX apps are worse on OSX because developers do not care enough about the OSX version (which is probably justified, considering the percentage of users of both systems). Besides, there are already systems which provide cross-platform support and I will site QML as just one of such systems. Ultimately, how well an application works on the systems is a property of how much time and effort a developer puts into it. And if a developer is tasked with making the same app for a web browser and the iPhone, then having to write common functionality once will give them more time to dedicate to the making the application feel right and will give consistency across platforms, both of which lead to an ultimately better user experience. And if the abstraction layer does not make it easier to develop, then it is time to get a new one.

Enter the Apple fanboys. Steve makes the case that Apple does not want to tie themselves to a single platform and seems to think that XCode is the only thing capable of cross-compilation or that Apple is somehow innovative in this regard. He also seems to think that the iPad's processor is not of ARM architecture and that, for some reason, Apple is artificially restricting people to running their apps on an emulator. If he is right, then Apple is epically stupid, since it is obvious they are not lacking in a compiler for whatever the A4 chip is. This is my favorite quote:

Apple is sowing the groundwork to make architecture changes seamless—developers will only need to flip a switch to give their apps blazing, native performance.

All debate about what the A4 is aside, it is ridiculous to think this argument is a legitimate justification for the programming language lockdown. If Apple's toolchain really does generate applications that are so superior to others that it outweighs the cost of having to develop two simultaneous apps, then people will just use their system. Apple would win on the merits of being better and would not need to lock down their system in the first place. Apple is no longer competing on the grounds of producing a better system; they are effectively saying that they do not wish to compete and are just locking Adobe out. Lack of competition ultimately hurts the customer.

But Adobe is not the only one Apple is screwing over with this policy - it raises legitimate concerns for abstractions like Qt-iPhone. They are probably safe, since Qt is mostly C++. However, some could argue that with uic and moc, that Qt code can no longer be considered "C++." Of more concern are tools like Unity 3D, whose front page boldly currently advertises "Unity iPhone 1.7, Now with iPad support." Sorry, guys! And what about the people who no longer wish to live in the world of imperative languages? Nope.

Its sad that the iPhone is what people will consider the spark that started the era of ubiquitous mobility, when that perception was nothing but an amazingly well-played marketing game.



UPDATE: The latest Unity mailing list letter contains this gem (emphasis mine):

As most folks already know, Apple introduced a new Terms of Service (ToS) with their iPhone OS 4 that has some Unity users concerned. While we don't yet have a definitive statement to share at this time, we have a healthy, ongoing dialogue with Apple and it seems clear that it's not their intention to block Unity and will continue our efforts to ensure compliance with the Terms.

I feel bad for the Unity developers, who will probably have to do a some amount of work to "insure compliance." What a pointless waste of energy. For more on this, check out David Helgason's blog. What surprises me is that Unity is in direct violation of the words of their terms of service. It is not really debatable — user applications simply are not written in "Objective-C, C, C++, or JavaScript."

This goes back to my emphasis: "...it seems clear that it's not their intention to block Unity..." So what is their intention? It is clear that Adobe is not getting this same treatment. Was Apple's only intention to block Adobe? That is astoundingly pathetic.

01 February 2010

XML Serialization, the XNA Content Pipeline and Wumpus World

When replying to this Stack Overflow question, I realized that the poster was needlessly storing and loading XML through XML serialization at game time instead of using XNA's Content Pipeline.  As many people wanting to make a decent game, he wanted to create an editor and he wanted that editor to use his in-game engine.  This is a great idea, as it will let those designing worlds for your game (and you) see what things will actually look like in the resulting output.  It also opens the door for build-and-play scenarios where you do all design work in one application without the burden of the design-save-build-save-get into proper mode in actual game-test loop.

Unfortunately, his mistake was that he fell into the trap of .NET's XmlSerializer.  It is an easy trap to fall into because it is so simple and convenient to use.  Besides, the XNA Custom Content Pipeline is not advertised well enough and is actually quite obnoxious to use correctly (and the documentation is rather dull).  Luckily for all of my two readers, I have read that documentation already!

In today's example, I'm going to be operating on the wonderful game of Wumpus World.  I'm going to be making visualization and editor for the game, the reader can tie in input controls or whatever to make it a full game.

Setting Up the Project
First things first, we need to set up the project.  Obviously, you'll want to start off with your XNA Game - mine is called "Wumpus."  To make a custom pipeline extension, right-click your solution, perform the clicks to add a new project, and in the "XNA Game Studio $version" section, find "Content Pipeline Extension Library."  The convention here is to append ".Pipeline" to the namespace of your project, so mine is named "Wumpus.Pipeline."  There is automatically a ContentProcessor which is so creatively titled ContentProcessor1.  I'm not sure who at Microsoft made the decision to put these useless default files in every project, but you can just delete it.

Anyway, to actually use the things in your pipeline extension, your content project in the XNA game needs to have a reference to the pipeline project.  To do this, right click the content project, click "Add Reference," then go to the "Projects" tab of the dialog that comes up and select "Wumpus.Pipeline" (or whatever you named your pipeline project).

So now your content project can build with the pipeline extensions, but how do we build the objects in the extension library?  You'll most likely want to use the objects you've already created in your library, so try to add a reference to your main game through the same method.  I say "try to add" because this will not work.  It will tell you that you have a circular dependency.  Why?  Right now, there are three projects in your solution:
  • Wumpus -- This implicitly references Wumpus (Content).
  • Wumpus (Content) -- This references Wumpus.Pipeline.
  • Wumpus.Pipeline -- This cannot reference Wumpus, because that would make a loop.
In reality, this should not be a problem, since the content build project only needs the pipeline project while building and does not even exist at run time, so there is no circular reference.  However, Visual Studio cannot figure this out, so we have to work around this.

The solution is to create another project containing the core components of your game and reference that from both the game engine and the pipeline extension.  This avoids any possibility of circular references.  So add a new Windows Game Library to your project.  The convention is to append ".Core" to the name of the library.  As you probably know, Visual Studio will assume that is the namespace you would like to use for all classes in the project.  However, you won't be putting classes in the Wumpus.Core namespace, so you can change that behavior by opening the properties window for the core library and changing the "Default Namespace" to Wumpus (or whatever you want).

So here is the solution structure as it stands:
  • Wumpus.Core -- This has it's own content project, but you don't need to put anything in it.
  • Wumpus -- References Wumpus (Content) and Wumpus.Core.
  • Wumpus (Content) -- References Wumpus.Pipeline.
  • Wumpus.Pipeline -- References Wumpus.Core.
Now we're shaping up!

Originally I had mentioned an editor which uses the in-game engine and all the objects, so let's do that.  Add another project to your solution, but this one is going to be a WPF Application (you could also do a Windows Forms application, but WPF makes GUI creation fun and easy).  I named mine "Wumpus.Editor."  Add a reference to Wumpus.Core (since you're going to be using objects from it) and Wumpus (since we want to use the engine components from that).

As clearly indicated from my wonderful drawing, Wumpus.Core is the center of the attention.

Code Architecture
In the interests of brevity, I'm going to leave out the parts of the code that are boring.  They're all part of the download package at the bottom of this post.  Basically, there are two classes: TileQuality, which is an enum with qualities such as breezy, gold, stinky and wumpus with bitwise flags and WumpusWorld, which contains a grid of TileQualitys.  They both reside in Wumpus.Core, since they are needed for all phases of design
 

Being able to compose a full application by integrating many small pieces and making them communicate is wonderful.  However, it requires strict adherence to the one object serves one purpose philosophy of object-oriented programming.  Unfortunately, the Microsoft.Xna.Framework.Game class is often a blatant violation of this philosophy, as it encourages putting all the drawing, user interaction, asset management and everything else into this one place.  Knowing this, it is important to put different parts of the application in different places.  For this demo, the only part that will be on its own will be the graphics engine.

So, create a new class in Wumpus (because that is where engine things go) called GraphicsEngine (creative, I know).  In a real project, you might want to make an interface like IEngineModule or something so that you can easily manage all the engine parts, but you can just implement IDisposable.  Since we're abstracting things, we question: What does a graphics engine need?  I'm thinking it needs a reference to the GraphicsDevice so that it can do its job and a ContentManager so that it can locate and load graphics resources.  This is my constructor signature:

public GraphicsEngine(GraphicsDevice device, ContentManager manager);

This gives us the freedom to use the graphics engine any time we have those two items, instead of relying on the engine to go out and discover those things itself (which is a very easy way to accidentally create class coupling).  This is the basis of dependency injection.  Of course, if we wanted to be real legit about this, we could plug in a real dependency injection framework and work out what needs what from some configuration, but that's overkill here.

The sole job of the graphics engine is, given a WumpusWorld, draw it to the display device:

public void Draw(WumpusWorld world);

We are going to call this from our game, so we will need to have an instance of GraphicsEngine in our WumpusGame class.  Following the XNA patterns, Initialize is probably the best place to create the graphics engine, since we should have a reference to a valid device (assuming you've made a GraphicsDeviceManager and initialized it, as per the default):

m_graphics = new GraphicsEngine(GraphicsDevice, Content);

So instead of performing the drawing ourselves in WumpusGame.Draw, we delegate this task to the GraphicsEngine and trust it to do everything right:

protected override void Draw(GameTime gameTime)
{
    m_graphics.Draw(m_world);
}

Pretty easy, huh?  Of course, m_world is null right now, so you will want to instantiate it somewhere (doing it in LoadContent makes the most semantic sense).  If you have the free time, you can edit the world by hand and play the game:
Wumpus World Game

I know, the artwork is amazing.

Saving and Loading
As much fun as it is to create these worlds in code, it would be nice if we could persist these things outside of code.  For many things, .NET's automatic XML serialization is a wonderful thing and can be used.  However, my WumpusWorld class has a two-dimensional indexed property (public TileQuality this[int row, int col] { get; set; }), which cannot be serialized, so I wrote my own methods for saving and loading:

public void Save(Stream stream);
public static WumpusWorld Load(Stream stream);

These methods will be called by anything that wants to save or load for the simple text format (hereby referred to as .wump).  Here is some example output:

4,4
None
Breeze
Breeze
Pit
Breeze
Pit
Breeze, Stench
Breeze, Gold
None
Breeze, Stench
Breeze, Wumpus
Stench
Dude
Breeze
Pit
Breeze

Originally, though, I had talked about extending XNA's content pipeline, so we're going to make a reader and writer to convert to and read from the XNB binary format.  This lets you do fun things like verify every single file at build time, take advantage of automatic compression and deployment and a slew of other features gained by drinking the XNA Kool-Aid.  OOOH YEAH!
  1. ContentImporter: Reads an object from disk into memory.  Input is a file name and the output is some .NET object.  In our case, we're reading a .wump text file and outputting a WumpusWorld.
  2. ContentProcessor: Takes a .NET object, runs some operations on it and outputs some other .NET object.  You can perform whatever arbitrary modification to the object you need to.  We are not using it here, but I'm mentioning it because these can be very helpful (Shawn Hargreave's Pre-Multiplied Alpha Processor is a good example).
  3. ContentTypeWriter: Takes the .NET object that has been imported and optionally processed and writes it to disk with a binary serializer.  The input is the processed .NET object and the output is a binary serialized file.  This is the last step that is taken by the pipeline project.
  4. ContentTypeReader: Reads from the binary serialized stream and outputs the .NET in-game object.  This is in your actual game engine code.
All of these classes are very simple -- here is the entire WumpusWorldImporter class:

[ContentImporter(".wump", DisplayName = "Wumpus World Importer")]
public class WumpusWorldImporter : ContentImporter<WumpusWorld>
{
    public override WumpusWorld Import(string filename, ContentImporterContext context)
    {
        //    use the built-in load function
        return WumpusWorld.Load(File.OpenRead(filename));
    }
}



Simple, eh?  The ContentImporter attribute lets Visual Studio know that when we add a .wump file to the content build project, we would like to automatically use this content importer and the text to display the name of the importer with.  Look in the code for all the other class, but there is not much that needs explanation.  Most of the ugly stuff is out of the way so that you are presented with this nice, clean interface.

Create a .wump file (might I suggest using the "example output" from a few paragraphs ago?) and adding it to the Wumpus's content build project.  If everything worked properly, Visual Studio will associate the importer, not attach a processor, locate the writer at build time and associate the correct reader when the ContentManager needs it.  In short, you will be able to load a WumpusWorld with a simple call like this:

m_world = Content.Load("World/wumpus");

Pretty slick stuff.

The Editor
So, we would really like people to be able to create new worlds not by editing some text file, but by using a GUI tool.  As great as the NeoForce Controls are, they just do not have the sheer amount of things in the .NET forms nor are they as easy to design with (unless someone has made a WYSIWYG editor).  Wouldn't it be great to somehow embed the GraphicsEngine that we have already made into a heavyweight GUI framework?  In this demo, I'll be using WPF, because it is shiny, new and fun to work with.

The classic way to work with WinForms is to use Microsoft's WinForms Graphics Device, which is quite helpful.  This allows you to make an XNA GraphicsDevice render to the surface of a regular control.  However, this relies on the control having a handle, which is not present WPF (at least in a way that you can see).  This is easily solved through use of WindowsFormsHost, which lets you host a WinForms control in a WPF environment.

"ctl_formsHost" />

On initialization, attach our world control as the child element of the host:

ctl_worldView = new WumpusWorldControl();
m_graphicsService = GraphicsDeviceService.AddRef(ctl_worldView.Handle,
                                                  ctl_worldView.Width,
                                                  ctl_worldView.Height);
ctl_formsHost.Child = ctl_worldView;

The final output of my editor looks like this:
Wumpus World Editor 
Stunning beauty.

Code Files
The code here is not at all a "finished product." It would be pretty easy to extend the editor by doing things like adding smell and breeze automatically when a Wumpus or pit is added or making a world larger than 4x4.

It is released under the Apache License, Version 2.0 and should be used for good, not evil.
Wumpus Source Code