Blog moved!

Hi all, I'm now using Jekyll for both my website and posts. This will be my last blogger post but I intend to leave the old posts here on blogger. New posts will show up under the notes section of my main site and you can subscribe to them using the new RSS feed. I hope to hear from you there.

-Keith

TxtSushi 0.5.1

I've just released TxtSushi 0.5.1 on hackage. The only reason for this release was to update the license from GPL to BSD (there are no new features or fixes).

I initially released under GPL because I wasn't really sure where I wanted to go with TxtSushi and I knew it was easier to go from GPL to a more liberal license than to move in the other direction. A recent thread on licensing on haskell cafe convinced me that BSD is the way to go.

Getting Started with F# and OpenGL on OS X

I am exploring programming with F# using mono on OS X. I'm thinking about using F# for a computer vision project and so I wanted to start out by playing around with some example code to see how it all works. This post is just to document the steps to get up and running. I'm a complete newbie to F# and mono so please give feedback if there are better ways to go about what I'm doing. I'd also like to eventually have a build environment that works well with mono on OS X and MS Visual Studio. Please leave a comment if you have any advice on this.

• Initially I installed mono 2.6.1. using the OS X Intel installer which worked without any issues.
• Next I downloaded and unzipped the F# 1.9.7.8 zip file from the F# downloads page. The zip file contains a convenient install-mono.sh script so all you have to do is

  chmod +x install-mono.sh && sudo ./install-mono.sh

• I decided to create bash scripts for the compiler and interpreter just for the sake of convenience:

  mv FSharp-1.9.7.8 ~/bin/

  The contents of ~/bin/fsc.bash:

  #!/bin/bash
  
  mono `dirname $0`/FSharp-1.9.7.8/bin/fsc.exe "$@"
  

  ... and likewise the contents of ~/fsi.bash:

  #!/bin/bash
  
  mono `dirname $0`/FSharp-1.9.7.8/bin/fsi.exe "$@"
  

OK great, we now have a working F# compiler and interpreter on OS X (make sure that ~/bin is on your $PATH though... you already knew that didn't you?). OK now let's try to work with one of the F# Samples to make sure things are working as expected. We can build and run the Samples101 example with this command:

fsc.bash --resource:SampleForm.resx \
    sample.fs sampleform.fs Beginners.fs \
    intermediate.fs program.fs
mono program.exe

hmm... almost there. It looks like there is a hard-coded "..\..\" that gets inserted in source code path. A minor code change in sample.fs fixes that:

60c60,61
<         let dir = System.IO.Path.Combine(appdir, @"..\..\")
---
>         //let dir = System.IO.Path.Combine(appdir, @"..\..\")
>         let dir = appdir

... now for a second try:

That's better. OK, now let's see about getting an OpenGL example running.

• First I downloaded the TAO Framework (version 2.1.0) which provides .NET bindings to OpenGL and other API's for gaming
• You can install all of the libraries into mono's global registry by:

  cd $TAO_HOME/bin
  for file in *.dll; do sudo gacutil -i $file; done

  hmm, I get the following error while trying to install FreeType:

  Installed Tao.FreeType.dll into the gac (/Library/Frameworks/Mono.framework/Versions/2.6.1/lib/mono/gac)
  
  ** (/Library/Frameworks/Mono.framework/Versions/2.6.1/lib/mono/2.0/gacutil.exe:253): WARNING **:
  Error parsing  \x87\xa0: Error on line 3 char 46: 'libglfw.so"' is not a valid name: '"'

  I'm going to ignore this for now since it doesn't seem to matter for my simple example code.

OK now let's translate the simplest TAO example to F# and see about getting it to run. This is a translation of the $TAO_HOME/source/examples/Redbook/Hello.cs example which is under the MIT License (see license text at end of post). Apologies for the lack of syntax hilighting:

open System
open Tao.FreeGlut
open Tao.OpenGl

let init () =
    // Select clearing color
    Gl.glClearColor (0.0f, 0.0f, 0.0f, 0.0f);

    // Initialize viewing values
    Gl.glMatrixMode Gl.GL_PROJECTION;
    Gl.glLoadIdentity ();
    Gl.glOrtho (0.0, 1.0, 0.0, 1.0, -1.0, 1.0)

let display () =
    // Clear all pixels.
    Gl.glClear Gl.GL_COLOR_BUFFER_BIT;

    // Draw white polygon (rectangle) with corners at (0.25, 0.25, 0.0) and
    // (0.75, 0.75, 0.0).
    Gl.glColor3f (1.0f, 1.0f, 1.0f);
    Gl.glBegin Gl.GL_POLYGON;
        Gl.glVertex3f (0.25f, 0.25f, 0.0f);
        Gl.glVertex3f (0.75f, 0.25f, 0.0f);
        Gl.glVertex3f (0.75f, 0.75f, 0.0f);
        Gl.glVertex3f (0.25f, 0.75f, 0.0f);
    Gl.glEnd ();

    // Don't wait!  Start processing buffered OpenGL routines.
    Gl.glFlush ()

// Declares initial window size, position, and display mode (single buffer and
// RGBA).  Open window with "Hello" in its title bar.  Call initialization
// routines.  Register callback function to display graphics.  Enter main loop
// and process events.
let main =
    Glut.glutInit ();
    Glut.glutInitDisplayMode (Glut.GLUT_SINGLE ||| Glut.GLUT_RGB);
    Glut.glutInitWindowSize (250, 250);
    Glut.glutInitWindowPosition (100, 100);
    ignore (Glut.glutCreateWindow "Hello");
    init ();
    Glut.glutDisplayFunc (new Glut.DisplayCallback(display));
    Glut.glutMainLoop ()

[<STAThread>]
do main

Now I just compile and run with:

fsc.bash -I $TAO_HOME/bin -r Tao.OpenGl.dll -r Tao.FreeGlut.dll hello.fs
mono hello.exe

Hey, it works! That wasn't so painful.
MIT License Copyright 2003-2005 Tao Framework Team http://www.taoframework.com All rights reserved. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Simple Recipe for Static Domain Hosting With Blog

Hopefully this post will help save someone time looking for similar hosting. My current hosting requirements are really simple. I need:

• static site hosting for keithsheppard.name
• blog hosting for blog.keithsheppard.name

Luckily blogger lets you use your own domain which takes care of blog.keithsheppard.name so I just need to find a static hosting service for keithsheppard.name. When I went looking for hosting though almost everything I found allows you to run at least PHP, perl scripts along with a MySQL database and costs $5/month or more. Using these hosts means that you have to pay for the extra resources, security and maintenance that goes along with allowing scripts and DB's to run. Another problem is that the cheaper services are often under-resourced which means added down time for you.

After a lot of trial and error I found hurricane electric $1/month static site hosting which has just worked with no problems worth mentioning. As a bonus you transfer your files using scp instead of the lame web-based file managers a lot of the other hosts make you use. HE also throws in a free domain name but I have my domain under dotster and I like keeping the hosting and name registration separate (that costs me an extra $15/year). So for what it's worth, this configuration has worked well for me.

UPDATE: It seems that hurricane electric is no longer advertising $1/month hosting, but no worries I found something better! You can host static content on your domain for free using github (I think something similar is possible with bitbucket). You can find instructions at http://pages.github.com/ and see my example site repo at https://github.com/keithshep/keithshep.github.com (note the CNAME file in my repo).

TxtSushi 0.5.0

Hiya! I've uploaded TxtSushi 0.5.0 to hackage and there are quite a few changes. I still need to update the documentation on http://keithsheppard.name/txt-sushi which probably won't happen for a week or so (Edit: I have made the updates) but I'll summarize the changes here:

• generalized the JOIN syntax to allow joins on arbitrary expressions (you used to only be able to join on column IDs). This test script demonstrates a join on a simple expression
• added initial support for the "for x in [col1 .. col4] yield expr" syntax. See this test script for an example of how it works.
• parser now works with UNIX, DOS and Mac newlines no matter which OS you're working on
• added integrated help for all SQL functions and operators. Use "tssql -help" to get a list of all functions and operators. Use "tssql -help sum" to get help on the SUM function
• added many new function definitions
• Grouped data syntax generalization: you can now group tables multiple times (via nested selects) and join those grouped tables to other tables (grouped or not). You can even mix grouped and non-grouped columns in the same expression.
• Many bug fixes and code simplifications (still learning how to write good Haskell). Most notably it is now very easy to extend tssql with new SQL functions and operations. Here's an example of what it takes to add a new function.
• added support for bool constants in SQL expressions
• improved error messages
• Declared the "classic join" performance bug a feature :-)
• and most importantly: changed the synopsis to "The SQL link in your *NIX chain"

Have fun and please send me your bug reports!

TxtSushi 0.4.0

I have just uploaded version 0.4.0 of TxtSushi with the following changes:

• General code improvements: fixed any -Wall warnings, improved help messages and SQL parser error messages
• External sort: added an -external-sort option to tssql which tells tssql to sort on disk for JOIN and ORDER BY. This was tested on a 10G CSV file and it worked but it took 10 hours on my macbook! I think that this can be improved.
• Nested select: TxtSushi now allows nested select statements
• Classic joins: added support for older style join syntax. These joins are still unoptimized though so their time complexity is row_count^2 rather than the row_count(log row_count) complexity when using the INNER JOIN syntax
• Transpose utilities: added transposecsv and transposetab utilities which will print a transposed version of the given table

Fellow Haskell Hackers Wanted:

I think TxtSushi fills a unique niche in the *NIX toolchain for tabular data. On one side you have python, perl, sed, awk which are all great for general purpose text processing but it takes a lot of hand rolled code to do table filtering/joining/transformation and it gets even worse if you want to be able to deal correctly with quoted fields. On the other side you have real databases which allow you to do all of these things but which tend to be heavyweight solutions that require data import/export and don't want anything to do with your other *NIX commands. TxtSushi is different because it takes your text tables (or event STDIN) as they are and doesn't want to be anything more than an SQL link in your *NIX tool chain.

The two long term goals that I have for TxtSushi are to improve it to the point where it is generally available on *NIX distro package managers and that it is "owned" and developed by a group instead of just me. These two goals really go together since I may not be able to do all of the work required to mature TxtSushi by myself. I thought about keeping these goals to myself until I could get to the point where the core architecture are solidified and the remaining work would amount to developing and integrating modules into the existing architecture, but I think things are just moving too slowly with me hacking alone. So, if you are interested in hacking TxtSushi with me let me know!

Off the top of my head here are some relatively self-contained contributions that can be made:

• Extensible SQL Functions: This one should be fun. The way I implemented SQL functions isn't very pretty. The function parsers are in SQLParser.hs toward the bottom and the SQL Function execution code is in a big function guard toward the bottom of SQLExecution. I think we should have an SQLFunction type which would bundle up both the parsing and execution for a single SQL function/operator. The real payoff is that we could then have an XMonad-like configuration file that allows users to define their own SQL functions (Eg: if a user wants to define a STD_DEV aggregate function, now they can as long as they know haskell).
• Improve table parsing: the table parsing code was the first thing I wrote and it shows. It should be replaced by something more like what you find in Real World Haskell: Using Parsec which is much more concise and tolerant of any end-of-line encoding
• External sort: The external sort algorithm works but could probably use a more expert Haskell hacker's eye to become efficient (there were a couple of reasons I couldn't just use the external-sort cabal library as-is).
• Efficient classic joins: The classic style WHERE joins should be just as efficient as the INNER JOIN joins. If you're interested in this I have an idea for what I think may be the easiest way to do this. See issue 7
• Support for SQL Case Statement: See issue 8
• Implement window functions like RANK() and ROW_NUMBER(): See issue 11
• Test cases? Bug reports? Some other idea you have? ...

If you're curious, the simplest way to browse the source is here http://patch-tag.com/r/keithshep/txt-sushi.

Server Died!

Update 1: The site is back up and running, hosted by izfree.com

Update 2: izfree.com was not working out so I switched to he.net. It's $1 per month for static file hosting. I'll report back after a while on how he.net is going.

I've been serving up my website on the cheap using dynamic DNS and a spare linux box at home. My linux box just died so my main site is down :-(. Luckily my blog is hosted by blogger which is why it's still doing fine.

I don't suppose anyone knows a good way to host a web site for free assuming all the content is just static pages? I need to be able to come with my own domain name too. I think I'm going to have to finally break down and actually spend money on hosting.