Comments on Keith's Blog: TxtSushi 0.4.0

Awesome :-) Thanks for the encouragement G. Lisp.....

2009-12-15T20:44:31.914-08:00

Awesome :-) Thanks for the encouragement G. Lisp... isn't that the language they use to torture poor AI students at UMaine who don't know what they're getting themselves into ;-)

Keith, TxtSushi has been a big help to me in a re...

2009-12-15T17:34:40.241-08:00

Keith, TxtSushi has been a big help to me in a recent project! Keep up the good work. If I knew haskel I would help. Don't you want to rewrite it in your favorite language, Lisp?

Mathematics of Linear Algebra

2009-10-10T01:27:36.715-07:00

Mathematics of Linear Algebra

Hi LiYang > Would an in-memory SQLite DB fulfil y...

2009-10-07T19:54:05.109-07:00

Hi LiYang

> Would an in-memory SQLite DB fulfil your SQL (and efficiency) needs? http://hackage.haskell.org/package/sqlite

I don't think so. Mostly because my goal is to be able to work with any valid rectangular CSV input stream with no other restrictions on the input. I have to work with pretty wacky (and huge) CSV files on a regular basis.

Here's an example that I think would be very painful in a real DB: http://patch-tag.com/r/keithshep/txt-sushi/snapshot/current/content/pretty/examples/query-imputed-SNPs.bash

> I've been using sqlite3 on the command line on a binary SQLite DB over my usual Unix plumbing on plain text DBs lately, and my god it's such a joy to work with. (Insomuch as one can be joyful when dealing with SQL.)

SQLite sounds cool! I use hsqldb for my java stuff and it sounds pretty similar. Real DBs are a great way to go when you're in control of the data format and you're keeping the data around for a while, but I love being able to use command-line directly on flat files in many cases.

> As for the question of efficiency: frankly, when one is dealing with a sufficiently large dataset where efficiency becomes an issue, using a tabular textual format for intermediate data isn't perhaps the best of ideas.

Except that the time isn't dominated by the CSV parsing step (I'm very willing to accept the parsing performance hit)... it's the sorting step which is using a binary on disk representation for the sort which is killing me. If the implementation of external-sort was good enough it could be as efficient as any other DB I think. I hope no one is holding their breath for that though :-)

Where by ‘the relational calculus’, I—of course—am...

2009-10-07T16:37:43.771-07:00

Where by ‘the relational calculus’, I—of course—am in fact referring to the relational algebra.

Would an in-memory SQLite DB fulfil your SQL (and ...

2009-10-07T16:33:18.256-07:00

Would an in-memory SQLite DB fulfil your SQL (and efficiency) needs? http://hackage.haskell.org/package/sqlite

I've been using sqlite3 on the command line on a binary SQLite DB over my usual Unix plumbing on plain text DBs lately, and my god it's such a joy to work with. (Insomuch as one can be joyful when dealing with SQL.)

As for the question of efficiency: frankly, when one is dealing with a sufficiently large dataset where efficiency becomes an issue, using a tabular textual format for intermediate data isn't perhaps the best of ideas.

I do support your notion of extensible SQL functions though.

A pragmatic implementation of the relational calculus, anyone? (Rather than the bastardisation that is SQL.)