Melding Monads

2011 December 30

Announcing postgresql-simple

Filed under: Uncategorized — lpsmith @ 5:04 am

I’ve done a bit of database programming off and on over the years, and when I started into a larger project and decided to use Haskell and PostgreSQL, I didn’t understand exactly how bad of a choice Haskell was for PostgreSQL development at the time. So postgresql-simple and postgresql-libpq is hopefully a small step towards remedying the situation. This work is a fork of Bryan O’Sullivan’s mysql-simple library and Grant Monroe’s libpq binding, with some code informed by Chris Done’s pgsql-simple library, and I’d to thank all for their terrific work.

So, there are five things that, taken together, distinguish postgresql-simple from the other PostgreSQL options on hackage:

1. postgresql-simple uses libpq, PostgreSQL’s official client library for C

Chris Done’s pgsql-simple is a pure Haskell implementation of a small subset of the PostgreSQL frontend/backend protocol. I do think there is technical merit to a native Haskell implementation, and in particular I’ve identified a potentially significant advantage that cannot be achieved via libpq. However writing a native implementation is a significant undertaking, and such efforts invariably lag behind in terms of features. And though pgsql-simple is experimental software that hasn’t been officially released and I don’t want to be unfair to it, pgsql-simple performs badly. I sped up a program I had been using to test pgsql-simple by over a factor of 20 simply by switching to postgresql-simple.

Among other things using libpq means that postgresql-simple gets the full range of connection and authentication options out of box, including support for Unix Domain Sockets, SSL, and GSSAPI.

2. postgresql-simple uses libpq-based string escaping.

HSQL doesn’t provide parameterized SQL statements at all, requiring programmers to dynamically generate sql and handle escaping issues themselves. And experience has shown punting on this issue is unacceptable, as dynamically generating SQL is risky and forcing programmers to handle escaping issues is an order of magnitude worse.

HDBC and pgsql-simple support parameterized SQL, but they attempt to do the escaping themselves. The problem is that this is usually a bit more subtle than it first appears, leading to bugs and potential vulnerabilities. For example, in PostgreSQL, escape syntax depends on the value of the standard_conforming_strings server setting, which libpq will detect and accommodate accordingly.

3. postgresql-simple makes no pretense of supporting prepared statements

In PostgreSQL, a prepared statement allows one to send off a parameterized sql statement to a backend, and then re-use that statement as many times as you like by just filling in the parameters. This means the text of the prepared statement does not repeatedly traverse the wire, the backend does not parse the sql query multiple times, the backend re-uses the query plan generated when the statement was prepared, and you get to use protocol-level parameters which can eliminate the need for escaping strings and converting numbers to base-10 and back.

Note that prepared statements are very often, but not universally, advantageous. Starting from nothing, they require two round trips to get any information out, so preparation can be disadvantageous for one-shot queries. Also, occasionally re-planning a query can be advantageous, so infrequently executed queries can be harmed by preparation as well.

HDBC nominally supports prepared statements, but in fact all HDBC-postgresql does is cache some of the preprocessing of the query. No preparation in the database sense ever occurs. So while the lack of support for prepared statements is a definite disadvantage, neither does postgresql-simple pretend to prepare statements when it doesn’t.

4. postgresql-simple provides a simple API that most programmers would find familiar

Takusen appears to be the one PostgreSQL database access library for Haskell that gets basic implementation details more-or-less correct. Unfortunately it exports an esoteric API that is not applicable in all situations. In particular, web applications often use various forms of connection caching or pooling, which is fundamentally incompatible with the deterministic connection resource guarantees provided by Takusen.

Also, the *-simple libraries have a relatively nice interface compared to HDBC. Ultimately that was the breaking point that caused me to spend the time to create postgresql-simple; no one database library did everything I needed, and while working on some new code where my mangled fork of HDBC made the most sense, I realized I really wished I was using pgsql-simple instead. Also, I paid attention to ensure that application code could add support for user-defined PostgreSQL types with a minimum amount of fuss and without modifying the library, something that neither HDBC nor pgsql-simple could really do.

5. Support for Listen/Notify

Listen/Notify is the perfect solution when you want to write a program that responds to changes made to a PostgreSQL database. Informing these programs when changes are available consumes less resources and provides lower latencies than periodic polling. And since you can use a rule or trigger to send the notification, these notifications can be robust; you don’t have to assume that the program making the changes even knows that anybody wants to be informed of the changes. Listen/notify is aware of and consistent with transactions; notifications don’t get sent until the transaction commits. Even if you are willing to put that kind of logic in the database clients, using listen/notify solves the otherwise sticky problem of finding and coordinating with the other clients.

Nothing else on hackage supports asynchronous notifications, unless you use a low-level libpq binding directly.

Looking Forward

Creating a mid-level database access library based on libpq is a significant undertaking, and there is an awful lot that isn’t supported, including prepared statements, binary data formats, copy in/copy out support, more and better support for PostgreSQL data types, and PostGIS support. Many other things could be improved, such as using libpq asynchronously, better end-to-end typechecking that discovers more errors at compile time, and other interface improvements. For example, SqlQQ is a simplistic quasiquoter to improve the literal syntax for multi-line SQL queries; one could certainly imagine extending the syntax to support including a $haskellvariable as a SQL parameter instead of going through the syntactic indirection of ? in the SQL string with haskellvariable in a separate Haskell parameter.

Funnily enough, I ran across this thread in which PostgreSQL luminaries were complaining about the quality of Python’s PostgreSQL libraries. And Python is considerably ahead of Haskell in this regard.


2010 August 25

Fun with the Lazy State Monad, Part 2

Filed under: Uncategorized — lpsmith @ 5:52 am

In a previous post, I demonstrated how to use the lazy state monad to turn Geraint Jones and Jeremy Gibbon’s breadth-first labeling algorithm into a reusable abstraction, and mentioned that I did not know how to combine the result with continuations. In this post, I will do exactly that.

A hat tip goes to Twan van Laarhoven for suggesting that I split my fresh operator into two, which I adopt in this post. In the comments, down is defined as:

type Fresh n a = State [n] a

down :: Fresh n a -> Fresh n a
down m = do
    (n:ns) <- get
    put ns
    a <- m
    ns' <- get
    put (n:ns')
    return a

Now, if we naïvely apply the continuation transformer in the Monad Transformer Library, the get and put operators are lifted as follows:

type FreshCC r n a = ContT r (State [n]) a

lift :: Monad m => m a -> ContT r m a
lift m = ContT (m >>=)

down :: FreshCC r n a -> FreshCC r n a
down m = do
    (n:ns) <- lift get
    lift (put ns)
    a <- m
    ns' <- lift get
    lift (put (n:ns'))
    return a

The problem is that this replaces the >>= of the lazy state monad with the >>= of the continuation transformer, and these two operators have different strictness properties. This, in turn, leads to non-termination. The trick is not to lift get and put, but rather conjugate down:

lower :: Monad m => ContT a m a -> m a
lower m = runContT m return

down :: FreshCC a n a -> FreshCC r n a
down m = lift $ do
    (n:ns) <- get
    put ns
    a <- lower m
    ns' <- get
    put (n:ns')
    return a

-- *or*

down = lift . original_down . lower

Now, the conjugation preserves the use of the lazy state monad’s >>= in the definition of down, however it changes the type of the argument from FreshCC r n a to FreshCC a n a! The other definitions contained in the previous post stay much the same.

Feel free to download freshcc.hs, a full working demonstration of this post. One can even use callCC, fresh, and down in the same computation and terminate! Sadly, I don’t have any demonstrations of this combination, nor do I have any applications in mind. My intuition about callCC is currently quite limited in this context.

I have implemented Dan Friedman’s angel-devil-milestone control operators in conjunction with fresh and down, and have used it to duplicate labels and change the shape of a spirited tree; but I’m definitely cheating at this point, as all I have done is observe that the definitions compute something. I have no control over what’s going on, nor do I know what the definitions are supposed to compute. (yet)

2010 May 2

BBC documentary on Science and Islam

Filed under: Uncategorized — lpsmith @ 3:11 pm

So I wanted to learn how to pronounce the name al-Khwārizmī. So I searched YouTube, and stumbled across Science and Islam hosted by Physicist Jim Al-Khalili, a BBC documentary series consisting of three parts: The Language of Science, The Empire of Reason, and The Power of Doubt.

I did not expect to enjoy the documentary as much as I did; I would like to buy the series on DVD, but I have not found a way to do this. It really opened my eyes: in spite of my western education, I knew something of the golden age of Islamic scholarship, but I failed properly appreciate its importance to the modern world.

I say “in spite of my western education” because many students would recognize the names of Aristotle, Socrates, and Plato, but not al-Khwārizmī, al-Haytham, or al-Biruni. And yet, the contributions of these Islamic scholars are in many ways more lasting and more profound than the Greeks. After all, al-Khwārizmī made large, foundational contributions to Algebra, which is more widely taught than Greek philosophy.

This is not to belittle the contributions of the Greeks; after all the Greeks had a profound influence on Islamic scholarship. The story starts with the creation of an Islamic empire, the largest empire the world had ever seen at the time, and the start of the Translation movement. The empire sought out written materials in any language from any culture, to be translated into Arabic and disseminated throughout the empire. Greek texts were among the first to be translated, and in many cases were given priority.

The widespread use of the Arabic language and lack of intellectual discrimination greatly accelerated the dissemination and creation of knowledge. al-Khwārizmī had the envious position of being one of the first humans in history to have access to both Indian and Greek mathematics, and he set about applying one body of work to the other, refining both and making new discoveries along the way.

al-Haytham wrote a seminal text on optics, and contributed significantly to the scientific method, and is considered one of the greatest scientists of the Middle Ages. al-Biruni measured the size of the Earth with unprecedented accuracy using a novel method based on trigonometry. al-Tusi headed up an program of astronomical observation that was of great importance to Copernicus, Newton, and other European scientists. One of the lasting contributions was questioning the Greek philosophical orthodoxy and an emphasis on repeatable experiments.

The medieval Islamic scholars gathered and disseminated knowledge widely, applied mathematics and practical science to measure the physical world to then-unprecedented accuracy, laid the groundwork that eventually grew into modern chemistry, revolutionized agriculture, industry, metallurgy, navigation, map making, and much more. It is difficult to overstate the importance of these efforts to later European science and scholarship or as a necessary step in the development of the modern world.

Eventually the golden age of Islamic scholarship came to an end; although Europeans tended to overstate the extent of the decline, it certainly did occur. The documentary attributes this to a large number of reasons, including the rejection of the printing press, the decline of the empire that funded these efforts, and entrenched political, economic, and religious interests. Perhaps there are some lessons here for today’s state of affairs.

« Newer Posts

Create a free website or blog at