It’s been some time since my last post and I’ve been far too busy to write any crap ;)

I have been working with OpenSSL for some time now and I really want to tell other people my own experience and what a p.i.t.a. programming with OpenSSL has been to me!

A little preface: I was using OpenSSL to add S/MIME capabilities to a mail processing library. I needed to be able to encrypt/decrypt and digitally sign/verify e-mail messages. The library is completely written in C/C++/Qt4 and needs to be platform independent (Win32, *nix and OSX). The S/MIME routines should be able to parse messages produced by other MUAs (Mail User Agents) and, of course, produce messages that other MUAs are able to decrypt and/or verify.

OpenSSL is being used throughout the project I am working on (the mail library is just a small part of a commercial business system software). This is why boss & me decided to use OpenSSL for the S/MIME routines.

After spending about one week to understand how to use some basic OpenSSL functions we had to hire one of the OpenSSL developers (Dr. Stephen Henson) to help us using the S/MIME APIs.

OpenSSL has reached version 0.9.8d on September 28th.The OpenSSL project officially started December 23rd 1998.
Almost eight years and it still hasn’t reached version 1.0. Looks quite strange to me!
Anyway, the main problem is that the OpenSSL documentation has reached, say, version 0.001.
In other words, there is almost no documentation at all. Some of the command line tools are documented, but most of the APIs are not. And the documentation is not up to date with the library.

I am very disappointed about this as OpenSSL is a great and really useful library and there’s really lots of software using it (both open and closed source).

Ok, let’s go back to the story. As I said, Steve helped us to setup some nice higher level methods in C++/Qt. The next step was to integrate these methods in the library. And this was the real p.i.t.a.!

The first problem: understand what kind of input OpenSSL was expecting and what kind of output it would produce. Yes, some documentation would have been quite useful here ;)
My first disappointment here was that the OpenSSL output contained some MIME header lines too. I just wanted a simple signature or a bunch of encrypted bytes!
My second disappointment was that the produced signatures were not being verified by common MUAs like Microsoft Outlook Express (MOE) and Thunderbird (TB).
After some hours or days spent struggling with code, OpenSSL source code and mailing lists I came up with the proper combination of flags to pass to the OpenSSL S/MIME functions.
I am currently using the following flags with the PKCS7_sign() and SMIME_write_PKCS7() functions: PKCS7_DETACHED | PKCS7_STREAM | PKCS7_BINARY | PKCS7_CRLFEOL.
The compatibility issue has been solved with the PKCS7_CRLFEOL flag - which is not mentioned in the documentation (obviously!).

Since I needed to have messages with CRLF line endings only I also had to convert all the line endings in the OpenSSL produced output because the OpenSSL MIME header lines end with a single LF (the signature and the encrypted data end with a CRLF - but I needed to have CRLF pairs only).

One last problem: i had to call BIO_set_mem_eof_return() before calling SMIME_read_PKCS7() to load the PKCS7 data to be verified/decrypted. The reason? Who knows! I get a "not enough data" error if I don’t call BIO_set_mem_eof_return(). The solution to this issue appears in a post on the OpenSSL mailing list but the author of the post himself has no idea about why this solves the problem.

Conclusion: it took me about one week to add S/MIME capabilities to my mail library using OpenSSL. After this experience I really wouldn’t suggest to use OpenSSL for S/MIME.
Besides of the problems mentioned above, there are lots of performance issues I did not mention at all. The current OpenSSL PKCS7 and S/MIME features are really simple and do not allow to work on a stream of bytes: the whole message needs to be stored in a memory array. This may not be such a problem with plain text messages or with 1GB memory PCs but it is not a nice solution.

That’s all folks. I will talk about OpenSSL again as soon as I will need to handle certificates for my S/MIME-enabled library.