Some major restructuring changes to ENGINE, including integrated cipher and
digest support, are on their way. Rather than having gigantic commit log messages and/or CHANGES entries, this change to the README will serve as an outline of what it all is and how it all works.
This commit is contained in:
parent
591ccf586d
commit
f185e725a0
@ -1,278 +1,211 @@
|
|||||||
NOTES, THOUGHTS, and EVERYTHING
|
Notes: 2001-09-24
|
||||||
-------------------------------
|
|
||||||
|
|
||||||
(1) Concurrency and locking ... I made a change to the ENGINE_free code
|
|
||||||
because I spotted a potential hold-up in proceedings (doing too
|
|
||||||
much inside a lock including calling a callback), there may be
|
|
||||||
other bits like this. What do the speed/optimisation freaks think
|
|
||||||
of this aspect of the code and design? There's lots of locking for
|
|
||||||
manipulation functions and I need that to keep things nice and
|
|
||||||
solid, but this manipulation is mostly (de)initialisation, I would
|
|
||||||
think that most run-time locking is purely in the ENGINE_init and
|
|
||||||
ENGINE_finish calls that might be made when getting handles for
|
|
||||||
RSA (and friends') structures. These would be mostly reference
|
|
||||||
count operations as the functional references should always be 1
|
|
||||||
or greater at run-time to prevent init/deinit thrashing.
|
|
||||||
|
|
||||||
(2) nCipher support, via the HWCryptoHook API, is now in the code.
|
|
||||||
Apparently this hasn't been tested too much yet, but it looks
|
|
||||||
good. :-) Atalla support has been added too, but shares a lot in
|
|
||||||
common with Ben's original hooks in bn_exp.c (although it has been
|
|
||||||
ENGINE-ified, and error handling wrapped around it) and it's also
|
|
||||||
had some low-volume testing, so it should be usable.
|
|
||||||
|
|
||||||
(3) Of more concern, we need to work out (a) how to put together usable
|
|
||||||
RAND_METHODs for units that just have one "get n or less random
|
|
||||||
bytes" function, (b) we also need to determine how to hook the code
|
|
||||||
in crypto/rand/ to use the ENGINE defaults in a way similar to what
|
|
||||||
has been done in crypto/rsa/, crypto/dsa/, etc.
|
|
||||||
|
|
||||||
(4) ENGINE should really grow to encompass more than 3 public key
|
|
||||||
algorithms and randomness gathering. The structure/data level of
|
|
||||||
the engine code is hidden from code outside the crypto/engine/
|
|
||||||
directory so change shouldn't be too viral. More important though
|
|
||||||
is how things should evolve ... this needs thought and discussion.
|
|
||||||
|
|
||||||
|
|
||||||
-----------------------------------==*==-----------------------------------
|
|
||||||
|
|
||||||
More notes 2000-08-01
|
|
||||||
---------------------
|
|
||||||
|
|
||||||
Geoff Thorpe, who designed the engine part, wrote a pretty good description
|
|
||||||
of the thoughts he had when he built it, good enough to include verbatim here
|
|
||||||
(with his permission) -- Richard Levitte
|
|
||||||
|
|
||||||
|
|
||||||
Date: Tue, 1 Aug 2000 16:54:08 +0100 (BST)
|
|
||||||
From: Geoff Thorpe
|
|
||||||
Subject: Re: The thoughts to merge BRANCH_engine into the main trunk are
|
|
||||||
emerging
|
|
||||||
|
|
||||||
Hi there,
|
|
||||||
|
|
||||||
I'm going to try and do some justice to this, but I'm a little short on
|
|
||||||
time and the there is an endless amount that could be discussed on this
|
|
||||||
subject. sigh ... please bear with me :-)
|
|
||||||
|
|
||||||
> The changes in BRANCH_engine dig deep into the core of OpenSSL, for example
|
|
||||||
> into the RSA and RAND routines, adding a level of indirection which is needed
|
|
||||||
> to keep the abstraction, as far as I understand. It would be a good thing if
|
|
||||||
> those who do play with those things took a look at the changes that have been
|
|
||||||
> done in the branch and say out loud how much (or hopefully little) we've made
|
|
||||||
> fools of ourselves.
|
|
||||||
|
|
||||||
The point here is that the code that has emerged in the BRANCH_engine
|
|
||||||
branch was based on some initial requirements of mine that I went in and
|
|
||||||
addressed, and Richard has picked up the ball and run with it too. It
|
|
||||||
would be really useful to get some review of the approach we've taken, but
|
|
||||||
first I think I need to describe as best I can the reasons behind what has
|
|
||||||
been done so far, in particular what issues we have tried to address when
|
|
||||||
doing this, and what issues we have intentionally (or necessarily) tried
|
|
||||||
to avoid.
|
|
||||||
|
|
||||||
methods, engines, and evps
|
|
||||||
--------------------------
|
|
||||||
|
|
||||||
There has been some dicussion, particularly with Steve, about where this
|
|
||||||
ENGINE stuff might fit into the conceptual picture as/when we start to
|
|
||||||
abstract algorithms a little bit to make the library more extensible. In
|
|
||||||
particular, it would desirable to have algorithms (symmetric, hash, pkc,
|
|
||||||
etc) abstracted in some way that allows them to be just objects sitting in
|
|
||||||
a list (or database) ... it'll just happen that the "DSA" object doesn't
|
|
||||||
support encryption whereas the "RSA" object does. This requires a lot of
|
|
||||||
consideration to begin to know how to tackle it; in particular how
|
|
||||||
encapsulated should these things be? If the objects also understand their
|
|
||||||
own ASN1 encodings and what-not, then it would for example be possible to
|
|
||||||
add support for elliptic-curve DSA in as a new algorithm and automatically
|
|
||||||
have ECC-DSA certificates supported in SSL applications. Possible, but not
|
|
||||||
easy. :-)
|
|
||||||
|
|
||||||
Whatever, it seems that the way to go (if I've grok'd Steve's comments on
|
|
||||||
this in the past) is to amalgamate these things in EVP as is already done
|
|
||||||
(I think) for ciphers or hashes (Steve, please correct/elaborate). I
|
|
||||||
certainly think something should be done in this direction because right
|
|
||||||
now we have different source directories, types, functions, and methods
|
|
||||||
for each algorithm - even when conceptually they are very much different
|
|
||||||
feathers of the same bird. (This is certainly all true for the public-key
|
|
||||||
stuff, and may be partially true for the other parts.)
|
|
||||||
|
|
||||||
ENGINE was *not* conceived as a way of solving this, far from it. Nor was
|
|
||||||
it conceived as a way of replacing the various "***_METHOD"s. It was
|
|
||||||
conceived as an abstraction of a sort of "virtual crypto device". If we
|
|
||||||
lived in a world where "EVP_ALGO"s (or something like them) encapsulated
|
|
||||||
particular algorithms like RSA,DSA,MD5,RC4,etc, and "***_METHOD"s
|
|
||||||
encapsulated interfaces to algorithms (eg. some algo's might support a
|
|
||||||
PKC_METHOD, a HASH_METHOD, or a CIPHER_METHOD, who knows?), then I would
|
|
||||||
think that ENGINE would encapsulate an implementation of arbitrarily many
|
|
||||||
of those algorithms - perhaps as alternatives to existing algorithms
|
|
||||||
and/or perhaps as new previously unimplemented algorithms. An ENGINE could
|
|
||||||
be used to contain an alternative software implementation, a wrapper for a
|
|
||||||
hardware acceleration and/or key-management unit, a comms-wrapper for
|
|
||||||
distributing cryptographic operations to remote machines, or any other
|
|
||||||
"devices" your imagination can dream up.
|
|
||||||
|
|
||||||
However, what has been done in the ENGINE branch so far is nothing more
|
|
||||||
than starting to get our toes wet. I had a couple of self-imposed
|
|
||||||
requirements when putting the initial abstraction together, and I may have
|
|
||||||
already posed these in one form or another on the list, but briefly;
|
|
||||||
|
|
||||||
(i) only bother with public key algorithms for now, and maybe RAND too
|
|
||||||
(motivated by the need to get hardware support going and the fact
|
|
||||||
this was a comparitively easy subset to address to begin with).
|
|
||||||
|
|
||||||
(ii) don't change (if at all possible) the existing crypto code, ie. the
|
|
||||||
implementations, the way the ***_METHODs work, etc.
|
|
||||||
|
|
||||||
(iii) ensure that if no function from the ENGINE code is ever called then
|
|
||||||
things work the way they always did, and there is no memory
|
|
||||||
allocation (otherwise the failure to cleanup would be a problem -
|
|
||||||
this is part of the reason no STACKs were used, the other part of
|
|
||||||
the reason being I found them inappropriate).
|
|
||||||
|
|
||||||
(iv) ensure that all the built-in crypto was encapsulated by one of
|
|
||||||
these "ENGINE"s and that this engine was automatically selected as
|
|
||||||
the default.
|
|
||||||
|
|
||||||
(v) provide the minimum hooking possible in the existing crypto code
|
|
||||||
so that global functions (eg. RSA_public_encrypt) do not need any
|
|
||||||
extra parameter, yet will use whatever the current default ENGINE
|
|
||||||
for that RSA key is, and that the default can be set "per-key"
|
|
||||||
and globally (new keys will assume the global default, and keys
|
|
||||||
without their own default will be operated on using the global
|
|
||||||
default). NB: Try and make (v) conflict as little as possible with
|
|
||||||
(ii). :-)
|
|
||||||
|
|
||||||
(vi) wrap the ENGINE code up in duct tape so you can't even see the
|
|
||||||
corners. Ie. expose no structures at all, just black-box pointers.
|
|
||||||
|
|
||||||
(v) maintain internally a list of ENGINEs on which a calling
|
|
||||||
application can iterate, interrogate, etc. Allow a calling
|
|
||||||
application to hook in new ENGINEs, remove ENGINEs from the list,
|
|
||||||
and enforce uniqueness within the global list of each ENGINE's
|
|
||||||
"unique id".
|
|
||||||
|
|
||||||
(vi) keep reference counts for everything - eg. this includes storing a
|
|
||||||
reference inside each RSA structure to the ENGINE that it uses.
|
|
||||||
This is freed when the RSA structure is destroyed, or has its
|
|
||||||
ENGINE explicitly changed. The net effect needs to be that at any
|
|
||||||
time, it is deterministic to know whether an ENGINE is in use or
|
|
||||||
can be safely removed (or unloaded in the case of the other type
|
|
||||||
of reference) without invalidating function pointers that may or
|
|
||||||
may not be used indavertently in the future. This was actually
|
|
||||||
one of the biggest problems to overcome in the existing OpenSSL
|
|
||||||
code - implementations had always been assumed to be ever-present,
|
|
||||||
so there was no trivial way to get round this.
|
|
||||||
|
|
||||||
(vii) distinguish between structural references and functional
|
|
||||||
references.
|
|
||||||
|
|
||||||
A *little* detail
|
|
||||||
-----------------
|
-----------------
|
||||||
|
|
||||||
While my mind is on it; I'll illustrate the bit in item (vii). This idea
|
This "description" (if one chooses to call it that) needed some major updating
|
||||||
turned out to be very handy - the ENGINEs themselves need to be operated
|
so here goes. This update addresses a change being made at the same time to
|
||||||
on and manipulated simply as objects without necessarily trying to
|
OpenSSL, and it pretty much completely restructures the underlying mechanics of
|
||||||
"enable" them for use. Eg. most host machines will not have the necessary
|
the "ENGINE" code. So it serves a double purpose of being a "ENGINE internals
|
||||||
hardware or software to support all the engines one might compile into
|
for masochists" document *and* a rather extensive commit log message. (I'd get
|
||||||
OpenSSL, yet it needs to be possible to iterate across the ENGINEs,
|
lynched for sticking all this in CHANGES or the commit mails :-).
|
||||||
querying their names, properties, etc - all happening in a thread-safe
|
|
||||||
manner that uses reference counts (if you imagine two threads iterating
|
|
||||||
through a list and one thread removing the ENGINE the other is currently
|
|
||||||
looking at - you can see the gotcha waiting to happen). For all of this,
|
|
||||||
*structural references* are used and operate much like the other reference
|
|
||||||
counts in OpenSSL.
|
|
||||||
|
|
||||||
The other kind of reference count is for *functional* references - these
|
ENGINE_TABLE underlies this restructuring, as described in the internal header
|
||||||
indicate a reference on which the caller can actually assume the
|
"eng_int.h", implemented in eng_table.c, and used in each of the "class" files;
|
||||||
particular ENGINE to be initialised and usable to perform the operations
|
tb_rsa.c, tb_dsa.c, etc.
|
||||||
it implements. Any increment or decrement of the functional reference
|
|
||||||
count automatically invokes a corresponding change in the structural
|
|
||||||
reference count, as it is fairly obvious that a functional reference is a
|
|
||||||
restricted case of a structural reference. So struct_ref >= funct_ref at
|
|
||||||
all times. NB: functional references are usually obtained by a call to
|
|
||||||
ENGINE_init(), but can also be created implicitly by calls that require a
|
|
||||||
new functional reference to be created, eg. ENGINE_set_default(). Either
|
|
||||||
way the only time the underlying ENGINE's "init" function is really called
|
|
||||||
is when the (functional) reference count increases to 1, similarly the
|
|
||||||
underlying "finish" handler is only called as the count goes down to 0.
|
|
||||||
The effect of this, for example, is that if you set the default ENGINE for
|
|
||||||
RSA operations to be "cswift", then its functional reference count will
|
|
||||||
already be at least 1 so the CryptoSwift shared-library and the card will
|
|
||||||
stay loaded and initialised until such time as all RSA keys using the
|
|
||||||
cswift ENGINE are changed or destroyed and the default ENGINE for RSA
|
|
||||||
operations has been changed. This prevents repeated thrashing of init and
|
|
||||||
finish handling if the count keeps getting down as far as zero.
|
|
||||||
|
|
||||||
Otherwise, the way the ENGINE code has been put together I think pretty
|
However, "EVP_CIPHER" underlies the motivation and design of ENGINE_TABLE so
|
||||||
much reflects the above points. The reason for the ENGINE structure having
|
I'll mention a bit about that first. EVP_CIPHER (and most of this applies
|
||||||
individual RSA_METHOD, DSA_METHOD, etc pointers is simply that it was the
|
equally to EVP_MD for digests) is both a "method" and a algorithm/mode
|
||||||
easiest way to go about things for now, to hook it all into the raw
|
identifier that, in the current API, "lingers". These cipher description +
|
||||||
RSA,DSA,etc code, and I was trying to the keep the structure invisible
|
implementation structures can be defined or obtained directly by applications,
|
||||||
anyway so that the way this is internally managed could be easily changed
|
or can be loaded "en masse" into EVP storage so that they can be catalogued and
|
||||||
later on when we start to work out what's to be done about these other
|
searched in various ways, ie. two ways of encrypting with the "des_cbc"
|
||||||
abstractions.
|
algorithm/mode pair are;
|
||||||
|
|
||||||
Down the line, if some EVP-based technique emerges for adequately
|
(i) directly;
|
||||||
encapsulating algorithms and all their various bits and pieces, then I can
|
const EVP_CIPHER *cipher = EVP_des_cbc();
|
||||||
imagine that "ENGINE" would turn into a reference-counting database of
|
EVP_EncryptInit(&ctx, cipher, key, iv);
|
||||||
these EVP things, of which the default "openssl" ENGINE would be the
|
[ ... use EVP_EncryptUpdate() and EVP_EncryptFinal() ...]
|
||||||
library's own object database of pre-built software implemented algorithms
|
|
||||||
(and such). It would also be cool to see the idea of "METHOD"s detached
|
|
||||||
from the algorithms themselves ... so RSA, DSA, ElGamal, etc can all
|
|
||||||
expose essentially the same METHOD (aka interface), which would include
|
|
||||||
any querying/flagging stuff to identify what the algorithm can/can't do,
|
|
||||||
its name, and other stuff like max/min block sizes, key sizes, etc. This
|
|
||||||
would result in ENGINE similarly detaching its internal database of
|
|
||||||
algorithm implementations from the function definitions that return
|
|
||||||
interfaces to them. I think ...
|
|
||||||
|
|
||||||
As for DSOs etc. Well the DSO code is pretty handy (but could be made much
|
(ii) indirectly;
|
||||||
more so) for loading vendor's driver-libraries and talking to them in some
|
OpenSSL_add_all_ciphers();
|
||||||
generic way, but right now there's still big problems associated with
|
cipher = EVP_get_cipherbyname("des_cbc");
|
||||||
actually putting OpenSSL code (ie. new ENGINEs, or anything else for that
|
EVP_EncryptInit(&ctx, cipher, key, iv);
|
||||||
matter) in dynamically loadable libraries. These problems won't go away in
|
[ ... etc ... ]
|
||||||
a hurry so I don't think we should expect to have any kind of
|
|
||||||
shared-library extensions any time soon - but solving the problems is a
|
|
||||||
good thing to aim for, and would as a side-effect probably help make
|
|
||||||
OpenSSL more usable as a shared-library itself (looking at the things
|
|
||||||
needed to do this will show you why).
|
|
||||||
|
|
||||||
One of the problems is that if you look at any of the ENGINE
|
The latter is more generally used because it also allows ciphers/digests to be
|
||||||
implementations, eg. hw_cswift.c or hw_ncipher.c, you'll see how it needs
|
looked up based on other identifiers which can be useful for automatic cipher
|
||||||
a variety of functionality and definitions from various areas of OpenSSL,
|
selection, eg. in SSL/TLS, or by user-controllable configuration.
|
||||||
including crypto/bn/, crypto/err/, crypto/ itself (locking for example),
|
|
||||||
crypto/dso/, crypto/engine/, crypto/rsa, etc etc etc. So if similar code
|
|
||||||
were to be suctioned off into shared libraries, the shared libraries would
|
|
||||||
either have to duplicate all the definitions and code and avoid loader
|
|
||||||
conflicts, or OpenSSL would have to somehow expose all that functionality
|
|
||||||
to the shared-library. If this isn't a big enough problem, the issue of
|
|
||||||
binary compatibility will be - anyone writing Apache modules can tell you
|
|
||||||
that (Ralf? Ben? :-). However, I don't think OpenSSL would need to be
|
|
||||||
quite so forgiving as Apache should be, so OpenSSL could simply tell its
|
|
||||||
version to the DSO and leave the DSO with the problem of deciding whether
|
|
||||||
to proceed or bail out for fear of binary incompatibilities.
|
|
||||||
|
|
||||||
Certainly one thing that would go a long way to addressing this is to
|
The important point about this is that EVP_CIPHER definitions and structures are
|
||||||
embark on a bit of an opaqueness mission. I've set the ENGINE code up with
|
passed around with impunity and there is no safe way, without requiring massive
|
||||||
this in mind - it's so draconian that even to declare your own ENGINE, you
|
rewrites of many applications, to assume that EVP_CIPHERs can be reference
|
||||||
have to get the engine code to create the underlying ENGINE structure, and
|
counted. One an EVP_CIPHER is exposed to the caller, neither it nor anything it
|
||||||
then feed in the new ENGINE's function/method pointers through various
|
comes from can "safely" be destroyed. Unless of course the way of getting to
|
||||||
"set" functions. The more of the code that takes on such a black-box
|
such ciphers is via entirely distinct API calls that didn't exist before.
|
||||||
approach, the more of the code that will be (a) easy to expose to shared
|
However existing API usage cannot be made to understand when an EVP_CIPHER
|
||||||
libraries that need it, and (b) easy to expose to applications wanting to
|
pointer, that has been passed to the caller, is no longer being used.
|
||||||
use OpenSSL itself as a shared-library. From my own explorations in
|
|
||||||
OpenSSL, the biggest leviathan I've seen that is a problem in this respect
|
|
||||||
is the BIGNUM code. Trying to "expose" the bignum code through any kind of
|
|
||||||
organised "METHODs", let alone do all the necessary bignum operations
|
|
||||||
solely through functions rather than direct access to the structures and
|
|
||||||
macros, will be a massive pain in the "r"s.
|
|
||||||
|
|
||||||
Anyway, I'm done for now - hope it was readable. Thoughts?
|
The other problem with the existing API w.r.t. to hooking EVP_CIPHER support
|
||||||
|
into ENGINE is storage - the OBJ_NAME-based storage used by EVP to register
|
||||||
|
ciphers simultaneously registers cipher *types* and cipher *implementations* -
|
||||||
|
they are effectively the same thing, an "EVP_CIPHER" pointer. The problem with
|
||||||
|
hooking in ENGINEs is that multiple ENGINEs may implement the same ciphers. The
|
||||||
|
solution is necessarily that ENGINE-provided ciphers simply are not registered,
|
||||||
|
stored, or exposed to the caller in the same manner as existing ciphers. This is
|
||||||
|
especially necessary considering the fact ENGINE uses reference counts to allow
|
||||||
|
for cleanup, modularity, and DSO support - yet EVP_CIPHERs, as exposed to
|
||||||
|
callers in the current API, support no such controls.
|
||||||
|
|
||||||
Cheers,
|
Another sticking point for integrating cipher support into ENGINE is linkage.
|
||||||
Geoff
|
Already there is a problem with the way ENGINE supports RSA, DSA, etc whereby
|
||||||
|
they are available *because* they're part of a giant ENGINE called "openssl".
|
||||||
|
Ie. all implementations *have* to come from an ENGINE, but we get round that by
|
||||||
|
having a giant ENGINE with all the software support encapsulated. This creates
|
||||||
|
linker hassles if nothing else - linking a 1-line application that calls 2 basic
|
||||||
|
RSA functions (eg. "RSA_free(RSA_new());") will result in large quantities of
|
||||||
|
ENGINE code being linked in *and* because of that DSA, DH, and RAND also. If we
|
||||||
|
continue with this approach for EVP_CIPHER support (even if it *was* possible)
|
||||||
|
we would lose our ability to link selectively by selectively loading certain
|
||||||
|
implementations of certain functionality. Touching any part of any kind of
|
||||||
|
crypto would result in massive static linkage of everything else. So the
|
||||||
|
solution is to change the way ENGINE feeds existing "classes", ie. how the
|
||||||
|
hooking to ENGINE works from RSA, DSA, DH, RAND, as well as adding new hooking
|
||||||
|
for EVP_CIPHER, and EVP_MD.
|
||||||
|
|
||||||
|
The way this is now being done is by mostly reverting back to how things used to
|
||||||
|
work prior to ENGINE :-). Ie. RSA now has a "RSA_METHOD" pointer again - this
|
||||||
|
was previously replaced by an "ENGINE" pointer and all RSA code that required
|
||||||
|
the RSA_METHOD would call ENGINE_get_RSA() each time on its ENGINE handle to
|
||||||
|
temporarily get and use the ENGINE's RSA implementation. Apart from being more
|
||||||
|
efficient, switching back to each RSA having an RSA_METHOD pointer also allows
|
||||||
|
us to conceivably operate with *no* ENGINE. As we'll see, this removes any need
|
||||||
|
for a fallback ENGINE that encapsulates default implementations - we can simply
|
||||||
|
have our RSA structure pointing its RSA_METHOD pointer to the software
|
||||||
|
implementation and have its ENGINE pointer set to NULL.
|
||||||
|
|
||||||
-----------------------------------==*==-----------------------------------
|
A look at the EVP_CIPHER hooking is most explanatory, the RSA, DSA (etc) cases
|
||||||
|
turn out to be degenerate forms of the same thing. The EVP storage of ciphers,
|
||||||
|
and the existing EVP API functions that return "software" implementations and
|
||||||
|
descriptions remain untouched. However, the storage takes more meaning in terms
|
||||||
|
of "cipher description" and less meaning in terms of "implementation". When an
|
||||||
|
EVP_CIPHER_CTX is actually initialised with an EVP_CIPHER method and is about to
|
||||||
|
begin en/decryption, the hooking to ENGINE comes into play. What happens is that
|
||||||
|
cipher-specific ENGINE code is asked for an ENGINE pointer (a functional
|
||||||
|
reference) for any ENGINE that is registered to perform the algo/mode that the
|
||||||
|
provided EVP_CIPHER structure represents. Under normal circumstances, that
|
||||||
|
ENGINE code will return NULL because no ENGINEs will have had any cipher
|
||||||
|
implementations *registered*. As such, a NULL ENGINE pointer is stored in the
|
||||||
|
EVP_CIPHER_CTX context, and the EVP_CIPHER structure is left hooked into the
|
||||||
|
context and so is used as the implementation. Pretty much how things work now
|
||||||
|
except we'd have a redundant ENGINE pointer set to NULL and doing nothing.
|
||||||
|
|
||||||
|
Conversely, if an ENGINE *has* been registered to perform the algorithm/mode
|
||||||
|
combination represented by the provided EVP_CIPHER, then a functional reference
|
||||||
|
to that ENGINE will be returned to the EVP_CIPHER_CTX during initialisation.
|
||||||
|
That functional reference will be stored in the context (and released on
|
||||||
|
cleanup) - and having that reference provides a *safe* way to use an EVP_CIPHER
|
||||||
|
definition that is private to the ENGINE. Ie. the EVP_CIPHER provided by the
|
||||||
|
application will actually be replaced by an EVP_CIPHER from the registered
|
||||||
|
ENGINE - it will support the same algorithm/mode as the original but will be a
|
||||||
|
completely different implementation. Because this EVP_CIPHER isn't stored in the
|
||||||
|
EVP storage, nor is it returned to applications from traditional API functions,
|
||||||
|
there is no associated problem with it not having reference counts. And of
|
||||||
|
course, when one of these "private" cipher implementations is hooked into
|
||||||
|
EVP_CIPHER_CTX, it is done whilst the EVP_CIPHER_CTX holds a functional
|
||||||
|
reference to the ENGINE that owns it, thus the use of the ENGINE's EVP_CIPHER is
|
||||||
|
safe.
|
||||||
|
|
||||||
|
The "cipher-specific ENGINE code" I mentioned is implemented in tb_cipher.c but
|
||||||
|
in essence it is simply an instantiation of "ENGINE_TABLE" code for use by
|
||||||
|
EVP_CIPHER code. tb_digest.c is virtually identical but, of course, it is for
|
||||||
|
use by EVP_MD code. Ditto for tb_rsa.c, tb_dsa.c, etc. These instantiations of
|
||||||
|
ENGINE_TABLE essentially provide linker-separation of the classes so that even
|
||||||
|
if ENGINEs implement *all* possible algorithms, an application using only
|
||||||
|
EVP_CIPHER code will link at most code relating to EVP_CIPHER, tb_cipher.c, core
|
||||||
|
ENGINE code that is independant of class, and of course the ENGINE
|
||||||
|
implementation that the application loaded. It will *not* however link any
|
||||||
|
class-specific ENGINE code for digests, RSA, etc nor will it bleed over into
|
||||||
|
other APIs, such as the RSA/DSA/etc library code.
|
||||||
|
|
||||||
|
ENGINE_TABLE is a little more complicated than may seem necessary but this is
|
||||||
|
mostly to avoid a lot of "init()"-thrashing on ENGINEs (that may have to load
|
||||||
|
DSOs, and other expensive setup that shouldn't be thrashed unnecessarily) *and*
|
||||||
|
to duplicate "default" behaviour. Basically an ENGINE_TABLE instantiation, for
|
||||||
|
example tb_cipher.c, implements a hash-table keyed by integer "nid" values.
|
||||||
|
These nids provide the uniquenness of an algorithm/mode - and each nid will hash
|
||||||
|
to a potentially NULL "ENGINE_PILE". An ENGINE_PILE is essentially a list of
|
||||||
|
pointers to ENGINEs that implement that particular 'nid'. Each "pile" uses some
|
||||||
|
caching tricks such that requests on that 'nid' will be cached and all future
|
||||||
|
requests will return immediately (well, at least with minimal operation) unless
|
||||||
|
a change is made to the pile, eg. perhaps an ENGINE was unloaded. The reason is
|
||||||
|
that an application could have support for 10 ENGINEs statically linked
|
||||||
|
in, and the machine in question may not have any of the hardware those 10
|
||||||
|
ENGINEs support. If each of those ENGINEs has a "des_cbc" implementation, we
|
||||||
|
want to avoid every EVP_CIPHER_CTX setup from trying (and failing) to initialise
|
||||||
|
each of those 10 ENGINEs. Instead, the first such request will try to do that
|
||||||
|
and will either return (and cache) a NULL ENGINE pointer or will return a
|
||||||
|
functional reference to the first that successfully initialised. In the latter
|
||||||
|
case it will also cache an extra functional reference to the ENGINE as a
|
||||||
|
"default" for that 'nid'. The caching is acknowledged by a 'uptodate' variable
|
||||||
|
that is unset only if un/registration takes place on that pile. Ie. if
|
||||||
|
implementations of "des_cbc" are added or removed. This behaviour can be
|
||||||
|
tweaked; the ENGINE_TABLE_FLAG_NOINIT value can be passed to
|
||||||
|
ENGINE_set_table_flags(), in which case the only ENGINEs that tb_cipher.c will
|
||||||
|
try to initialise from the "pile" will be those that are already initialised
|
||||||
|
(ie. it's simply an increment of the functional reference count, and no real
|
||||||
|
"initialisation" will take place).
|
||||||
|
|
||||||
|
RSA, DSA, DH, and RAND all have their own ENGINE_TABLE code as well, and the
|
||||||
|
difference is that they all use an implicit 'nid' of 1. Whereas EVP_CIPHERs are
|
||||||
|
actually qualitatively different depending on 'nid' (the "des_cbc" EVP_CIPHER is
|
||||||
|
not an interoperable implementation of "aes_256_cbc"), RSA_METHODs are
|
||||||
|
necessarily interoperable and don't have different flavours, only different
|
||||||
|
implementations. In other words, the ENGINE_TABLE for RSA will either be empty,
|
||||||
|
or will have a single ENGING_PILE hashed to by the 'nid' 1 and that pile
|
||||||
|
represents ENGINEs that implement the single "type" of RSA there is.
|
||||||
|
|
||||||
|
Cleanup - the registration and unregistration may pose questions about how
|
||||||
|
cleanup works with the ENGINE_PILE doing all this caching nonsense (ie. when the
|
||||||
|
application or EVP_CIPHER code releases its last reference to an ENGINE, the
|
||||||
|
ENGINE_PILE code may still have references and thus those ENGINEs will stay
|
||||||
|
hooked in forever). The way this is handled is via "unregistration". With these
|
||||||
|
new ENGINE changes, an abstract ENGINE can be loaded and initialised, but that
|
||||||
|
is an algorithm-agnostic process. Even if initialised, it will not have
|
||||||
|
registered any of its implementations (to do so would link all class "table"
|
||||||
|
code despite the fact the application may use only ciphers, for example). This
|
||||||
|
is deliberately a distinct step. Moreover, registration and unregistration has
|
||||||
|
nothing to do with whether an ENGINE is *functional* or not (ie. you can even
|
||||||
|
register an ENGINE and its implementations without it being operational, you may
|
||||||
|
not even have the drivers to make it operate). What actually happens with
|
||||||
|
respect to cleanup is managed inside eng_lib.c with the "engine_cleanup_***"
|
||||||
|
functions. These functions are internal-only and each part of ENGINE code that
|
||||||
|
could require cleanup will, upon performing its first allocation, register a
|
||||||
|
callback with the "engine_cleanup" code. The other part of this that makes it
|
||||||
|
tick is that the ENGINE_TABLE instantiations (tb_***.c) use NULL as their
|
||||||
|
initialised state. So if RSA code asks for an ENGINE and no ENGINE has
|
||||||
|
registered an implementation, the code will simply return NULL and the tb_rsa.c
|
||||||
|
state will be unchanged. Thus, no cleanup is required unless registration takes
|
||||||
|
place. ENGINE_cleanup() will simply iterate across a list of registered cleanup
|
||||||
|
callbacks calling each in turn, and will then internally delete its own storage
|
||||||
|
(a STACK). When a cleanup callback is next registered (eg. if the cleanup() is
|
||||||
|
part of a gracefull restart and the application wants to cleanup all state then
|
||||||
|
start again), the internal STACK storage will be freshly allocated. This is much
|
||||||
|
the same as the situation in the ENGINE_TABLE instantiations ... NULL is the
|
||||||
|
initialised state, so only modification operations (not queries) will cause that
|
||||||
|
code to have to register a cleanup.
|
||||||
|
|
||||||
|
What else? The bignum callbacks and associated ENGINE functions have been
|
||||||
|
removed for two obvious reasons; (i) there was no way to generalise them to the
|
||||||
|
mechanism now used by RSA/DSA/..., because there's no such thing as a BIGNUM
|
||||||
|
method, and (ii) because of (i), there was no meaningful way for library or
|
||||||
|
application code to automatically hook and use ENGINE supplied bignum functions
|
||||||
|
anyway. Also, ENGINE_cpy() has been removed (although an internal-only version
|
||||||
|
exists) - the idea of providing an ENGINE_cpy() function probably wasn't a good
|
||||||
|
one and now certainly doesn't make sense in any generalised way. Some of the
|
||||||
|
RSA, DSA, DH, and RAND functions that were fiddled during the original ENGINE
|
||||||
|
changes have now, as a consequence, been reverted back. This is because the
|
||||||
|
hooking of ENGINE is now automatic (and passive, it can interally use a NULL
|
||||||
|
ENGINE pointer to simply ignore ENGINE from then on).
|
||||||
|
|
||||||
|
Hell, that should be enough for now ... comments welcome: geoff@openssl.org
|
||||||
|
|
||||||
|
Loading…
x
Reference in New Issue
Block a user