Re: [pkcs11] Points about AEAD and Counter mode APIs - recap from today

[Michael StJohns <msj@nthpermutation.com>]

Please excuse the top posting.  I'd actually spent a bit of time commenting on your comments and realized that we were just moving the deck chairs around.  Suffice it to say that I don't think the current proposal is correct enough and I don't think it's easy to morph to something that is.

One of the reasons for that is that I realized we actually had three issues, not two:

The third issue I didn't call out before is the need for a per-message encrypt/decrypt or sign/verify.  This is different, but related to the per-message IV generator stuff and has a lot of the same considerations, but it is ultimately driven by the proliferation of network based HSMs. Performance for systems based on these devices is going to be constrained by the number of round trips necessary to do an encryption.  The classic Init/Update/Encrypt on a per message basis is expensive in those terms.  I grok this is actually more of a concern than the IV generation.

So (without considering IV generation or AEAD at this point):

What do we need to encrypt/decrypt each message that is common across all messages?

a) The key
b) The mechanism

What do we need to encrypt/decrypt each message that is specific to the message?:

a) An IV
b) Data to be encrypted/decrypted

So to make this work, we need a call to set up persistent state consisting of the key and the mechanism, and that state either has to be implicit to the session, or explicit to an object passed in to the per-message call(s).  Wan-teh proposes this be part of the implicit session state, I'd prefer it to be an explicit state, but I can live with his approach.

So this gets me to:

C_NewAssociation (CK_SESSION_HANDLE hSession, CK_MECHANISM_PTR pMechanism, CK_OBJECT_HANDLE hKey);

which sets up implicit session state containing the key and mechanism.  It also tells the session the types of calls it will be getting (so the proper error messages are returned if there are attempts to use intermediate C_EncryptUpdate calls for example).

And

C_EncryptMessage (CK_SESSION_HANDLE hSession, CK_VOID_PTR pIV, CK_BOOLEAN bLastSegment,
    CK_BYTE_PTR pData, CK_ULONG ulDataLen, CK_BYTE_PTR pEncryptedData, CK_ULONG_PTR pulEncryptedDataLen);

C_DecryptMessage (CK_SESSION_HANDLE hSession, CK_VOID_PTR pIV, CK_BOOLEAN bLastSegment,
    CK_BYTE_PTR pEncryptedData, CK_ULONG ulEncryptedData, CK_BYTE_PTR pData, CK_ULONG_PTR pulDataLen);

C_SignMessage (CK_SESSION_HANDLE hSession, CK_BOOLEAN pFirstSegment, CK_BOOLEAN bLastSegment,
    CK_BYTE_PTR pData, CK_ULONG ulDataLen, CK_BYTE_PTR pIntegrityTag, CK_ULONG_PTR pulIntegrityTagLen);

C_VerifyMessage(CK_SESSION_HANDLE hSession, CK_BOOLEAN pFirstSegment, CK_BOOLEAN bLastSegment,
      CK_BYTE_PTR pData, CK_ULONG ulDataLen, CK_BYTE_PTR pIntegrityTag, CK_ULONG ulIntegrityTagLen);

where

pIV is the pointer to the iv, format of this defined by the mechanism (doing it this way to allow for IV generator later).  If iv is non-null, then the current segment is the first segment of the message

bLastSegment is a boolean which indicates this is the last segment of message and that padding etc should be done prior to emitting the last of the cipher text.

if pIV is null (or pFirstSegment is false) and bLastSegment is false, this is equivalent to an update operation.

if pIV is non-null (or pFirstSegment is true)  and bLastSegment is true, this segment is the complete message.

All of the above act pretty much as macros for a C_*Init/C_*Update/C_*Final set of calls.

C_NewAssociation uses the same calling conventions as C_EncryptInit.

C_CloseAssociation (CK_SESSION_HANDLE hSession) is used to close and discard persistent key and mechanism state.


Second issue - AEAD ciphers.   Two sub cases - generic (Mods to C_EncryptInit/C_EncryptUpdate/C_EncryptFinal calls) and message (Mods to C_EncryptMessage).

The first case is one not addressed by Wan Teh - how to fit AEAD ciphers into the current non-association model.

AEAD is complex because of the wide range of possible approaches (sign over ciphertext - i.e. after encryption, sign over plain text, AAD before encrypted data, AAD after encrypted data, AAD mixed in with encrypted data) and side issues like McGrew's API proposals.    Also, there are a pair of functions that already exist that are somewhat applicable C_SignEncryptUpdate and C_DecryptVerifyUpdate.  For below I've left sign before encrypt vs encrypt before sign as part of the mechanism definition - so there's no indication of the order in the following calls.

Again leaving the IV generation issues to later and dealing first with the current API model we create two new initialization functions and two new final functions:

C_SignEncryptInit (CK_SESSION_HANDLE hSession,  CK_MECHANISM_PTR pMechanism, CK_OBJECT_HANDLE hKey,
        CK_VOID_PTR pIV,
        CK_BYTE_PTR aadPrefixData, CK_ULONG aadPrefixDataLen);

C_DecryptVerifyInit (CK_SESSION_HANDLE hSession,  CK_MECHANISM_PTR pMechanism, CK_OBJECT_HANDLE hKey,
        CK_VOID_PTR pIV,
        CK_BYTE_PTR aadPrefixData, CK_ULONG aadPrefixDataLen);

In the above, I've called out the IV as an explicit argument.  Doing it the way I did above means that we can probably define mechanisms that work both with an IV generator and with a supplied IV. 

The existing C_SignEncryptUpdate and C_DecryptVerifyUpdate can be used to process intermediate data.

The final calls are used to pad, encrypt, output and to emit the integrity tag (or the reverse order for decrypt/verify).

C_SignEncryptFinal (CK_SESSION_HANDLE hSession, 
        CK_BYTE_PTR aadSuffixData, CK_ULONG aadSuffixDataLen,
        CK_BYTE_PTR pLastEncryptedPart, CK_ULONG_PTR pulLastEncryptedPartLen,
        CK_BYTE_PTR pIntegrityTag, CK_ULONG_PTR pulIntegrityTagLen);

C_DecryptVerifyFinal (CK_SESSION_HANDLE hSession,
        CK_BYTE_PTR aadSuffixData, CK_ULONG aadSuffixDataLen,
        CK_BYTE_PTR pLastDecryptedPart, CK_ULONG_PTR pulLastDecryptedPartLen,
        CK_BYTE_PTR pIntegrityTag, CK_ULONG ulIntegrityTagLen);


The AAD can preceed and/or follow the data to be encrypted.  That means that a generic API can handle McGrew's weird  AAD | ciphertext | L(AAD) construct as an extension to the generic CCM and GCM models.  The mechanism MAY constrain this, but I wouldn't define any that does.

Continuing on with AEAD,  for the message models we get  new message functions (same C_NewAssociation, C_CloseAssociation functions plus):


C_EncryptSignMessage (CK_SESSION_HANDLE hSession, CK_VOID_PTR iv, CK_BOOLEAN bLastSegment,
    CK_BYTE_PTR pPrefixAADData, CK_ULONG ulPrefixAADDataLen,
    CK_BYTE_PTR pData, CK_ULONG ulDataLen, CK_BYTE_PTR pEncryptedData, CK_ULONG_PTR pulEncryptedDataLen
    CK_BYTE_PTR pSuffixAADData, CK_ULONG ulSuffixAADDataLen,
    CK_BYTE_PTR pIntegrityTag, CK_ULONG_PTR pulIntegrityTagLen);

C_DecryptVerifyMessage (CK_SESSION_HANDLE hSession, CK_VOID_PTR iv, CK_BOOLEAN bLastSegment,
    CK_BYTE_PTR pPrefixAADData, CK_ULONG ulPrefixAADDataLen,
    CK_BYTE_PTR pEncryptedData, CK_ULONG ulEncryptedData, CK_BYTE_PTR pData, CK_ULONG_PTR pulDataLen
    CK_BYTE_PTR pSuffixAADData, CK_ULONG ulSuffixAADDataLen,
    CK_BYTE_PTR pIntegrityTag, CK_ULONG_PTR pulIntegrityTagLen);

where

pIV is the pointer to the iv, format of this defined by the mechanism (doing it this way to allow for IV generator later).  If iv is non-null, then the current segment is the first segment of the message

bLastSegment is a boolean which indicates this is the last segment of message and that padding etc should be done prior to emitting the last of the cipher text.

if pIV is null and bLastSegment is false, this is equivalent to an update operation.

if pIV is non-null and bLastSegment is true, this segment is the complete message.

AAD can be both prefixed and suffixed to the encrypted message.

Finally, we get to actually the least interesting of all of these - IV generation. 


One of the things I'm concerned with is that the IV generation model differs from protocol to protocol, rather than from mode to mode.  So defining one IV generator as part of the cryptographic mode model seems to short sighted.   And it also doesn't work all that well if you consider that the decrypt operation can't always generate the IV, but is getting it as supplied in the message (or derived from protocol elements from all over the message).

This is where my specification of "CK_VOID_PTR pIV" above comes in:

1) Existing mechanisms that specify the IV as part of the mechanism, continue to take their IVs from the mechanism structure for the non-message functions.  For the existing mechanisms using the message functions, the length of the IV is derived from the mechanism  or parameter data and the per-message IV is specified as if bExplicitIV = CK_TRUE were specified (see 2 below).

2) New mechanisms that want to use an IV generator will include these two fields in their mechanism params:

a) CK_BOOLEAN bExplicitIV
b) CK_ULONG ivLength

If bExplicitIV is false, then pIV on the init call is a pointer to handle for an IV generator object.

if bExplicitIV is true, then pIV on the init call is a pointer to a CK_BYTE - an array of IV bytes of length ivLength.


IV generator objects are session storage objects.  They have a specific generator mechanism and specific set of attributes that are set at creation (and can be reset for new associations).  They have a way to read the next generated and current IVs (for inclusion in the protocol message for example).

New types of generator objects can be defined as new protocols are written - but for the start we will define the pseudo-standard CCM (Appendix A counter format function), GCM and CTR generic models.


This approach

a) allows for any length of message or encrypted stream (e.g. while the mechanism may restrict this, the API doesn't)
b) makes no assumption about whether or not decrypted data may be released before the integrity tag is verified. (again, mechanism or mode of HSM may restrict this).
c) has no requirement for pre-call marshalling and formatting of the crypto stream (e.g. no assumptions about exactly where the integrity tag is and its length nor where the AAD is and its length - no memory to memory copies should be required prior to the call).
d) can deal with AAD preceeding and following the ciphertext/data to be encrypted (future proofing)
e) handles message based encryption as a separate and backwards compatible item from AAD and IV generation. (e.g. can actually do per-message CBC stuff within an association using new calls).
f) divorces IV generation specification  from mechanism specification - same mechanism can use multiple IV generation mechanisms (future proofing).
g) mirrors, extends or combines existing functions so easy transitions for future programming.
h) possible to use IV generator for decrypt, but not required.
i) includes functions to do message verification and signature


My general desire here is that we don't want to have to re-do these things because someone has come up with yet another weird AEAD approach and our APIs won't support it.  I want to be able to break things down to composable building blocks.   I also want to stay as close to the original spec (e.g. for CCM and GCM the NIST documents) as possible rather than possibly niche documents like the McGrew API.

Or to put it another way - the difference between an engineer and a bureaucrat is that an engineer takes large insolvable problems and breaks them down into small solvable pieces, where a bureaucrat takes a bunch of small solvable pieces and combines them into one large insolvable mass.    I'm trying to keep the pieces useful.

Mike

On 2/5/2014 7:16 PM, Wan-Teh Chang wrote:

Hi Michael,

Thank you for your comments.

On Wed, Feb 5, 2014 at 2:55 PM, Michael StJohns <msj@nthpermutation.com> wrote:

1) Two issues:
   - per-message IV generation for Counter-based mechanisms
    - Support for AEAD input of AAD and input/output of integrity tag.

2) IV Generation
    - not specific to AEAD
    - Applies especially to counter mode based mechanisms (CCM, GCM, CTR)
        as IV duplication is deadly.
    - but there's a general requirement for "unpredictable" generation of
IVs for
      most mechanisms (e.g. could use for CBC and other modes that use IVs)
    - CCM and GCM have *example* IV generation mechanisms that are used
       widely (e.g. CCM appendix A), but the specifications do no restrict
to those
       mechanisms.
        -- E.g. need approach that doesn't lock encryption mechanism to IV
            generation mechanism. (Decomposition of functions).

This is supported by the new proposal I posted today
(pkcs11-encryption-v2-api.txt).

I defined a new AES-GCM mechanism:

#define CKM_AES_GCM_V2 0x00000700

that uses a new mechanism parameters structure:

typedef struct CK_GCM_PARAMS_V2 {
  CK_ULONG ulNonceLen;
  CK_ULONG ulTagLen;
  CK_MECHANISM_PTR pNonceMech;  /* For encryption:
                                 * - If NULL, the entire nonce is provided by
                                 *   the caller.
                                 * - If not NULL, specifies how the nonce is
                                 *   generated partially or fully by the crypto
                                 *   module.
                                 *
                                 * For decryption: must be NULL and the entire
                                 * nonce is provided by the caller. */
} CK_GCM_PARAMS_V2;

The pNonceMech field specifies the IV generation mechanism. So you can
use CKM_AES_GCM_V2 (an encryption algorithm) with any suitable IV
generation mechanism. I specified two IV generation mechanisms that
may be used with CKM_AES_GCM_V2:

#define CKM_GCM_NONCE_DETERMINISTIC   0x00000750
#define CKM_GCM_NONCE_RBG_BASED       0x00000751

and they each have their own mechanism parameters structures.

    - IV generation state, unlike cryptographic message state, is persistent
across
        a "communication"
    - Current C_Encrypt/Update/Final APIs designed for per-message
cryptographic state
       and NOT persistent inter-message state throughout a communication.
    - Need new C_NewAssociation, C_CloseAssociation functions to wrap key,
encryption
      mechanism and IV generator object/mechanism (object because of
persistence issues)

I know you suggested this before. This is the only suggestion of yours
that we didn't implement. We continue to associate the key, encryption
mechanism, and IV generator object (if created by a parameter of the
encryption mechanism such as the pNonceMech field of CK_GCM_PARAMS_V2)
with the session implicitly. The IV generator object (if created by
the encryption mechanism) persists across messages, until the new
C_EncryptV2Final function is called.

    - Use new C_EncryptMessage or C_Encrypt/Update/Final/Message construct
with
      passed in association handle (vs current passed in session handle).
Can be macro
      for existing functions.

The new proposal has the new C_EncryptMessage or
C_Encrypt/Update/Final/Message functions. They are poorly named:
C_EncryptV2 for single-part mode and the { C_EncryptV2Begin,
C_EncryptV2Update, C_EncryptV2End } functions for multi-part mode. The
only difference that they use the implicit association in the session
rather than an explicit association handle.

3) AEAD
    - Could use marshalled cryptostream  (which includes AAD and integrity
tag) (e.g. mcgrews API),
        - but not necessarily future proof.
        - can require memory-memory copies (expensive and an issue for
things like scatter/gather processing)
        - depends on documents specific to TLS

(Although the only use case in my presentation is TLS, I also studied
how AES-GCM is used in IPsec: http://tools.ietf.org/html/rfc5282.)

This is all correct. I am going for simplicity of the API. The current
proposal allows us to use the same functions for AEAD and non-AEAD
mechanisms. Alternatively, we can add AEAD-specific functions that
have an integrity tag output argument, but one can also imagine that
the output of an AEAD algorithm may not have a cleanly separable
integrity tag.

Note: In the conference call today I asked if there is a protocol that
marshalls AES-GCM ciphertext in a different way from what David McGrew
specified. I just found one: http://tools.ietf.org/html/rfc5083. So a
memory-memory copy of a <= 16-byte integrity tag would be necessary
for AES-GCM for S/MIME. That seems acceptable.

    - Better to use separate args for AAD and integrity tag

Although integrity tag is not a separate argument, AAD is a separate
argument in my proposal.

    - Although NIST FIPS requires that decrypted data not be released unless
integrity
        tag verifies (CCM and GCM), this SHOULD NOT BE A RESTRICTION OF THE
API as
        other future AEAD functions may not have this restriction.
        -- Provide Init/Update/Final to handle large messages, or even
messages that
           underlying implementation can't handle (e.g. size of internal
buffer)
        -- enforce release of decrypted data as a mechanism specific or mode
(FIPS) specific
           requirement

The new proposal supports this. Each mechanism independently specifies
whether it allows multi-part mode.

In summary, I believe that among your suggestions, only two are not
implemented by the new proposal:
- Introduce the notion of an association handle, as opposed to relying
on the single implicit association in a session.
- Use a separate output argument for integrity tag for AEAD.

Wan-Teh Chang