john-dev - Help! mskrb5 64-32-bit builds crashing hard while native builds does not?

Follow @Openwall on Twitter for new release announcements and other news

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Date: Sun, 08 May 2011 17:01:00 +0200
From: magnum <rawsmooth@...dband.net>
To: john-dev@...ts.openwall.com
Subject: Help! mskrb5 64-32-bit builds crashing hard while native builds does
 not?

I have a weird problem. My mskrb5 format works fine on 64-bit and 
32-bit, but not when building a 64-32 target and running it on 64-bit.

It smashes the stack. When I trace it, everything is fine until I call 
RC4_set_key() the first time, after that stack is crap (I can't step 
inside RC4_set_key, maybe I should look for a debug version of the lib)

Is there anything wrong with this code? I can't see what, but maybe it's 
something really silly as usual. Disregard the OMP pragma, it happens 
with or without OMP. But why does it NOT happen on a 32-bit machine? Is 
that a clue?

static void crypt_all(int count)
{
   HMACMD5Context ctx;
   unsigned char K3[KEY_SIZE], cleartext[PLAINTEXT_OFFSET + BINARY_SIZE];
   RC4_KEY key;
   int i;

#ifdef _OPENMP
#pragma omp parallel for default(none) private(i, ctx, K3, cleartext, 
key) shared(count, output, saved_key, saltblob)
#endif
   for (i = 0; i < count; i++) {
     // K1 is stored in saved_key[index]
     // CHECKSUM and TIMESTAMP are just defines, they are actually
     // concatenated to saltblob[]

     // K3 = HMAC-MD5(K1, CHECKSUM)
     hmac_md5(saved_key[i], CHECKSUM, CHECKSUM_SIZE, K3, &ctx);

     // RC4(K3, TIMESTAMP) decrypt part of the timestamp
     RC4_set_key(&key, KEY_SIZE, K3);
     RC4(&key, PLAINTEXT_OFFSET + BINARY_SIZE, TIMESTAMP,
         cleartext);

     // 15th byte and on is our partial binary
     memcpy(output[i], &cleartext[PLAINTEXT_OFFSET], BINARY_SIZE);
   }
}

KEY_SIZE is 16. I have tried defining K3, cleartext, saved_key and 
output as static and/or double the size they need and this does not 
change a thing.

I just upgraded my Ubuntu for newer gcc and (I guess) openssl and 
nothing changed.

Any hints welcome! This drives me crazy. All other formats works fine.

magnum

Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.