Openwall GNU/*/Linux - a small security-enhanced Linux distro for servers
[<prev] [next>] [day] [month] [year] [list]
Date: Mon, 27 Jul 2009 06:01:04 -0500
From: "JimF" <jfoug@....net>
To: <john-users@...ts.openwall.com>
Subject: Generation method I used for prepend and append 'rules'

Here are the steps I used to build a nice prepend / append
ruleset for usage in John.   This information is being
provided as is.  It was thrown together, to satisfy a
personal test (successful) of my own. It was coded quickly,
and not done in a very clean manner.  I make no apologies
for it.  use it or modify it to your own needs. Or simply
use the rules I have provided, prebuilt.  

NOTE this method can crack some pretty long, and pretty 
complex passwords.

First a little information.  The data used, was a set of 
cracked passwords. There were NOT a simple john.pot, but
a list of all the passwords used. If there were multiple
words of the same, I wanted to have each multiple copy.
To do this, I took the cracked list, and used john -show
and piped that to a file. Then used cut  So

Step 1.  Crack a lot of passwords, or get a good cracked db

Step 2.  Run john -show -format=???  hashes_lst > crk

Step 3.  Get 'just' the passwords:   cut -d: -f2 crk > pass_all

Ok, now we have a list of the cracked passwords. What we now
want to do, is to 'find' things that are added to normal 
dictionary words (the database I worked from was mostly 
English, so I used English words). Obtain the proper language
dictionaries from openwall.net  For this example, I used
all of the dictionaries from the English directories.

Step 4.  Get the dictionary words.  I concatenated all the 
word files from 'English' section from the openwall lists.

Step 5.  Sort this dictionary list.  The sorted file is 
named english-sorted.all


Step 6.  Run the program pre_post_cnt and redirect output. NOTE 
as in original source, the file names pass_all and 
english-sorted.all ARE required names.

   pre_post_cnt > data    

the pre_post_cnt.cpp is provided. It is a simple hacked out
program, not well done, but it works (for me ;)  The output
will be 3 fields The fields are:
count 'string' (Type)    Type will be (Pre) or (Post). As the
program runs (it may take a couple minutes), it will inform
you to stderr what it is doing (which password is being worked
on).  When all are done, it will output and exit.
NOTE this file will only output (as written), if there are 11
instances or more of a prepend/append sequence.  This can be 
adjusted by editing the source.

Step 7.  Grep out (Pre)   grep (Pre) data > pre_data
Step 8.  grep out (Post)  grep (Post) data > post_data
Note, I use wc to make sure that all of the lines were accounted
for, they should be, but it never hurts to check.

Step 9.  Now, sort both per and post data

sort -n -r pre_data  > prepend-lst
sort -n -r post_data > append-lst

The default pre_post_cnt.cpp only outputs if > 10 'hits', but 
if you have coded to see ALL of them, you most likely will want
to remove all of the singleton and probably even more from your
prepend-lst and append-lst files.

Ok, we are almost home.

Step 10, build prepend.lst and append.lst files.  These files
contain ONLY the string data.  To get them do:

cut -d\` -f2 prepend-lst > prepend.lst
cut -d\` -f2 append-lst  > append.lst     
NOTE the back quote under the ~ char  If this char is used, modify
the pre_post_cnt.cpp to use a different letter. 
NOTE2, there will usually be more append rules (not sure why, but
this is what I saw). That being so, I will often make sure more
prepends get added, by requiring fewer to 'keep' them in
pre_post_cnt.cpp

step 11.  Build the rules.  Run the program JohnWordMerge.c  
(sorry about the naming, but that is it :)  

JohnWordMerge > rules

Step 12.  Edit john.conf (or john.ini) and add these rules as
a new [Rules.Wordlist] section.  If using the newer version 
of john, that .Wordlist can be any value, and then the 
rules=wordlist can tell john which section to use.

Enhancements:

Some up-casing, lower-casing (all upper was pretty good), Case, 
w-space removal, w-space removeal lower/upper, upcase after 
preprocessing (best in my test), etc all can crack more passwords.
Not nearly as many as the 'base', but by the time you get to using
rules like this, much of the wordlist based cracking has been done
and only the 'hard' crap is left.

Also, devising a method of doing pre AND append on the same
word would
certainly crack some words. I have not tried doing this, but it is 
not beyond the grasp of techniques shown here. 

Good luck, and happy cracking,  Jim.

The pre-built set of rules I have originally put together for this can be
found on the wiki page.  It was too large to send to the email list.

http://openwall.info/wiki/john/usage-examples

// counts the 'pre' and 'post' words found in pass_all
// The pre-post words are what we find pre/post which
// the 'remainder' makes up a word found in english-sorted.all

#define _CRT_SECURE_NO_WARNINGS

#include <stdio.h>
#include <string.h>
#include <stdlib.h>

#define MAX_LEN 15

#define PASS_ALL_CNT 232000
#define DICT_CNT 1250000
#define DICT_FSIZE  14000000
#define PASS_ALL_FSIZE 2300000

#define DICT_FNAME "english-sorted.all"
#define PASS_FNAME "pass_all"

char *WordsPre[MAX_LEN+1][PASS_ALL_CNT];
char *WordsPost[MAX_LEN+1][PASS_ALL_CNT];
int nCntPreWords[MAX_LEN+1][PASS_ALL_CNT];
int nCntPostWords[MAX_LEN+16][PASS_ALL_CNT];
int nWordsPre[MAX_LEN+1];
int nWordsPost[MAX_LEN+1];
char *Passes[PASS_ALL_CNT];
int nPasses;
char *Dict[DICT_CNT];
unsigned nDict;
char *pBuf, *pBufOrig;

bool SearchDict(const char *key)
{
  int lo = 0;
  int num = nDict, hi = nDict-1;
  int mid, half, result;

  while (lo <= hi)
  {
	if ((half = num / 2) != 0)
	{
	  mid = lo + (num & 1 ? half : (half - 1));
	  if (!(result = strcmp(key, Dict[mid])))
		return true;
	  else if (result < 0)
	  {
		hi = mid;
		//num = num & 1 ? half : half-1;
		num = hi-lo+1;
	  }
	  else
	  {
		if (lo==mid)
		  ++lo;
		else
		  lo = mid;
		//num = half;
		num = hi-lo+1;
	  }
	}
	else if (num)
	  return !strcmp(key, Dict[lo]);
	else
	  break;
  }
  return false;
}

void main()
{
  int i, j, k, len;
  char *cp;
  FILE *in;
  // we simply allocate ONE buffer, large enough for all file data,
  // and all possible prepend/append data.
  // the english-sorted.all used is just under 14mb.  The pass_all is
  // just over 2.1mb, and the other sizes are the max count and lenght
  // of all theortical possible prepend and append stuff. Thus, we
  // assure ourselves of a large enough buffer, and thus do not need
  // to check for overflow issues.
  int alloc_sz = (3+4+5+6+7+8+9+10+11+12+13)*2*PASS_ALL_CNT + DICT_FSIZE + PASS_ALL_FSIZE;

  pBuf = new char[alloc_sz];
  pBufOrig = pBuf;
  in = fopen(DICT_FNAME, "r");
  len = fread(pBuf, 1, DICT_FSIZE, in);
  pBuf[len] = 0;
  fclose(in);
  cp = pBuf;
  pBuf += (strlen(pBuf) + 1);
  cp = strtok(cp, "\r\n");
  while (cp)
  {
	Dict[nDict++] = cp;
	cp = strtok(NULL, "\r\n");
  }
  in = fopen(PASS_FNAME, "r");
  len = fread(pBuf, 1, PASS_ALL_FSIZE, in);
  pBuf[len] = 0;
  fclose(in);
  cp = pBuf;
  pBuf += (strlen(pBuf) + 1);
  cp = strtok(cp, "\r\n");
  while (cp)
  {
	Passes[nPasses++] = cp;
	cp = strtok(NULL, "\r\n");
  }

  for (i = 0;  i < nPasses; ++i)
  {
	// nab characters off front of each password, then search 
	// for an english word from the remaining characters.  If
	// a search is found, then store the pre-pend chars into
	// the proper slot (well, SEARCH for the pre-pend prior to
	// adding a new entry.

	char pre[16];  // note max prepend will be 15 bytes.
	if (i && i%1000==0)
	  fprintf(stderr, "%d post\r", i);
	int plen = strlen(Passes[i]);
	for (j = 1; j < plen-1 && j < MAX_LEN+1; ++j)
	{
	  if (SearchDict(&Passes[i][j]))
	  {
		// Found a 'pre'
		memcpy(pre, Passes[i], j);
		pre[j] = 0;
		for (k = 0; k < nWordsPre[j]; ++k)
		{
		  if (!strcmp(WordsPre[j][k], pre))
		  {
			++nCntPreWords[j][k];
			*pre = 0;
			break;
		  }
		}
		if (*pre)
		{
		  strcpy(pBuf, pre);
		  nCntPreWords[j][nWordsPre[j]] = 1;
		  WordsPre[j][nWordsPre[j]++] = pBuf;
		  pBuf += (j+1);
		}
	  }
	}
  }
  for (i = 0; i < nPasses; ++i)
  {
	char post[512];  // note max postpend will be 15 bytes.
	int len = strlen(Passes[i]);
	j = 1;
	if (i && i%1000==0)
	  fprintf(stderr, "%d post\r", i);
	if (len > 16)
	  j = len-15;
	memcpy(post, Passes[i], j);
	for (; j < len-2; ++j)
	{
	  post[j-1] = Passes[i][j-1];
	  post[j] = 0;
	  if (SearchDict(post))
	  {
		// Found a 'post'
		bool bkeep=true;
		for (k = 0; k < nWordsPost[len-j]; ++k)
		{
		  if (!strcmp(WordsPost[len-j][k], &Passes[i][j]))
		  {
			++nCntPostWords[len-j][k];
			bkeep = false;
			break;
		  }
		}
		if (bkeep)
		{
		  strcpy(pBuf, &Passes[i][j]);
		  nCntPostWords[len-j][nWordsPost[len-j]] = 1;
		  WordsPost[len-j][nWordsPost[len-j]++] = pBuf;
		  pBuf += (len-j)+1;
		}
	  }
	}
  }

  for (i = 1; i < 16; ++i)
  {
//	printf ("nWordsPre[%d] = %d\n", i, nWordsPre[i]);
	for (j = 0; j < nWordsPre[i]; ++j)
          if (nCntPreWords[i][j] > 10)
		printf ("%05d `%s`  (Pre)\n", nCntPreWords[i][j], WordsPre[i][j]);
  }
  for (i = 1; i < 16; ++i)
  {
//	printf ("nWordsPost[%d] = %d\n", i, nWordsPost[i]);
	for (j = 0; j < nWordsPost[i]; ++j)
          if (nCntPostWords[i][j] > 10)
		printf ("%05d `%s`  (Post)\n", nCntPostWords[i][j], WordsPost[i][j]);
  }
}

// **************************************************************************
// Word Merger module for 'building' possible passwords.
//
// This has been designed to work directly with johnripper.  It will
// end up building a [List.Rules:Wordlist] sections of john.conf, which
// properly adds rules for the words to be prepended and appended to 
// the dictionary (using --rules command line switch)
//
// Usage:
//   2 files are needed (can work with just one of them).  These files
//   are append.lst and prepend.lst  They are in the format of a 'valid'
//   john dictionary file (1 word per line, lines starting with #!comment 
//   are comment lines).
//
//   There are some other 'mangling' rules that can be 'added' that will 
//   allow modifications of the per-word, the post-word, or the word read
//   from the dictionary file by john, or allow addition of things at any 
//   of the 4 points of insersion (in front of per (or pw), between pre/post
//   and pw, or after the pw (or post).  These 'insersion points' are:
//      1PrePw Pre2Pw PrePw3  1PwPost Pw2Post PwPost3
// **************************************************************************


#include <stdio.h>
#include <stdlib.h>
#include <string.h>

char *sPre[100000], *sPost[100000];
int nPre, nPost;
int nRules;

void LoadPre();
void LoadPost();
void LoadOpts(int argc, char **argv);
void DumpPre(int i);
void DumpPost(int i);

int main(int argc, char **argv)
{
  int i;
  fprintf(stderr, "JohnRipper JoinWordMerge tool, v0.01 JimF.\nFree usage rights granted to all\n\n");
  LoadOpts(argc, argv);
  LoadPre();
  LoadPost();
  if (nPre==0 && nPost==0)
	return 0-!!printf("JoinWordMerge requires at least one of the files append.lst or prepend.lst\nto be in the current directory\n");
  for (i = 0; i < nPre || i < nPost; ++i)
  {
	DumpPost(i);
	DumpPre(i);
  }

  fprintf(stderr, "%d rules created (%d pre and %d post)\n", nRules, nPre, nPost);
  return 0;
}

void LoadPre()
{
  FILE *in = fopen("prepend.lst", "r");
  char Line[16384];
  if (!in)
	return;
  // Only problem is if line is a comment longer than 16k.
  // It will ignore the first part of the comment, and then
  // add the other part(s) of the comment as new words.
  // Oh well, if people want to give bogus data, they should
  // expect bogus output.
  fgets(Line, sizeof(Line), in);
  Line[sizeof(Line)-1] = 0;
  while (!feof(in))
  {
	if (strncmp(Line, "#!comment", 9))
	{
	  strtok(Line, "\r\n");
	  sPre[nPre] = malloc(strlen(Line)+1);
	  strcpy(sPre[nPre++], Line);
	}
	fgets(Line, sizeof(Line), in);
	Line[sizeof(Line)-1] = 0;
  }
}

void LoadPost()
{
  FILE *in = fopen("append.lst", "r");
  char Line[16384];
  if (!in)
	return;
  fgets(Line, sizeof(Line), in);
  Line[sizeof(Line)-1] = 0;
  while (!feof(in))
  {
	if (strncmp(Line, "#!comment", 9))
	{
	  strtok(Line, "\r\n");
	  sPost[nPost] = malloc(strlen(Line)+1);
	  strcpy(sPost[nPost++], Line);
	}
	fgets(Line, sizeof(Line), in);
	Line[sizeof(Line)-1] = 0;
  }
}

void LoadOpts(int argc, char **argv)
{
  // do nothing at this time.
}

void DumpPre(int i)
{
  if (i < nPre)
  {
	 int j;
	 // Note, the 'pre' words are written in reverse order 
	 // (due to how john processes when processing rules, at least at v1.7.3.1-pl.5)
	 for (j = strlen(sPre[i])-1; j >= 0; --j)
	   printf ("^[%c]", sPre[i][j]);
	 printf("\n");
	 ++nRules;
  }
}

void DumpPost(int i)
{
  if (i < nPost)
  {
	int j;
	for (j = 0; j < strlen(sPost[i]); ++j)
	  printf ("$[%c]", sPost[i][j]);
	printf("\n");
	++nRules;
  }
}


-- 
To unsubscribe, e-mail john-users-unsubscribe@...ts.openwall.com and reply
to the automated confirmation request that will be sent to you.

Powered by blists - more mailing lists

Your e-mail address:

Powered by Openwall GNU/*/Linux - Powered by OpenVZ