Date: Thu, 18 Feb 2021 20:16:18 +0100 From: Solar Designer <solar@...nwall.com> To: announce@...ts.openwall.com, passwdqc-users@...ts.openwall.com Subject: [openwall-announce] passwdqc 2.0.0 Hi, This is to announce passwdqc 2.0.0, a new version of our password/passphrase strength checking and enforcement tool set: https://www.openwall.com/passwdqc/ There's also a corresponding update of passwdqc for Windows: https://www.openwall.com/passwdqc/windows/ The upstream repository for passwdqc has recently been moved to Git and onto GitHub, and all of the changes since the previously announced release (1.4.0 in December 2019) have been made in Git: https://github.com/openwall/passwdqc The major change, corresponding to the major version number update, is addition of support for external wordlist, denylist, and binary filter files. With these, passwdqc can be configured to deny passwords and passphrases that are based on lines of a tiny external text file (the "wordlist" option), directly appear in a tiny external text file (the "denylist" option), or/and directly appear in a maybe huge binary filter file (the "filter" option). While usage of larger external text files is inefficient, the binary filters are very efficient. Under the hood passwdqc binary filters are improved cuckoo filters, a modern probabilistic data structure. They never produce false negatives (that is, all passwords that are meant to be denied will always be). They occasionally produce false positives (that is, a previously unseen password might be unintentionally denied), but with our settings and algorithm this probability is negligible (fewer than 1 in a billion). Looking up a password against a filter requires at most two tiny random disk reads, which is very quick and lightweight for the server. The binary filters can be created and otherwise managed with the newly added pwqfilter(1) program. It can create a binary filter from a list of plaintexts or from MD4 or NTLM hashes. The latter are supported in a way that enables importing of HIBP (Pwned Passwords) database revisions into passwdqc binary filters. A lot of effort went into optimizing pwqfilter for speed, compactness of the resulting filters, and lower false positive rate. For example, creating a 98.00% load factor cuckoo filter from the pwned-passwords-ntlm-ordered-by-hash-v7.txt file, which is 21 GiB (22 GB) and contains 613+ million lines, takes around 8 minutes on i7-4770K with 32 GiB RAM. The resulting binary filter file is 2.3 GiB (2.5 GB), and (with the right optimization option to pwqfilter) it has a false positive rate of around 1 in 1.15 billion. With a lower target load factor, the file can be created much quicker and have an even lower false positive rate, but it would then be larger. As you can see from the website, we also use this as an opportunity to sell pre-generated binary filter files. Given that the pwqfilter program is free, users have a choice between generating filters on their own and purchasing ours - or maybe both, for different input data sets. (If the use of binary filters is desired in a given environment at all.) pwqfilter works on arbitrary plain text strings or hex-encoded hashes, and it can also be reused in lieu of grep(1) for many purposes, even unrelated to passphrases and security. Considering such possible reuse and similarity to grep, pwqfilter includes several command-line options that match grep's, doesn't use command-line option names that would conflict with grep's, and its exit codes are compatible with grep's. For applications where false positives would be highly undesirable, much lower false positive rates like 1 in a billion billion can be achieved by using much lower target load factors like those below 44%. (A classic cuckoo filter wouldn't achieve that much improvement in false positive rate at lower load, but ours is adaptive. Might be worth writing an Openwall article on.) To separate the major change above from other changes, a few intermediate releases have also been made, without them having been announced separately. I'll summarize other significant changes here: Changes since 1.5.0 to 2.0.0: Added support for external wordlist, denylist, and binary filter. (Described above in much detail, won't repeat here.) Merged changes needed for building with Visual Studio on Windows. This includes a refactoring of the random passphrase generator code to make it shared between platforms. In other words, the main passwdqc project, which is free software, now includes changes (freely licensed, indeed) that had been made to its source files in the non-free Windows product (without hurting portability). This simplifies further maintenance for us and might be handy for someone else wanting to reuse passwdqc source code in a Visual Studio or another native Windows project (maybe even competing with ours). Changes since 1.4.1 to 1.5.0: Updated the included wordlist to avoid some inappropriate words in randomly generated passphrases while not removing any words from the "word-based" check, and also to have plenty of extra words for subsequent removal of more words that might be considered inappropriate from the initial 4096 that are used for randomly generated passphrases. Most of the added words came from EFF Diceware, BIP-0039, and our own processing of Project Gutenberg Australia books. (This change has already resulted in constructive discussion and a likely upcoming contribution on the very first GitHub issue. Arguably, without a prior change like this our project was not ready for GitHub: we'd be getting pull requests removing words from the list of exactly 4096, which we'd have to be closing without action, causing frustration for everyone.) Changes since 1.4.0 to 1.4.1: Set default for "max" to 72 (was 40). The previous setting was based on a reading of RFC 1939, which in practice did not matter. The new one is based on bcrypt's truncation at 72, which actually still matters. Documented "similar" in pwqcheck(1) help message and manual page. This is a setting that was supported before and documented for other passwdqc components before, but was apparently erroneously omitted from here. Alexander
Powered by blists - more mailing lists
Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.