Day 18 – Dissecting the Go-Ethereum keystore files using Raku tools

Generally the Ethereum (Web3) keystore file is a kind of container for the private key, it has the specific structure mostly related to encryption details. Actually you will not find the private key there as a plain text, but the keystore file has everything to decrypt the private key… with some tricks surely.

When you work with Geth as a backend to access the blockchain, you have to work with accounts and therefore the «address/password» pairs or relevant private keys. Honestly, use of credential pairs is good enough for the most of the tasks, but since you want to boost the performance of your application and use it with authentication-less endpoints — you have to get some Geth-specific features directly in your app. Transaction signing, for example.

Transactions must be signed by a private key obviously, so you can somehow get private key and store it somewhere you want, but more flexible approach — manage existed keystore files.

Also you might be interested in Geth accounts management via keystore file direct access — fortunately Geth reloads keystores on the fly. Of course, I have to warn you about these hacking practices: you can corrupt or delete your account and eventually lose the access to your data on blockchain, but as a research or experiment — it’s ok 😜.

What is the Ethereum (Web3) keystore file

Overview

Roughly speaking, a keystore file is an encrypted representation of the private key. By structure — it is a JSON file with the following content:

{
    "address": "92745e7e310d18e23384511b50fad184cb7cf826",
    "crypto": {
        "cipher": "aes-128-ctr",
        "ciphertext": "6eaf8f9485a714ed30cf38c8ebbb78dc52c0fe4120adb998c0d0b70fe64d6aee",
        "cipherparams": {
            "iv": "fda483b2d6595dde7f2157a6e3611a03"
        },
        "kdf": "scrypt",
        "kdfparams": {
            "dklen": 32,
            "n": 262144,
            "p": 1,
            "r": 8,
            "salt": "5a1dba123aed0b365371b84b83af0be5691b06d4411d750144eabbb59be0efac"
        },
        "mac": "9e5bab612ac8c325c29ead18f619163edf41d831a1ef731a51ce4649c0e7d49e"
    },
    "id": "845c31f0-ac2c-4216-b8eb-76886eaa0cc1",
    "version": 3
}

The main member is the version. It defines the version of the keystore file and hence the encryption approach. I focus on the latest (3rd) generation of Ethereum keystores.

The other JSON members in keystore file are:

• id is the random universally unique identifier generated on keystore file creation. Importing an account with Geth from a keystore file (and exporting it if needed) will by default lead to keystore with different id. If this is not desired, you have to use 3rd-party tools allow to reuse or explicitly define id;
• address relies on private key and might be fetched with Bitcoin::Core::Secp256k1 module;
• crypto has all enctyption details to get private key from the keystore file, encrypted private key is stored in ciphertext.

Actual scope (real world tasks)

I mentioned a few real world tasks we can solve via direct keystore management, but let me show a hot one at least. I maintain a few full nodes for Sepolia test network. On node deployment there are two options for address management: use automatically generated address or import an existing one.

Both cases have their own specifics. The common thing: both need a positive account balance (some available funds) to start interacting with test network.

Tools (aka hacks) to get private key from keystore file

The first case (automatically generated address) is the trivial one if you do not want to manage (track) your balance with the 3rd party wallet: you just need to get some funds from the faucet (I use the faucet by Alchemy) and track funds transfer with Etherscan.

The magic happens when you want to add your automatically generated address to 3rd-party wallets. You have to fetch somehow private key from the keystore file initially generated in Geth and import it, for example, into MetaMask. I wrote a simple Node.js script as a hacker tool for that task. It’s a bit excessive and obviously not flexible approach — you should have Node.js installed, also script needs Keythereum package as a primary dependency. So eventually I had to install all that stuff to my Sepolia node server, added a few symlinks (script looks for keystore file in keystore folder explicitly) and run script from privileged user:

node file2privkey.js
# 632735b66ad875108deef039be855aae7f702653fcc2b2efb1e5666c1306f2fd

Another hacking approach — use Python. It’s a bit more scalable than Node.js: Python is pre-installed in the most of the popular Linux distributions and the script has just a single external dependency:

python3 file2privkey.py
# 632735b66ad875108deef039be855aae7f702653fcc2b2efb1e5666c1306f2fd

Since you got the raw private key, you can easily import it into Metamask and manage/track your account (balance at least) via a friendly UI.

Importing new accounts (private keys) into Geth client

The second case (import existing private key) into Geth client has two options under the hood as well. We can use Node.js again for raw private key to keystore file conversion (dependence to ethereumjs-wallet package). Another approach: built-in Clef account manager — new accounts might be added via the command line:

clef --advanced --stdio-ui --loglevel=6 --keystore=/ethereum-local-network/geth.local/geth-node2/keystore/ importraw /pk

Input argument --keystore points to folder, where Geth stores its keystore files; importraw option triggers raw private key import; /pk is just the path to the sample text file with raw private key. Import via clef is used in Pheix CI/CD at GitLab.

Implementation in Raku

I started with the quick research — how it’s implemented in Geth client written in Go.

Initial state (it looks promising)

Not a lot of work and it looks like Raku has everything in its ecosystem for straight-forward implementation:

• Parse JSON with JSON::Fast module;
• Decrypt V3 key:
  • get derived key with Crypt::LibScrypt module;
  • get keccak hash from derived key and keystore’s ciphertext with Node::Ethereum::Keccak256::Native module;
  • decrypt ciphertext by derived key and keystore’s input vector with AES128: with OpenSSL or Gcrypt modules [ref1, ref2];
• verify decrypted V3 key (raw private key) and get Ethereum address with Bitcoin::Core::Secp256k1 module;
• generate UUID v4 with Crypt::Random module — in case we want to create keystore file from raw private key.

Problems (are on the most important steps)

The main issues are at decryption, unfortunately the most important step. First issue is at Crypt::LibScrypt: custom KDF parameters could not be used because of module limitations. By default only scrypt-hash is implemented, it uses KDF constants hardcoded in Crypt::LibScrypt. Obviously those constants impact hash generation, so to get it working with Geth keystores we have to add binding to libscrypt_scrypt function from Native libscrypt library.

Next issue is at OpenSSL and Gcrypt, both do not include AES128-CTR implementation. I will consider details below. Finally I found the problem at Bitcoin::Core::Secp256k1 — compression was not configurable there, but we need to support SECP256K1_EC_UNCOMPRESSED alongside SECP256K1_EC_COMPRESSED.

Let’s raise a few Pull Requests

Crypt::LibScrypt

On initial step I added a few principal updates to Crypt::LibScrypt module, as it was mentioned above we need a binding to internal hashing function libscrypt_scrypt. Finally the next bindings (and exported wrappers) were added:

• libscrypt_salt_gen — method for salt generation, «must-have» feature for keystore file generation;
• libscrypt_scrypt — main libscrypt‘s hashing method, totally configurable via user-defined KDF (Key Derivation Function) parameters;
• libscrypt_mcf — method to transpose raw hash buffer to modular crypt format (MCF), stringified hash in MCF might be verified by scrypt-verify;

Pull request: https://github.com/jonathanstowe/Crypt-LibScrypt/pull/1/files.

OpenSSL

Updates to OpenSSL are quite straight-forward: I added AES128-CTR binding EVP_aes_128_ctr and implemented new encrypt and decrypt multi methods with mandatory :$aes128ctr argument (to call EVP_aes_128_ctr from).

Pull request: https://github.com/sergot/openssl/pull/103/files.

Yet another binding for GNU Libgcrypt

Initially I tried Gcrypt as the module with AES128-CTR in place. Gcrypt is the great set of bindings for GNU libgcrypt. My expectation was: Gcrypt has a bindings for AES family with all available modes. Quote from module README:

It provides functions for all cryptograhic building blocks: symmetric cipher algorithms (AES, Arcfour, Blowfish, Camellia, CAST5, ChaCha20 DES, GOST28147, Salsa20, SEED, Serpent, Twofish) and modes (ECB, CFB, CBC, OFB, CTR, CCM, GCM, OCB, POLY1305, AESWRAP).

The first finding was: the mode switch is broken in Gcrypt since it was released. Just a typo in Gcrypt/Constants.rakumod: enum with modes is defined as Gcrypt::CipherMode, but in generic Gcrypt::Cipher class, mode is set up from Gcrypt::CipherModes 🤷. So looks like none tested it before, a bit of a dangerous beginning.

I did a quick fix and started tests, but decryption bellow gives wrong $secret buffer:

my buf8 $secret = Crypt::LibGcrypt::Cipher.new(:algorithm(GCRY_CIPHER_AES), :key($derivedkey.subbuf(0, 16)), :mode('CTR'), :iv($iv)).decrypt($ciphertext, :bin(True));

To debug that, I wrote simple C-application, where I compared libgcrypt and libssl decryption — it worked 100% similar to Raku implementation, libssl gave correct secret and libgcrypt gave incorrect one (but the same as I got from Raku script). That finding showed that I missed something in libgcrypt implementation, so I started investigation (googling actually). Next finding was: in libgcrypt embedded tests for AES in CTR mode counter vector ctr was used instead of initial vector iv. So I modified my C-application and 🎉 I finally got correct secret via libgcrypt.

So what updates should be added to Raku Gcrypt module eventually? Actually a few ones:

• fix Gcrypt::CipherModes typo;
• add binding to gcry_cipher_setctr;
• set counter vector with setctr multi method;
• constructor upgrade — set up counter vector if needed.

While working on Gcrypt, my pull requests for Crypt::LibScrypt and OpenSSL were still pending (now are pending as well), so I decided to fork Gcrypt to Crypt::LibGcrypt, because another one stuck PR will frustrate me finally.

Upgrade secp256k1 Raku binding

The quickest step! I’m the maintainer of Bitcoin::Core::Secp256k1, so I just pushed the updates to the source base 😍. The idea behind: add an option to get uncompressed key with serialize_public_key_to_compressed method. Looks a bit controversial (from naming perspective), but it’s just the feature for key serialization without compression.

Manage keystore files with 🤫

So looks like we got all «puzzles» working and finally after an evening spent on Node::Ethereum::KeyStore::V3 module coding, we are ready for 👇

Quick start

This module has everything under the hood to manage your keystore files and raw public keys. You can easily decrypt existed keystore:

use Node::Ethereum::KeyStore::V3;

my $password  = 'node1';
my $decrypted = Node::Ethereum::KeyStore::V3.new(:keystorepath('./data/92745E7e310d18e23384511b50FAd184cB7CF826.keystore')).decrypt_key(:$password);

$decrypted<buf8>.say;
$decrypted<str>.say;

# Buf[uint8]:0x<63 27 35 B6 6A D8 75 10 8D EE F0 39 BE 85 5A AE 7F 70 26 53 FC C2 B2 EF B1 E5 66 6C 13 06 F2 FD>
# 632735b66ad875108deef039be855aae7f702653fcc2b2efb1e5666c1306f2fd

New keystore file could be generated from raw public key via a few calls:

use Node::Ethereum::KeyStore::V3;

my $secret    = '632735b66ad875108deef039be855aae7f702653fcc2b2efb1e5666c1306f2fd';
my $password  = 'node1';
my $ksobject  = Node::Ethereum::KeyStore::V3.new(:keystorepath('./sample-keystore.json'));
my $keycrypto = $ksobject.keystore((:$password, :$secret);

$ksobject.save(:$keystore);

Command line utility

Node::Ethereum::KeyStore::V3 module distribution has ethkeystorev3-cli utility onboard. After the module installation it’s available via command line:

ethkeystorev3-cli
# Usage:
#   ethkeystorev3-cli --keystorepath= --password= [--privatekey=]

Keystore file decryption:

ethkeystorev3-cli --keystorepath=$HOME/sample-keystore.json --password=111
# 632735b66ad875108deef039be855aae7f702653fcc2b2efb1e5666c1306f2fd

Keystore file generation:

ethkeystorev3-cli --keystorepath=$HOME/sample-keystore-gen.json --password=111 --privatekey=632735b66ad875108deef039be855aae7f702653fcc2b2efb1e5666c1306f2fd
# keystore /home/user/sample-keystore-gen.json is successfully saved

Dependencies

Since my pull request to Crypt::LibScrypt is still pending, I decided to maintain my fork of Crypt::LibScrypt with personal auth. For now Node::Ethereum::KeyStore::V3 module depends on Crypt::LibScrypt:ver<0.0.7+>:auth<zef:knarkhov> (available in fez ecosystem).

Afterword

Ethelia service

Ethelia is a secure, authoritative and reliable Ethereum blockchain storage provider for different kinds of lightweight data.

Blockchain technology gives decentralization, security and resistance against data corruption or falsifying out of the box, and Ethelia delivers to end user the ability to store tamper-proof and sensitive data there.

Сonsider a network with functionally different nodes: IoT, programmable logic controllers, smart home systems, micro-services, standalone multi-threading or mobile applications. Every node runs own duty cycle with state changes and events emission.

In general non-critical state changes or usual events should not be logged, but once the node detects some anomaly or exceptional behavior, importance of logging increases exponentially. Such events might be stored on blockchain by Ethelia and the fact of storing will guarantee the data integrity and consistency.

Ethelia is live and runs Raku-driven backend.

Registering via Telegram Bot

I like sign up/in approach from Midjourney — do it via Discord. For Ethelia it’s done the same way, but Telegram is used instead. Registration model is described here.

Basic idea is — customer has to pass the registration interview with the bot and if interview is passed, bot registers new customer on Ethelia’s node. Telegram bot is written in Raku, the module for interviewing is at early beta and still under developing, currently I’m trying to combine static questions with the dynamic GPT4 ones (generated on the fly according the interview flow).

Node::Ethereum::KeyStore::V3 module is used there at the final step for actual account creation (keystore file generation to Geth keystore folder).

Since you are registered, you can post your lightweight data to a different Ethereum networks (our private PoA, Sepolia and mainnet) via Ethelia’s endpoint. The only limitation for this moment — API access token is available only during the auth session in web control panel.

Demo pitch

You are welcome to try it out! Thank you for reading. Happy Xmas 🎄🎅☃️🎁⛄🦌🎄