Writing Node.js Apps from Scratch

Satirical posts regarding the bloat of modern day apps are a hot topic of both memes and dev discussion boards lately. The discussions note a trend to include tons of libraries, tooling, UI theming, and more just to get a simple web app off the ground. Plugins, libraries, and packages seem to spiral out of control as apps pile on more and more features. As an example, the fallout when one NPM package was unpublished from the NPM registry caused a decent amount of websites to die as we saw from the left-pad fiaso. Headlines immediately started to surface questioning the ability of many developers to write basic software.

In my opinion, calling developers inept for including modules in their code is a bit harsh. Most developers are inherently lazy and don’t want to rewrite existing code. Other developers adopt a “build on the shoulders of giants” mentality. But for some you may not know where to start! Maybe you’re just getting started with Node.js so you decide to crank out scaffolding from a code generator like Yeoman. While this is a good course of action, building a module from scratch will help you discover what you truly need to execute code from the command line, import your module in other modules, and understand how all the Node.js pieces fit together.

With that in mind let’s work together to build a lightweight module in Node.js. Follow along or check out the source!

What to expect

The goal of this article to create a CLI friendly password generator that we can use as a standalone application or include in other node apps as an imported module. I’m calling the project TinFoil as a homage to super paranoid crypto nerds that want full control of their passwords. If password generators aren’t your thing that’s fine too. You’ll see from the code that the logic of the password generator isn’t overly complex. I’m more concerned with creating a toolchain to build node modules not a bullet-proof password generator. A quick disclaimer, I don’t claim to be a crypto expert or pretend to play one on the internet. Use common sense before using any of the passwords to lock down sensitive information.

Without further ado, here’s a checklist of the steps we’ll be taking to create our module.

1. Creating the file system scaffolding
2. Initializing the module
3. Installing our dependencies
4. Setting up our linter
5. Setting up our test suite
6. Writing our code
7. Outputting to the Command Line
8. Celebration
   
   

Prerequisites

For the purposes of this post I’m going to be using VS Code, Node 6.7, which I installed with NVM - Node Version Manager, and iTerm in a Mac environment. Nothing in this post is tied to a particular OS, IDE, or Terminal application so feel free to use whatever combination of the three you like.

File system scaffolding

Navigate to where you like to stash your code, create a directory, and change your directory to the newly created one.

mkdir TinFoil && cd TinFoil

Create an index.js file in this directory with touch index.js. This will be our entrypoint into our application for our CLI. For the module code create a directory. Typically I like to stash everything in a /src directory. Creating an encapsulated directory for our code allows more specificity for our test runner and linter which will come in handy later. Specificity can be more granular later as the complexity of our app grows but this is fine for now.

In the src directory create two files touch tinfoil.js && touch tinfoil.spec.js. The tinfoil.js will contain our logic and exported module. The tinfoil.spec.js file will contain our test code. There are some other files that we’ll need to create later but this should be a good starting point.

Initializing the module

Now that we’ve leveled the ground and poured the concrete, we’re going to run npm init which kicks off a script that will generate some Node.js specific bits for our module. Fill out the commands as best you see fit and hit enter when you’re done. The end result of this command is the package.json file. The package.json file is a manifest of sorts for our module. It defines a list of dependencies to install, scripts to run, where to publish the module, and the git repository to file the source code.

Installing dependencies

Before we install the dependencies I wanted to offer a quick sidebar. Skip to the next paragraph if you want to be all business and continue with the tutorial. One of my hobbies is backpacking. As a Californian, I have the majestic Sierra Nevada mountain range (The Sierras) in my backyard. A few times a year I pack up a bag with the bare essentials and head off into the mountains for a few nights. Every time I come out I feel envigorated, having charged up my batteries with some fresh air, reflection, exercise, and natural beauty. After a few trips, I got obsessed with being an ‘ultralight’ backpacker. Every strap was shortened, ounce shaved, and toothbrush handle cut in half. I got my loadout so close to the metal that on one trip I had left all my rain gear in my car to save weight. I ended up getting caught in a really bad storm that dumped a lot of snow (in August no less, thanks global warming) on me and my backpacking group overnight. We could have definitely froze to death and were lucky to get out with mild hypothermia. I probably would not have noticed the extra 2 pounds (1 kilogram) in extra pack weight but I definitely noticed its absence. Long story long, it’s fine to be a minimalist but keep your quality of life in mind when shaving weight. Get your loadout down to the bare essentials THEN start to tack on extra all the while looking at what you bring in with scrutiny. How much overhead does it add? What are the tradeoffs? Can you use it locally and not have to deploy it to production? With that in mind let’s look aim a careful eye at our code and look at the dependencies we’ll bring in.

For our first dependency we will install… nothing. Seriously, we don’t need to install anything at all. The entire code base will run without any core dependencies but let’s remember our quality of life mantra and install one thing. To see how dependencies are read from the package.json file install Chalk, mostly because I like pretty colors. To install Chalk simply run npm install --save chalk. This installs Chalk to the local node_modules directory for our project and adds the entry under "dependencies" in our package.json file.

Next, we’re going to install our "devDependencies". These dependencies are modules that we’ll be using for our local development but wouldn’t get packaged when we deployed to production. We’re going to focus on three types of modules for our local dev tooling for now: testing, linting, and security. For linting we’re going to using the ESLint npm as well as a couple of plugins. Run the following command to install these and add them to the "devDependencies" entry in our package.json file.

npm install --save-dev eslint eslint-config-airbnb eslint-plugin-import eslint-plugin-jasmine eslint-plugin-json

You may see some errors regarding UNMET PEER DEPENDENCIES in the console. These errors are a result of mismatched versions of various linting plugins. If you forsee your project using React or JSX in its future you can go ahead and install the extra plugins. Since our app will not have a front end we can safely ignore these messages.

Next, we’re going to add a couple of other "devDependencies" that will run our test suite and check our code for security vunerabilities. From the command line run: npm install --save-dev nsp chai mocha. Open the package.json file and you should see Chalk, Mocha, Chai, NSP, and our linting plugins are all nested under "devDependencies".

Setting up the linter

Now that we have our linter installed we need to set up our project to be ‘lint-able’. Step one is to create the file .eslintrc in the root of our project. Next, we’ll need to add all the various rules we want to enforce. Obviously, these are just my preference and your mileage will vary but it’s a good starting point. I’ll paste the blob here and we can go over what each item does.

{
"extends": ["airbnb/base", "plugin:jasmine/recommended", "plugin:import/errors", "plugin:import/warnings"],
"plugins": ["jasmine", "json"],
"env": {
"jasmine": true,
"node": true
},
"rules": {
"no-console": 0,
"import/no-extraneous-dependencies": ["error", { "devDependencies": true }]
}
}

The "extends" line dictates which base linting rules we’ll be inheriting from. In this case we’re extending from an AirBnB preset as well as the Jasmine plugin (our testing framework) and plugins relating to importing. The "plugins" section is defining some extra plugins that we won’t be extending from but enforcing outright. You’ll also notice that we can omit the eslint-plugin- prefix when declaring our plugin array. Next up we have the "env" definition. This section defines the environment in which the linter will be running. We make two declarations here "node" and "jasmine". These values also have predefined rule sets that ESLint will use when scanning the code. Lastly, we list our own "rules" to enforce. Normally, the linter will bark at you if you have console.* statements in the code. This is to prevent unwanted messages from appearing in the browser. Since logging from the Node app is invisible to our users we want to override that. We could also use another logging tool like Winston or Pino but for now console.log will work for our needs. The other directive simply indicates that we’re fine with declaring "devDependencies" that are extraneous. The linter thinks it’s a bad smell if we require packages in code that aren’t in our "dependencies" list. Since our "devDependencies" only fire when we run our test suite we’re going to suppress that linting error.

Now that we have our rule configuration file setup we need a way to run it. My recommendation is two-fold. First, we’ll create an NPM script to run our linter on demand. Second, we’ll install an ESLint extension to our VS Code tool so we can also lint in real time. This is especially helpful to developers who might be new to Node development in that they’ll see linting errors in real time as they code and hopefully correct them immediately.

To add a custom linting script, edit the package.json file. You’ll notice a "scripts" section. We’re going to add a new entry here. Inside the object add "lint": "node_modules/eslint/bin/eslint.js ./src/**". This tells the linter to use the eslint bin file from our local node_modules folder. This is a good practice to ensure that everyone on your team is using the same version of the tools you use and also keeps the tooling encapsulated to your project instead of polluting the global npm space. The linter will scan all files recursively in the /src directory for errors. Awesome! It doesn’t do much now but that will change once we start writing code.

The second step is to install the ESLint plugin in VS Code. In VS Code this step is as simple as clicking on the Extensions tab on the left pane, searching for ESLint, selecting install, and restarting VS Code. Sublime Text, Atom, and the various JetBrains IDEs also have similar ways of installing ESLint plugins and I encourage you to seek them out and install them.

Setting up our test suite

In the spirit of TDD (Test Driven Development) we’re going to install our tool suite and start to write some tests before we write any code. First, we’ll setup our test script to run. Edit the "package.json" file once again and we’re going to add two new scripts. The existing test command will read something like, “echo “Error: no test specified” && exit 1”. This was the default value created by the npm init command. We’re going to change that command to the following:

node_modules/.bin/mocha --reporter spec ./src/\*\*.spec.js

We’re going to use the mocha bin file that’s local to our project. The --reporter flag is configuring Mocha to output the test results in the BDD friendly spec format. We’re also going to run assertions against any file in the ./src directory with .spec in the filename. The reason that npm init put in default values for the "test" command is that NPM has a few default scripts that can always run in a project as well as hooks that we can use to do other neat stuff. Taking advantage of these hooks we’re going to add a "posttest" script that’s going to lint our code as well as scan for any vunerabilities.

npm run lint && node_modules/.bin/nsp check

The script will run the lint script that we defined earlier then also run the NSP check against our codebase. At this point you should be able to run npm t from the command line. The output won’t be much but you should see that it scanned the directories for tests, tried to lint nothing, and scanned for vunerabilities. Neato!

Writing the code

If you stuck it out this far then you might be thinking that we done a whole lot of scaffolding and zero code writing. You would be correct sir or madam. However, I wanted to maintain scruntiny of our modules and plugins that we brought in and do some due diligence to keep everything lean and clean. I was also extra verbose. Most of those commands shouldn’t take long to run and at some point you can create a boilerplate scaffold that you can use in the future when bootstrapping new projects. With all of that out of the way let’s write some code.

Initially we want to define our TinFoil module so our first test is going to verify that we have a TinFoil module and that it has a password generator function that we can call. In our index.spec.js file we’re going to require our BDD based Chai expect syntax and also our TinFoil module. Then we’re going to make an expectation that the TinFoil module has a publicly accessible function called generateRandomPassword.

const expect = require('chai').expect;
const TinFoil = require('./tinfoil').TinFoil;

describe('Password Generator', () => {
it('should have a password generator function', () => {
expect(Object.prototype.hasOwnProperty.call(TinFoil, 'generateRandomPassword')).to.equal(true);
});
});

We can now save and run npm t from the command line. The test will fail of course so let’s edit src/tinfoil.js and get our tests passing! Once we’ve opened up the file we need to do a couple things. First we need to export Tinfoil so we can attach it to the exports namespace under its own TinFoil namespace and also declare a publicly accessible method that we can test.

// Here we attach TinFoil to exports and wrap it in a closure that self executes
exports.TinFoil = (() => {

// Any consts, lets, and functions here will be private
function generateRandomPassword(length = 8, requireSpecial = true) {
return true;
}

// Anything after the return will be public
return {
generateRandomPassword: (length, requireSpecial) =>
generateRandomPassword(length, requireSpecial),
}
})();

I’ve added some notes about how we’re exporting TinFoil and defining public and private functions and variables. Initially we’ll have quite a few public declarations as that’s the only way we can expose the functions and variables to our test suite. We can lock those down later if necessary. Run npm t and now we can see that the test will pass. In the method signature generateRandomPassword I’ve added a couple of default parameters. The length param sets a default length of 8 characters and we also require special characters in the password by default. We can adjust those parameters to meet certain requirements if need be. For example, we could pass in false as the second param if a password only accepted letters and numbers. Bonus: Try to write some tests to require a certain minimum password length or only allow a certain amount of special characters in the password.

Before diving in to the technical details too deeply let’s go over the overall strategy for generating passwords. The basic premise is to build an array of single character strings, randomly pull different indices from the array, then re-assemble the password and return it to the user. The first step will be to setup our source strings that we’ll be converting into arrays. We’ll be storing these as private variables to TinFoil. While they’re well known now if we were to use dictionaries or word lists in the future we wouldn’t want to expose them to the outside world.

// Private
const alphabet = 'abcdefghijklmnopqrstuvwxyz';
const alphabetCaps = alphabet.toUpperCase();
const numbers = '0123456789';
const symbols = '!@#$%^&\*(){}?[]<>,.';

Next we need to convert these to arrays and build up our selection array. Remember that requiring special characters is an optional parameter to our password generation function so we’ll pass in a boolean to this selection array builder to choose whether or not to include special characters into our array. In theory we could move the private variables we defined above into this function for more testability. For testability we’re going to inject our private variables into the function so we can make testing easier in the future.

function buildSelectionArray(alpha, caps, nums, syms) {
let selectionArray = [];

// Split our strings into single character arrays
const alphabetArray = alpha.split('');
const capsArray = caps.split('');
const numArray = nums.split('');

// Concat chains all the arrays together
selectionArray = selectionArray.concat(alphabetArray, capsArray, numArray);

// If we have symbols passed in go ahead and split and add to the selectionArray
if (syms) {
const symbolArray = syms.split('');
selectionArray = selectionArray.concat(symbolArray);
}

return selectionArray;
}

Now that we have our selection array we can write a function that returns a random number between 0 and the index of the end of our selection array.

function getRandomArrayIndex(min = 0, max) {
return Math.floor(Math.random() \* ((max - (min + 1)) + min));
}

The final step is to wire it all up in our generateRandomPassword function. We’ll execute based on the steps we outlined in the beginning:

• Build up our selection array
• Get our max index from our selection array
• Iterate over the selection array plucking a random length until our desire length is reached
• Assemble and return the password
  
  

function generateRandomPassword(length = 8, requireSpecial = true) {
// Build selection array
let selectionArray = [];
if (requireSpecial) {
selectionArray =
buildSelectionArray(alphabet, alphabetCaps, numbers, symbols);
} else {
selectionArray =
buildSelectionArray(alphabet, alphabetCaps, numbers);
}

// Since arrays start at the 0th item remove one from the total length
// so we don't encounter array out of bounds errors
const maxIndex = selectionArray.length - 1;
const passwordArray = [];

// Select an item from our selection array at a random index and push it
// into our passwordArray
for (let i = 0; i < length; i += 1) {
const selectionIndex = getRandomArrayIndex(0, maxIndex);
passwordArray.push(selectionArray[selectionIndex]);
}

// Convert our array into a string
return passwordArray.join('');
}

For the sake of simplicity I’ve omitted the test code for now. You can check out the TinFoil Github repo if you’re curious about how to write more tests for this module. At this point we have a module that can be pulled into another project and used to generate passwords. Maybe you need to generate temporary passwords for users? We’re still missing the command line component so let’s get that wired up.

Outputting to the Command Line

In order to get some output to our command line we’re going to add another NPM script that will execute our code from our entrypoint in the index.js file. To make the output slightly dynamic we’re going to take advantage of Node Environment Variables. These variables get attached to the process.env in node so that we can access them later when our node process is running. TinFoil will be taking advantage of two env vars PASSWORD_LENGTH and PASSWORD_SPECIAL. Our password generator will look for these values and use them or fall back to a default otherwise. Our index.js file will require Chalk and Tinfoil then output the result to the console.

const chalk = require('chalk');
const TinFoil = require('./src/tinfoil').TinFoil;

// Booleans don't really exist as Env Vars so we cheat a little here
console.log(chalk.cyan('Password:'), chalk.yellow(TinFoil.generateRandomPassword(process.env.PASSWORD_LENGTH || 12, process.env.PASSWORD_SPECIAL || 'true')));

In our package.json file we’re going to add a new item under "scripts" to run our password generator - "password": "node index.js". Our scripts will end up containing the following.

"scripts": {
"password": "node index.js",
"test": "node_modules/.bin/mocha --reporter spec ./src/**.spec.js",
"lint": "node_modules/eslint/bin/eslint.js ./src/**",
"posttest": "npm run lint && node_modules/.bin/nsp check"
},

To test the basic functionality simply run npm run password from the project directory. You should see a shiny new password output to the console! If you want to change the length or remove the special character requirement you can set the Node Env Vars we defined earlier.

PASSWORD_LENGTH=16 PASSWORD_SPECIAL=false npm run password

A 16 digit password should be output in the console!

Celebration

In closing, hopefully you learned a little bit about how to create custom node applications from the ground up. We built a little nugget of usefulness that can be included in another Node.js application or ran as a standalone application. We scruntinized every dependency we brought in while maintaining a solid tool chain and improving our quality of life. Once again feel free to download the source from Github and fork, comment, create issues, etc. Thanks and happy coding!