Recently I’ve been making a few requests to different services to download data, process it, and store it elsewhere. I’ve written similar logic for different transformations, that I identified some repeating logic and separated it into it’s own library.

@codejamboree/web-requests-queue

I touched on this a bit yesterday with my adventures in publishing TypeScript for different module loaders and environments. The library as a few features, but the main one is to prevent overloading a web server with too many requests. I can queue up many requests and continue performing other operations, and the library will space out each individual request to be made at an appropriate time. If the server responds that it is too busy, I can pause all requests and continue where I left off later.

In addition, the library has a few helper utilities that can download a file, read a response and return a string, or return a parsed value from a JSON response.

Here are some examples of its use:

// Setting request rate
webRequest.configure({
  requestPerPeriod: 10,
  secondsPerPeriod: 60
});

// simple get request
webRequest.queue(url).then(req => req.end());

// making many requests
Promise.all([1, 2, 3].map(
  id => webRequest.queue(url + '?id=' id)
).then(() => console.write('done'));

// pause queued requests
webRequest.configure({ pause: true });

// resume queued requests
webRequest.configure({ pause: false });

// cancel queued requests
webRequests.cancel();

// getting statistics
const stats = webRequest.info();
// returns an object similar to
const stats = {
  requested: 2,
  queued: 32, 
  firstAt: new Date("2024-08-30T23:51:22.818Z"),
  paused: false
}

// downloading files
webRequest.toFile(filePath, url);

// getting a web page as a string
const html = await webRequest.asString(url);
console.log(html);

// parsing JSON
const user = await webRequest.parseJson(url);

// parsing JSON with revivor
const revivor = (key, value) => key === 'date' ? new Date(value) : value;
const user = await webRequest.parseJsonWithRevivor(revivor, url);

// downloading to stream
webRequest.toStream(process.stdout, url);

For the most part, I’m already simplifying code in other projects by replacing http quests with the webRequest queue object.

For the most part, I’m trying to cut down on any external dependencies. I tried doing this with testing as well, and ended up making my own bare-bones test platform. It originally started out at just calling a function to verify everything worked, but over time I also added the ability to scan a folder for test files, detect ignored and focused files/test suites, add an expectation module, created a function mocker, and mocked Date, https.request, process.hrtime, performance.now (console.time*), and stdout (underlying console logging).

The testing library may be deserving enough for its own project. There are things that are probably done in a horrible way – especially the part where I stringify variables for “equal” comparisons. It was a quick and dirty solution to simply solve a problem without much need for thought when it comes to complex objects. I even added a JSON replacer so that instantiated objects will serialize with constructor names, and functions will write out a string value along with their name if they have one. It’s not perfect, and I can’t imagine the performance would be adequate with a large number of tests – for the the time being it works.

The code is available for all to see on github.

• Source Code: https://github.com/CodeJamboree/web-request-queue
• NPM Package: https://www.npmjs.com/package/@codejamboree/web-request-queue
• npm i -s @codejamboree/web-request-queue