8.4.2

Also they'll work in the node API too now

CHANGELOG.md

@@ -1,40 +1,48 @@
 # Changelog
-All notable changes to this project will be documented in this file.
+All notable changes to CyberChef will be documented in this file.
+
+### [8.4.0] - 2018-08-23
+- 'To Base85' and 'From Base85' operations added [@PenguinGeorge] | [#340]
+
+### [8.3.0] - 2018-08-21
+- 'To MessagePack' and 'From MessagePack' operations added [@artemisbot] | [#338]
 
 ### [8.2.0] - 2018-08-21
-- Added information links to most operations, accessible in the description popover @PenguinGeorge #298
+- Information links added to most operations, accessible in the description popover [@PenguinGeorge] | [#298]
 
 ### [8.1.0] - 2018-08-19
-- 'Dechunk HTTP response' operation added @sevzero #311
+- 'Dechunk HTTP response' operation added [@sevzero] | [#311]
 
 ## [8.0.0] - 2018-08-05
-- Codebase rewritten using [ES modules](https://hacks.mozilla.org/2018/03/es-modules-a-cartoon-deep-dive/) and [classes](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Classes) #284
-- Operation architecture restructured to make adding new operations a lot simpler #284
-- A script has been added to aid in the creation of new operations by running `npm run newop` @n1474335 #284
-- 'Magic' operation added - [automated detection of encoded data](https://github.com/gchq/CyberChef/wiki/Automatic-detection-of-encoded-data-using-CyberChef-Magic) @n1474335 #239
-- UI updated to use [Bootstrap Material Design](https://fezvrasta.github.io/bootstrap-material-design/) @n1474335 #248
-- `JSON`, `File` and `List<File>` Dish types added @n1474335 #284
-- `OperationError` type added for better handling of errors thrown by operations @d98762625 #296
-- A `present()` method has been added, allowing operations to pass machine-friendly data to subsequent operations whilst presenting human-friendly data to the user @n1474335 #284
-- Set operations added @d98762625 #281
-- 'To Table' operation added @JustAnotherMark #294
-- 'Haversine distance' operation added @Dachande663 #325
-- Started keeping a changelog @n1474335
+- Codebase rewritten using [ES modules](https://hacks.mozilla.org/2018/03/es-modules-a-cartoon-deep-dive/) and [classes](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Classes) [@n1474335] [@d98762625] [@artemisbot] [@picapi] | [#284]
+- Operation architecture restructured to make adding new operations a lot simpler [@n1474335] | [#284]
+- A script has been added to aid in the creation of new operations by running `npm run newop` [@n1474335] | [#284]
+- 'Magic' operation added - [automated detection of encoded data](https://github.com/gchq/CyberChef/wiki/Automatic-detection-of-encoded-data-using-CyberChef-Magic) [@n1474335] | [#239]
+- UI updated to use [Bootstrap Material Design](https://fezvrasta.github.io/bootstrap-material-design/) [@n1474335] | [#248]
+- `JSON`, `File` and `List<File>` Dish types added [@n1474335] | [#284]
+- `OperationError` type added for better handling of errors thrown by operations [@d98762625] | [#296]
+- A `present()` method has been added, allowing operations to pass machine-friendly data to subsequent operations whilst presenting human-friendly data to the user [@n1474335] | [#284]
+- Set operations added [@d98762625] | [#281]
+- 'To Table' operation added [@JustAnotherMark] | [#294]
+- 'Haversine distance' operation added [@Dachande663] | [#325]
+- Started keeping a changelog [@n1474335]
 
 ## [7.0.0] - 2017-12-28
-- Added support for loading, processing and downloading files up to 500MB @n1474335 #224
+- Added support for loading, processing and downloading files up to 500MB [@n1474335] | [#224]
 
 ## [6.0.0] - 2017-09-19
-- Added threading support, moving all recipe processing into a [Web Worker](https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Using_web_workers) to increase performance and allow long-running operations to be cancelled @n1474335 #173
-- Created modules so that operations relying on large libraries can be downloaded separately as required, reducing the initial loading time for the app @n1474335 #173
+- Threading support added. All recipe processing moved into a [Web Worker](https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Using_web_workers) to increase performance and allowing long-running operations to be cancelled [@n1474335] | [#173]
+- Module system created so that operations relying on large libraries can be downloaded separately as required, reducing the initial loading time for the app [@n1474335] | [#173]
 
 ## [5.0.0] - 2017-03-30
--  Configured Webpack build process, Babel transpilation and ES6 imports and exports @n1474335 #95
+-  Webpack build process configured with Babel transpilation and ES6 imports and exports [@n1474335] | [#95]
 
 ## [4.0.0] - 2016-11-28
--  Initial open source commit @n1474335
+-  Initial open source commit [@n1474335] | [b1d73a72](https://github.com/gchq/CyberChef/commit/b1d73a725dc7ab9fb7eb789296efd2b7e4b08306)
 
 
+[8.4.0]: https://github.com/gchq/CyberChef/releases/tag/v8.4.0
+[8.3.0]: https://github.com/gchq/CyberChef/releases/tag/v8.3.0
 [8.2.0]: https://github.com/gchq/CyberChef/releases/tag/v8.2.0
 [8.1.0]: https://github.com/gchq/CyberChef/releases/tag/v8.1.0
 [8.0.0]: https://github.com/gchq/CyberChef/releases/tag/v8.0.0
@@ -42,3 +50,27 @@ All notable changes to this project will be documented in this file.
 [6.0.0]: https://github.com/gchq/CyberChef/releases/tag/v6.0.0
 [5.0.0]: https://github.com/gchq/CyberChef/releases/tag/v5.0.0
 [4.0.0]: https://github.com/gchq/CyberChef/commit/b1d73a725dc7ab9fb7eb789296efd2b7e4b08306
+
+[@n1474335]: https://github.com/n1474335
+[@d98762625]: https://github.com/d98762625
+[@artemisbot]: https://github.com/artemisbot
+[@picapi]: https://github.com/picapi
+[@Dachande663]: https://github.com/Dachande663
+[@JustAnotherMark]: https://github.com/JustAnotherMark
+[@sevzero]: https://github.com/sevzero
+[@PenguinGeorge]: https://github.com/PenguinGeorge
+
+[#95]: https://github.com/gchq/CyberChef/pull/299
+[#173]: https://github.com/gchq/CyberChef/pull/173
+[#224]: https://github.com/gchq/CyberChef/pull/224
+[#239]: https://github.com/gchq/CyberChef/pull/239
+[#248]: https://github.com/gchq/CyberChef/pull/248
+[#281]: https://github.com/gchq/CyberChef/pull/281
+[#284]: https://github.com/gchq/CyberChef/pull/284
+[#294]: https://github.com/gchq/CyberChef/pull/294
+[#296]: https://github.com/gchq/CyberChef/pull/296
+[#298]: https://github.com/gchq/CyberChef/pull/298
+[#311]: https://github.com/gchq/CyberChef/pull/311
+[#325]: https://github.com/gchq/CyberChef/pull/325
+[#338]: https://github.com/gchq/CyberChef/pull/338
+[#340]: https://github.com/gchq/CyberChef/pull/340

package-lock.json

@@ -1,6 +1,6 @@
 {
   "name": "cyberchef",
-  "version": "8.2.0",
+  "version": "8.4.2",
   "lockfileVersion": 1,
   "requires": true,
   "dependencies": {
@@ -4197,14 +4197,12 @@
         "balanced-match": {
           "version": "1.0.0",
           "bundled": true,
-          "dev": true,
-          "optional": true
+          "dev": true
         },
         "brace-expansion": {
           "version": "1.1.11",
           "bundled": true,
           "dev": true,
-          "optional": true,
           "requires": {
             "balanced-match": "^1.0.0",
             "concat-map": "0.0.1"
@@ -4219,20 +4217,17 @@
         "code-point-at": {
           "version": "1.1.0",
           "bundled": true,
-          "dev": true,
-          "optional": true
+          "dev": true
         },
         "concat-map": {
           "version": "0.0.1",
           "bundled": true,
-          "dev": true,
-          "optional": true
+          "dev": true
         },
         "console-control-strings": {
           "version": "1.1.0",
           "bundled": true,
-          "dev": true,
-          "optional": true
+          "dev": true
         },
         "core-util-is": {
           "version": "1.0.2",
@@ -4349,8 +4344,7 @@
         "inherits": {
           "version": "2.0.3",
           "bundled": true,
-          "dev": true,
-          "optional": true
+          "dev": true
         },
         "ini": {
           "version": "1.3.5",
@@ -4362,7 +4356,6 @@
           "version": "1.0.0",
           "bundled": true,
           "dev": true,
-          "optional": true,
           "requires": {
             "number-is-nan": "^1.0.0"
           }
@@ -4377,7 +4370,6 @@
           "version": "3.0.4",
           "bundled": true,
           "dev": true,
-          "optional": true,
           "requires": {
             "brace-expansion": "^1.1.7"
           }
@@ -4489,8 +4481,7 @@
         "number-is-nan": {
           "version": "1.0.1",
           "bundled": true,
-          "dev": true,
-          "optional": true
+          "dev": true
         },
         "object-assign": {
           "version": "4.1.1",
@@ -4623,7 +4614,6 @@
           "version": "1.0.2",
           "bundled": true,
           "dev": true,
-          "optional": true,
           "requires": {
             "code-point-at": "^1.0.0",
             "is-fullwidth-code-point": "^1.0.0",
@@ -7566,6 +7556,11 @@
       "integrity": "sha1-LRDAa9/TEuqXd2laTShDlFa3WUI=",
       "dev": true
     },
+    "notepack.io": {
+      "version": "2.1.3",
+      "resolved": "https://registry.npmjs.org/notepack.io/-/notepack.io-2.1.3.tgz",
+      "integrity": "sha512-AgSt+cP5XMooho1Ppn8NB3FFaVWefV+qZoZncYTUSch2GAEwlYLcIIbT5YVkMlFeNHnfwOvc4HDlbvrB5BRxXA=="
+    },
     "npm-run-path": {
       "version": "2.0.2",
       "resolved": "https://registry.npmjs.org/npm-run-path/-/npm-run-path-2.0.2.tgz",

package.json

@@ -1,6 +1,6 @@
 {
   "name": "cyberchef",
-  "version": "8.2.0",
+  "version": "8.4.2",
   "description": "The Cyber Swiss Army Knife for encryption, encoding, compression and data analysis.",
   "author": "n1474335 <n1474335@gmail.com>",
   "homepage": "https://gchq.github.io/CyberChef",
@@ -108,6 +108,7 @@
     "moment-timezone": "^0.5.21",
     "node-forge": "^0.7.5",
     "node-md6": "^0.1.0",
+    "notepack.io": "^2.1.3",
     "nwmatcher": "^1.4.4",
     "otp": "^0.1.3",
     "popper.js": "^1.14.4",

src/core/config/Categories.json

@@ -25,6 +25,8 @@
             "From Base32",
             "To Base58",
             "From Base58",
+            "To Base85",
+            "From Base85",
             "To Base",
             "From Base",
             "To BCD",
@@ -47,7 +49,9 @@
             "Change IP format",
             "Encode text",
             "Decode text",
-            "Swap endianness"
+            "Swap endianness",
+            "To MessagePack",
+            "From MessagePack"
         ]
     },
     {
@@ -313,7 +317,9 @@
             "To Camel case",
             "To Kebab case",
             "BSON serialise",
-            "BSON deserialise"
+            "BSON deserialise",
+            "To MessagePack",
+            "From MessagePack"
         ]
     },
     {

src/core/lib/Base85.mjs

@@ -0,0 +1,45 @@
+/**
+ * Base85 resources.
+ *
+ * @author PenguinGeorge [george@penguingeorge.com]
+ * @copyright Crown Copyright 2018
+ * @license Apache-2.0
+ */
+
+/**
+ * Base85 alphabet options.
+ */
+export const ALPHABET_OPTIONS = [
+    {
+        name: "Standard",
+        value: "!-u",
+    },
+    {
+        name: "Z85 (ZeroMQ)",
+        value: "0-9a-zA-Z.\\-:+=^!/*?&<>()[]{}@%$#",
+    },
+    {
+        name: "IPv6",
+        value: "0-9A-Za-z!#$%&()*+\\-;<=>?@^_`{|~}",
+    }
+];
+
+
+/**
+ * Returns the name of the alphabet, when given the alphabet.
+ *
+ * @param {string} alphabet
+ * @returns {string}
+ */
+export function alphabetName(alphabet) {
+    alphabet = alphabet.replace("'", "&apos;");
+    alphabet = alphabet.replace("\"", "&quot;");
+    alphabet = alphabet.replace("\\", "&bsol;");
+    let name;
+
+    ALPHABET_OPTIONS.forEach(function(a) {
+        if (escape(alphabet) === escape(a.value)) name = a.name;
+    });
+
+    return name;
+}

src/core/operations/DropBytes.mjs

@@ -5,7 +5,6 @@
  */
 
 import Operation from "../Operation";
-import OperationError from "../errors/OperationError";
 
 /**
  * Drop bytes operation
@@ -20,7 +19,7 @@ class DropBytes extends Operation {
 
         this.name = "Drop bytes";
         this.module = "Default";
-        this.description = "Cuts a slice of the specified number of bytes out of the data.";
+        this.description = "Cuts a slice of the specified number of bytes out of the data. Negative values are allowed.";
         this.inputType = "ArrayBuffer";
         this.outputType = "ArrayBuffer";
         this.args = [
@@ -50,14 +49,25 @@ class DropBytes extends Operation {
      * @throws {OperationError} if invalid input
      */
     run(input, args) {
-        const start = args[0],
-            length = args[1],
-            applyToEachLine = args[2];
-
-        if (start < 0 || length < 0)
-            throw new OperationError("Error: Invalid value");
+        let start = args[0],
+            length = args[1];
+        const applyToEachLine = args[2];
 
         if (!applyToEachLine) {
+            if (start < 0) { // Take from the end
+                start = input.byteLength + start;
+            }
+
+            if (length < 0) { // Flip start point
+                start = start + length;
+                if (start < 0) {
+                    start = input.byteLength + start;
+                    length = start - length;
+                } else {
+                    length = -length;
+                }
+            }
+
             const left = input.slice(0, start),
                 right = input.slice(start + length, input.byteLength);
             const result = new Uint8Array(left.byteLength + right.byteLength);
@@ -82,10 +92,28 @@ class DropBytes extends Operation {
         }
         lines.push(line);
 
-        let output = [];
+        let output = [],
+            s = start,
+            l = length;
         for (i = 0; i < lines.length; i++) {
-            output = output.concat(lines[i].slice(0, start).concat(lines[i].slice(start+length, lines[i].length)));
+            if (s < 0) { // Take from the end
+                s = lines[i].length + s;
+            }
+
+            if (l < 0) { // Flip start point
+                s = s + l;
+                if (s < 0) {
+                    s = lines[i].length + s;
+                    l = s - l;
+                } else {
+                    l = -l;
+                }
+            }
+
+            output = output.concat(lines[i].slice(0, s).concat(lines[i].slice(s+l, lines[i].length)));
             output.push(0x0a);
+            s = start;
+            l = length;
         }
         return new Uint8Array(output.slice(0, output.length-1)).buffer;
     }

src/core/operations/Filter.mjs

@@ -8,6 +8,7 @@ import Operation from "../Operation";
 import Utils from "../Utils";
 import {INPUT_DELIM_OPTIONS} from "../lib/Delim";
 import OperationError from "../errors/OperationError";
+import XRegExp from "xregexp";
 
 /**
  * Filter operation
@@ -21,7 +22,7 @@ class Filter extends Operation {
         super();
 
         this.name = "Filter";
-        this.module = "Default";
+        this.module = "Regex";
         this.description = "Splits up the input using the specified delimiter and then filters each branch based on a regular expression.";
         this.inputType = "string";
         this.outputType = "string";
@@ -55,7 +56,7 @@ class Filter extends Operation {
         let regex;
 
         try {
-            regex = new RegExp(args[1]);
+            regex = new XRegExp(args[1]);
         } catch (err) {
             throw new OperationError(`Invalid regex. Details: ${err.message}`);
         }

src/core/operations/FindReplace.mjs

@@ -6,6 +6,7 @@
 
 import Operation from "../Operation";
 import Utils from "../Utils";
+import XRegExp from "xregexp";
 
 /**
  * Find / Replace operation
@@ -50,6 +51,11 @@ class FindReplace extends Operation {
                 "name": "Multiline matching",
                 "type": "boolean",
                 "value": true
+            },
+            {
+                "name": "Dot matches all",
+                "type": "boolean",
+                "value": false
             }
         ];
     }
@@ -60,16 +66,17 @@ class FindReplace extends Operation {
      * @returns {string}
      */
     run(input, args) {
-        const [{option: type}, replace, g, i, m] = args;
+        const [{option: type}, replace, g, i, m, s] = args;
         let find = args[0].string,
             modifiers = "";
 
         if (g) modifiers += "g";
         if (i) modifiers += "i";
         if (m) modifiers += "m";
+        if (s) modifiers += "s";
 
         if (type === "Regex") {
-            find = new RegExp(find, modifiers);
+            find = new XRegExp(find, modifiers);
             return input.replace(find, replace);
         }
 
@@ -77,7 +84,7 @@ class FindReplace extends Operation {
             find = Utils.parseEscapedChars(find);
         }
 
-        find = new RegExp(Utils.escapeRegex(find), modifiers);
+        find = new XRegExp(Utils.escapeRegex(find), modifiers);
 
         return input.replace(find, replace);
     }

src/core/operations/FromBase85.mjs

@@ -0,0 +1,105 @@
+/**
+ * @author PenguinGeorge [george@penguingeorge.com]
+ * @copyright Crown Copyright 2018
+ * @license Apache-2.0
+ */
+
+import Operation from "../Operation";
+import OperationError from "../errors/OperationError";
+import Utils from "../Utils";
+import {alphabetName, ALPHABET_OPTIONS} from "../lib/Base85";
+
+/**
+ * From Base85 operation
+ */
+class FromBase85 extends Operation {
+
+    /**
+     * From Base85 constructor
+     */
+    constructor() {
+        super();
+
+        this.name = "From Base85";
+        this.module = "Default";
+        this.description = "Base85 (also called Ascii85) is a notation for encoding arbitrary byte data. It is usually more efficient that Base64.<br><br>This operation decodes data from an ASCII string (with an alphabet of your choosing, presets included).<br><br>e.g. <code>BOu!rD]j7BEbo7</code> becomes <code>hello world</code><br><br>Base85 is commonly used in Adobe's PostScript and PDF file formats.";
+        this.infoURL = "https://wikipedia.org/wiki/Ascii85";
+        this.inputType = "string";
+        this.outputType = "byteArray";
+        this.args = [
+            {
+                name: "Alphabet",
+                type: "editableOption",
+                value: ALPHABET_OPTIONS
+            },
+        ];
+    }
+
+    /**
+     * @param {string} input
+     * @param {Object[]} args
+     * @returns {byteArray}
+     */
+    run(input, args) {
+        const alphabet = Utils.expandAlphRange(args[0]).join(""),
+            encoding = alphabetName(alphabet),
+            result = [];
+
+        if (alphabet.length !== 85 ||
+            [].unique.call(alphabet).length !== 85) {
+            throw new OperationError("Alphabet must be of length 85");
+        }
+
+        if (input.length === 0) return [];
+
+        const matches = input.match(/<~(.+?)~>/);
+        if (matches !== null) input = matches[1];
+
+        let i = 0;
+        let block, blockBytes;
+        while (i < input.length) {
+            if (encoding === "Standard" && input[i] === "z") {
+                result.push(0, 0, 0, 0);
+                i++;
+            } else {
+                let digits = [];
+                digits = input
+                    .substr(i, 5)
+                    .split("")
+                    .map((chr, idx) => {
+                        const digit = alphabet.indexOf(chr);
+                        if (digit < 0 || digit > 84) {
+                            throw `Invalid character '${chr}' at index ${idx}`;
+                        }
+                        return digit;
+                    });
+
+                block =
+                    digits[0] * 52200625 +
+                    digits[1] * 614125 +
+                    (i + 2 < input.length ? digits[2] : 84) * 7225 +
+                    (i + 3 < input.length ? digits[3] : 84) * 85 +
+                    (i + 4 < input.length ? digits[4] : 84);
+
+                blockBytes = [
+                    (block >> 24) & 0xff,
+                    (block >> 16) & 0xff,
+                    (block >> 8) & 0xff,
+                    block & 0xff
+                ];
+
+                if (input.length < i + 5) {
+                    blockBytes.splice(input.length - (i + 5), 5);
+                }
+
+                result.push.apply(result, blockBytes);
+                i += 5;
+            }
+        }
+
+        return result;
+    }
+
+}
+
+export default FromBase85;

src/core/operations/FromMessagePack.mjs

@@ -0,0 +1,47 @@
+/**
+ * @author Matt C [matt@artemisbot.uk]
+ * @copyright Crown Copyright 2018
+ * @license Apache-2.0
+ */
+
+import Operation from "../Operation";
+import OperationError from "../errors/OperationError";
+import notepack from "notepack.io";
+
+/**
+ * From MessagePack operation
+ */
+class FromMessagePack extends Operation {
+
+    /**
+     * FromMessagePack constructor
+     */
+    constructor() {
+        super();
+
+        this.name = "From MessagePack";
+        this.module = "Code";
+        this.description = "Converts MessagePack encoded data to JSON. MessagePack is a computer data interchange format. It is a binary form for representing simple data structures like arrays and associative arrays.";
+        this.infoURL = "https://wikipedia.org/wiki/MessagePack";
+        this.inputType = "ArrayBuffer";
+        this.outputType = "JSON";
+        this.args = [];
+    }
+
+    /**
+     * @param {ArrayBuffer} input
+     * @param {Object[]} args
+     * @returns {JSON}
+     */
+    run(input, args) {
+        try {
+            const buf = Buffer.from(new Uint8Array(input));
+            return notepack.decode(buf);
+        } catch (err) {
+            throw new OperationError(`Could not decode MessagePack to JSON: ${err}`);
+        }
+    }
+
+}
+
+export default FromMessagePack;

src/core/operations/Register.mjs

@@ -6,6 +6,7 @@
 
 import Operation from "../Operation";
 import Dish from "../Dish";
+import XRegExp from "xregexp";
 
 /**
  * Register operation
@@ -20,7 +21,7 @@ class Register extends Operation {
 
         this.name = "Register";
         this.flowControl = true;
-        this.module = "Default";
+        this.module = "Regex";
         this.description = "Extract data from the input and store it in registers which can then be passed into subsequent operations as arguments. Regular expression capture groups are used to select the data to extract.<br><br>To use registers in arguments, refer to them using the notation <code>$Rn</code> where n is the register number, starting at 0.<br><br>For example:<br>Input: <code>Test</code><br>Extractor: <code>(.*)</code><br>Argument: <code>$R0</code> becomes <code>Test</code><br><br>Registers can be escaped in arguments using a backslash. e.g. <code>\\$R0</code> would become <code>$R0</code> rather than <code>Test</code>.";
         this.infoURL = "https://wikipedia.org/wiki/Regular_expression#Syntax";
         this.inputType = "string";
@@ -40,6 +41,11 @@ class Register extends Operation {
                 "name": "Multiline matching",
                 "type": "boolean",
                 "value": false
+            },
+            {
+                "name": "Dot matches all",
+                "type": "boolean",
+                "value": false
             }
         ];
     }
@@ -53,13 +59,14 @@ class Register extends Operation {
      */
     async run(state) {
         const ings = state.opList[state.progress].ingValues;
-        const [extractorStr, i, m] = ings;
+        const [extractorStr, i, m, s] = ings;
 
         let modifiers = "";
         if (i) modifiers += "i";
         if (m) modifiers += "m";
+        if (s) modifiers += "s";
 
-        const extractor = new RegExp(extractorStr, modifiers),
+        const extractor = new XRegExp(extractorStr, modifiers),
             input = await state.dish.get(Dish.STRING),
             registers = input.match(extractor);
 

src/core/operations/TakeBytes.mjs

@@ -5,7 +5,6 @@
  */
 
 import Operation from "../Operation";
-import OperationError from "../errors/OperationError";
 
 /**
  * Take bytes operation
@@ -20,7 +19,7 @@ class TakeBytes extends Operation {
 
         this.name = "Take bytes";
         this.module = "Default";
-        this.description = "Takes a slice of the specified number of bytes from the data.";
+        this.description = "Takes a slice of the specified number of bytes from the data. Negative values are allowed.";
         this.inputType = "ArrayBuffer";
         this.outputType = "ArrayBuffer";
         this.args = [
@@ -50,15 +49,27 @@ class TakeBytes extends Operation {
      * @throws {OperationError} if invalid value
      */
     run(input, args) {
-        const start = args[0],
-            length = args[1],
-            applyToEachLine = args[2];
+        let start = args[0],
+            length = args[1];
+        const applyToEachLine = args[2];
 
-        if (start < 0 || length < 0)
-            throw new OperationError("Error: Invalid value");
+        if (!applyToEachLine) {
+            if (start < 0) { // Take from the end
+                start = input.byteLength + start;
+            }
+
+            if (length < 0) { // Flip start point
+                start = start + length;
+                if (start < 0) {
+                    start = input.byteLength + start;
+                    length = start - length;
+                } else {
+                    length = -length;
+                }
+            }
 
-        if (!applyToEachLine)
             return input.slice(start, start+length);
+        }
 
         // Split input into lines
         const data = new Uint8Array(input);
@@ -77,9 +88,26 @@ class TakeBytes extends Operation {
         lines.push(line);
 
         let output = [];
+        let s = start,
+            l = length;
         for (i = 0; i < lines.length; i++) {
-            output = output.concat(lines[i].slice(start, start+length));
+            if (s < 0) { // Take from the end
+                s = lines[i].length + s;
+            }
+
+            if (l < 0) { // Flip start point
+                s = s + l;
+                if (s < 0) {
+                    s = lines[i].length + s;
+                    l = s - l;
+                } else {
+                    l = -l;
+                }
+            }
+            output = output.concat(lines[i].slice(s, s+l));
             output.push(0x0a);
+            s = start;
+            l = length;
         }
         return new Uint8Array(output.slice(0, output.length-1)).buffer;
     }

src/core/operations/ToBase85.mjs

@@ -0,0 +1,93 @@
+/**
+ * @author PenguinGeorge [george@penguingeorge.com]
+ * @copyright Crown Copyright 2018
+ * @license Apache-2.0
+ */
+
+import Operation from "../Operation";
+import OperationError from "../errors/OperationError";
+import Utils from "../Utils";
+import {alphabetName, ALPHABET_OPTIONS} from "../lib/Base85";
+
+/**
+ * To Base85 operation
+ */
+class ToBase85 extends Operation {
+
+    /**
+     * To Base85 constructor
+     */
+    constructor() {
+        super();
+
+        this.name = "To Base85";
+        this.module = "Default";
+        this.description = "Base85 (also called Ascii85) is a notation for encoding arbitrary byte data. It is usually more efficient that Base64.<br><br>This operation encodes data in an ASCII string (with an alphabet of your choosing, presets included).<br><br>e.g. <code>hello world</code> becomes <code>BOu!rD]j7BEbo7</code><br><br>Base85 is commonly used in Adobe's PostScript and PDF file formats.<br><br><strong>Options</strong><br><u>Alphabet</u><ul><li>Standard - The standard alphabet, referred to as Ascii85</li><li>Z85 (ZeroMQ) - A string-safe variant of Base85, which avoids quote marks and backslash characters</li><li>IPv6 - A variant of Base85 suitable for encoding IPv6 addresses (RFC 1924)</li></ul><u>Include delimiter</u><br>Adds a '<~' and '~>' delimiter to the start and end of the data. This is standard for Adobe's implementation of Base85.";
+        this.infoURL = "https://wikipedia.org/wiki/Ascii85";
+        this.inputType = "byteArray";
+        this.outputType = "string";
+        this.args = [
+            {
+                name: "Alphabet",
+                type: "editableOption",
+                value: ALPHABET_OPTIONS
+            },
+            {
+                name: "Include delimeter",
+                type: "boolean",
+                value: false
+            }
+        ];
+    }
+
+    /**
+     * @param {byteArray} input
+     * @param {Object[]} args
+     * @returns {string}
+    */
+    run(input, args) {
+        const alphabet = Utils.expandAlphRange(args[0]).join(""),
+            encoding = alphabetName(alphabet),
+            includeDelim = args[1];
+        let result = "";
+
+        if (alphabet.length !== 85 ||
+            [].unique.call(alphabet).length !== 85) {
+            throw new OperationError("Error: Alphabet must be of length 85");
+        }
+
+        if (input.length === 0) return "";
+
+        let block;
+        for (let i = 0; i < input.length; i += 4) {
+            block = (
+                ((input[i])          << 24) +
+                ((input[i + 1] || 0) << 16) +
+                ((input[i + 2] || 0) << 8)  +
+                ((input[i + 3] || 0))
+            ) >>> 0;
+
+            if (encoding !== "Standard" || block > 0) {
+                let digits = [];
+                for (let j = 0; j < 5; j++) {
+                    digits.push(block % 85);
+                    block = Math.floor(block / 85);
+                }
+
+                digits = digits.reverse();
+
+                if (input.length < i + 4) {
+                    digits.splice(input.length - (i + 4), 4);
+                }
+
+                result += digits.map(digit => alphabet[digit]).join("");
+            } else {
+                result += (encoding === "Standard") ? "z" : null;
+            }
+        }
+
+        return includeDelim ? `<~${result}~>` : result;
+    }
+}
+
+export default ToBase85;

src/core/operations/ToMessagePack.mjs

@@ -0,0 +1,50 @@
+/**
+ * @author Matt C [matt@artemisbot.uk]
+ * @copyright Crown Copyright 2018
+ * @license Apache-2.0
+ */
+
+import Operation from "../Operation";
+import OperationError from "../errors/OperationError.mjs";
+import notepack from "notepack.io";
+
+/**
+ * To MessagePack operation
+ */
+class ToMessagePack extends Operation {
+
+    /**
+     * ToMessagePack constructor
+     */
+    constructor() {
+        super();
+
+        this.name = "To MessagePack";
+        this.module = "Code";
+        this.description = "Converts JSON to MessagePack encoded byte buffer. MessagePack is a computer data interchange format. It is a binary form for representing simple data structures like arrays and associative arrays.";
+        this.infoURL = "https://wikipedia.org/wiki/MessagePack";
+        this.inputType = "JSON";
+        this.outputType = "ArrayBuffer";
+        this.args = [];
+    }
+
+    /**
+     * @param {JSON} input
+     * @param {Object[]} args
+     * @returns {ArrayBuffer}
+     */
+    run(input, args) {
+        try {
+            if (ENVIRONMENT_IS_WORKER()) {
+                return notepack.encode(input);
+            } else {
+                return notepack.encode(input).buffer;
+            }
+        } catch (err) {
+            throw new OperationError(`Could not encode JSON to MessagePack: ${err}`);
+        }
+    }
+
+}
+
+export default ToMessagePack;

test/tests/operations/Code.mjs

@@ -9,7 +9,7 @@
  */
 import TestRegister from "../../TestRegister";
 
-const JPATH_TEST_DATA = {
+const JSON_TEST_DATA = {
     "store": {
         "book": [{
             "category": "reference",
@@ -184,7 +184,7 @@ TestRegister.addTests([
     },
     {
         name: "JPath Expression: Empty expression",
-        input: JSON.stringify(JPATH_TEST_DATA),
+        input: JSON.stringify(JSON_TEST_DATA),
         expectedOutput: "Invalid JPath expression: we need a path",
         recipeConfig: [
             {
@@ -195,7 +195,7 @@ TestRegister.addTests([
     },
     {
         name: "JPath Expression: Fetch of values from specific object",
-        input: JSON.stringify(JPATH_TEST_DATA),
+        input: JSON.stringify(JSON_TEST_DATA),
         expectedOutput: [
             "\"Nigel Rees\"",
             "\"Evelyn Waugh\"",
@@ -211,7 +211,7 @@ TestRegister.addTests([
     },
     {
         name: "JPath Expression: Fetch of all values with matching key",
-        input: JSON.stringify(JPATH_TEST_DATA),
+        input: JSON.stringify(JSON_TEST_DATA),
         expectedOutput: [
             "\"Sayings of the Century\"",
             "\"Sword of Honour\"",
@@ -229,7 +229,7 @@ TestRegister.addTests([
     },
     {
         name: "JPath Expression: All data in object",
-        input: JSON.stringify(JPATH_TEST_DATA),
+        input: JSON.stringify(JSON_TEST_DATA),
         expectedOutput: [
             "[{\"category\":\"reference\",\"author\":\"Nigel Rees\",\"title\":\"Sayings of the Century\",\"price\":8.95},{\"category\":\"fiction\",\"author\":\"Evelyn Waugh\",\"title\":\"Sword of Honour\",\"price\":12.99},{\"category\":\"fiction\",\"author\":\"Herman Melville\",\"title\":\"Moby Dick\",\"isbn\":\"0-553-21311-3\",\"price\":8.99},{\"category\":\"fiction\",\"author\":\"J. R. R. Tolkien\",\"title\":\"The Lord of the Rings\",\"isbn\":\"0-395-19395-8\",\"price\":22.99}]",
             "{\"color\":\"red\",\"price\":19.95}",
@@ -244,7 +244,7 @@ TestRegister.addTests([
     },
     {
         name: "JPath Expression: Last element in array",
-        input: JSON.stringify(JPATH_TEST_DATA),
+        input: JSON.stringify(JSON_TEST_DATA),
         expectedOutput: "{\"category\":\"fiction\",\"author\":\"J. R. R. Tolkien\",\"title\":\"The Lord of the Rings\",\"isbn\":\"0-395-19395-8\",\"price\":22.99}",
         recipeConfig: [
             {
@@ -255,7 +255,7 @@ TestRegister.addTests([
     },
     {
         name: "JPath Expression: First 2 elements in array",
-        input: JSON.stringify(JPATH_TEST_DATA),
+        input: JSON.stringify(JSON_TEST_DATA),
         expectedOutput: [
             "{\"category\":\"reference\",\"author\":\"Nigel Rees\",\"title\":\"Sayings of the Century\",\"price\":8.95}",
             "{\"category\":\"fiction\",\"author\":\"Evelyn Waugh\",\"title\":\"Sword of Honour\",\"price\":12.99}"
@@ -269,7 +269,7 @@ TestRegister.addTests([
     },
     {
         name: "JPath Expression: All elements in array with property",
-        input: JSON.stringify(JPATH_TEST_DATA),
+        input: JSON.stringify(JSON_TEST_DATA),
         expectedOutput: [
             "{\"category\":\"fiction\",\"author\":\"Herman Melville\",\"title\":\"Moby Dick\",\"isbn\":\"0-553-21311-3\",\"price\":8.99}",
             "{\"category\":\"fiction\",\"author\":\"J. R. R. Tolkien\",\"title\":\"The Lord of the Rings\",\"isbn\":\"0-395-19395-8\",\"price\":22.99}"
@@ -283,7 +283,7 @@ TestRegister.addTests([
     },
     {
         name: "JPath Expression: All elements in array which meet condition",
-        input: JSON.stringify(JPATH_TEST_DATA),
+        input: JSON.stringify(JSON_TEST_DATA),
         expectedOutput: [
             "{\"category\":\"fiction\",\"author\":\"Evelyn Waugh\",\"title\":\"Sword of Honour\",\"price\":12.99}",
             "{\"category\":\"fiction\",\"author\":\"Herman Melville\",\"title\":\"Moby Dick\",\"isbn\":\"0-553-21311-3\",\"price\":8.99}",
@@ -298,7 +298,7 @@ TestRegister.addTests([
     },
     {
         name: "JPath Expression: All elements in object",
-        input: JSON.stringify(JPATH_TEST_DATA),
+        input: JSON.stringify(JSON_TEST_DATA),
         expectedOutput: [
             "{\"category\":\"reference\",\"author\":\"Nigel Rees\",\"title\":\"Sayings of the Century\",\"price\":8.95}",
             "{\"category\":\"fiction\",\"author\":\"Herman Melville\",\"title\":\"Moby Dick\",\"isbn\":\"0-553-21311-3\",\"price\":8.99}"
@@ -331,5 +331,65 @@ TestRegister.addTests([
                 "args": ["/div/p[@class=\"a\"]", "\\n"]
             }
         ]
+    },
+    {
+        name: "To MessagePack: no content",
+        input: "",
+        expectedError: true,
+        recipeConfig: [
+            {
+                "op": "To MessagePack",
+                "args": []
+            },
+            {
+                "op": "To Hex",
+                "args": ["Space"]
+            }
+        ]
+    },
+    {
+        name: "From MessagePack: no content",
+        input: "",
+        expectedOutput: "Could not decode MessagePack to JSON: Error: Could not parse",
+        recipeConfig: [
+            {
+                "op": "From Hex",
+                "args": ["Space"]
+            },
+            {
+                "op": "From MessagePack",
+                "args": []
+            }
+        ]
+    },
+    {
+        name: "To MessagePack: valid json",
+        input: JSON.stringify(JSON_TEST_DATA),
+        expectedOutput: "81 a5 73 74 6f 72 65 83 a4 62 6f 6f 6b 94 84 a8 63 61 74 65 67 6f 72 79 a9 72 65 66 65 72 65 6e 63 65 a6 61 75 74 68 6f 72 aa 4e 69 67 65 6c 20 52 65 65 73 a5 74 69 74 6c 65 b6 53 61 79 69 6e 67 73 20 6f 66 20 74 68 65 20 43 65 6e 74 75 72 79 a5 70 72 69 63 65 cb 40 21 e6 66 66 66 66 66 84 a8 63 61 74 65 67 6f 72 79 a7 66 69 63 74 69 6f 6e a6 61 75 74 68 6f 72 ac 45 76 65 6c 79 6e 20 57 61 75 67 68 a5 74 69 74 6c 65 af 53 77 6f 72 64 20 6f 66 20 48 6f 6e 6f 75 72 a5 70 72 69 63 65 cb 40 29 fa e1 47 ae 14 7b 85 a8 63 61 74 65 67 6f 72 79 a7 66 69 63 74 69 6f 6e a6 61 75 74 68 6f 72 af 48 65 72 6d 61 6e 20 4d 65 6c 76 69 6c 6c 65 a5 74 69 74 6c 65 a9 4d 6f 62 79 20 44 69 63 6b a4 69 73 62 6e ad 30 2d 35 35 33 2d 32 31 33 31 31 2d 33 a5 70 72 69 63 65 cb 40 21 fa e1 47 ae 14 7b 85 a8 63 61 74 65 67 6f 72 79 a7 66 69 63 74 69 6f 6e a6 61 75 74 68 6f 72 b0 4a 2e 20 52 2e 20 52 2e 20 54 6f 6c 6b 69 65 6e a5 74 69 74 6c 65 b5 54 68 65 20 4c 6f 72 64 20 6f 66 20 74 68 65 20 52 69 6e 67 73 a4 69 73 62 6e ad 30 2d 33 39 35 2d 31 39 33 39 35 2d 38 a5 70 72 69 63 65 cb 40 36 fd 70 a3 d7 0a 3d a7 62 69 63 79 63 6c 65 82 a5 63 6f 6c 6f 72 a3 72 65 64 a5 70 72 69 63 65 cb 40 33 f3 33 33 33 33 33 a9 6e 65 77 73 70 61 70 65 72 92 83 a6 66 6f 72 6d 61 74 aa 62 72 6f 61 64 73 68 65 65 74 a5 74 69 74 6c 65 af 46 69 6e 61 6e 63 69 61 6c 20 54 69 6d 65 73 a5 70 72 69 63 65 cb 40 06 00 00 00 00 00 00 83 a6 66 6f 72 6d 61 74 a7 74 61 62 6c 6f 69 64 a5 74 69 74 6c 65 ac 54 68 65 20 47 75 61 72 64 69 61 6e a5 70 72 69 63 65 02",
+        recipeConfig: [
+            {
+                "op": "To MessagePack",
+                "args": []
+            },
+            {
+                "op": "To Hex",
+                "args": ["Space"]
+            }
+        ]
+    },
+    {
+        name: "From MessagePack: valid msgpack",
+        input: "81 a5 73 74 6f 72 65 83 a4 62 6f 6f 6b 94 84 a8 63 61 74 65 67 6f 72 79 a9 72 65 66 65 72 65 6e 63 65 a6 61 75 74 68 6f 72 aa 4e 69 67 65 6c 20 52 65 65 73 a5 74 69 74 6c 65 b6 53 61 79 69 6e 67 73 20 6f 66 20 74 68 65 20 43 65 6e 74 75 72 79 a5 70 72 69 63 65 cb 40 21 e6 66 66 66 66 66 84 a8 63 61 74 65 67 6f 72 79 a7 66 69 63 74 69 6f 6e a6 61 75 74 68 6f 72 ac 45 76 65 6c 79 6e 20 57 61 75 67 68 a5 74 69 74 6c 65 af 53 77 6f 72 64 20 6f 66 20 48 6f 6e 6f 75 72 a5 70 72 69 63 65 cb 40 29 fa e1 47 ae 14 7b 85 a8 63 61 74 65 67 6f 72 79 a7 66 69 63 74 69 6f 6e a6 61 75 74 68 6f 72 af 48 65 72 6d 61 6e 20 4d 65 6c 76 69 6c 6c 65 a5 74 69 74 6c 65 a9 4d 6f 62 79 20 44 69 63 6b a4 69 73 62 6e ad 30 2d 35 35 33 2d 32 31 33 31 31 2d 33 a5 70 72 69 63 65 cb 40 21 fa e1 47 ae 14 7b 85 a8 63 61 74 65 67 6f 72 79 a7 66 69 63 74 69 6f 6e a6 61 75 74 68 6f 72 b0 4a 2e 20 52 2e 20 52 2e 20 54 6f 6c 6b 69 65 6e a5 74 69 74 6c 65 b5 54 68 65 20 4c 6f 72 64 20 6f 66 20 74 68 65 20 52 69 6e 67 73 a4 69 73 62 6e ad 30 2d 33 39 35 2d 31 39 33 39 35 2d 38 a5 70 72 69 63 65 cb 40 36 fd 70 a3 d7 0a 3d a7 62 69 63 79 63 6c 65 82 a5 63 6f 6c 6f 72 a3 72 65 64 a5 70 72 69 63 65 cb 40 33 f3 33 33 33 33 33 a9 6e 65 77 73 70 61 70 65 72 92 83 a6 66 6f 72 6d 61 74 aa 62 72 6f 61 64 73 68 65 65 74 a5 74 69 74 6c 65 af 46 69 6e 61 6e 63 69 61 6c 20 54 69 6d 65 73 a5 70 72 69 63 65 cb 40 06 00 00 00 00 00 00 83 a6 66 6f 72 6d 61 74 a7 74 61 62 6c 6f 69 64 a5 74 69 74 6c 65 ac 54 68 65 20 47 75 61 72 64 69 61 6e a5 70 72 69 63 65 02",
+        expectedOutput: JSON.stringify(JSON_TEST_DATA),
+        recipeConfig: [
+            {
+                "op": "From Hex",
+                "args": ["Space"]
+            },
+            {
+                "op": "From MessagePack",
+                "args": []
+            }
+        ]
     }
 ]);