v0.14.16

11 commits from 4 contributors.

Changes

• fix nats-streaming version in peerDependencies.
• RedisCacher pingInterval option. #961
• Update NATS transporter messages to debug. #963
• update d.ts file. #964 #966
• update dependencies

v0.14.15

15 commits from 5 contributors.

Changes

• fix nats version in peerDependencies.
• convert url to servers in nats@2. #954
• add typing for mcall settled option. #957
• revert TS ThisType issue in 0.14.14. #958
• update dependencies
• add useTag259ForMaps: false default option for CBOR serializer to keep the compatibility.

v0.14.14

105 commits from 11 contributors.

New CBOR serializer #905

CBOR (cbor-x) is a new serializer but faster than any other serializers.

Example

// moleculer.config.js
module.exports = {
    logger: true,
    serializer: "CBOR"
};

Benchmark

Suite: Serialize packet with 10bytes
√ JSON               509,217 rps
√ Avro               308,711 rps
√ MsgPack             79,932 rps
√ ProtoBuf           435,183 rps
√ Thrift              93,324 rps
√ Notepack           530,121 rps
√ CBOR             1,016,135 rps

   JSON (#)            0%        (509,217 rps)   (avg: 1μs)
   Avro           -39.38%        (308,711 rps)   (avg: 3μs)
   MsgPack         -84.3%         (79,932 rps)   (avg: 12μs)
   ProtoBuf       -14.54%        (435,183 rps)   (avg: 2μs)
   Thrift         -81.67%         (93,324 rps)   (avg: 10μs)
   Notepack        +4.11%        (530,121 rps)   (avg: 1μs)
   CBOR           +99.55%      (1,016,135 rps)   (avg: 984ns)

settled option in broker.mcall

The broker.mcall method has a new settled option to receive all Promise results. Without this option, if you make a multi-call and one call is rejected, the response will be a rejected Promise and you don't know how many (and which) calls were rejected.

If settled: true, the method returns a resolved Promise in any case and the response contains the statuses and responses of all calls.

Example

const res = await broker.mcall([
    { action: "posts.find", params: { limit: 2, offset: 0 },
    { action: "users.find", params: { limit: 2, sort: "username" } },
    { action: "service.notfound", params: { notfound: 1 } }
], { settled: true });
console.log(res);

The res will be something similar to

[
    { status: "fulfilled", value: [/*... response of `posts.find`...*/] },
    { status: "fulfilled", value: [/*... response of `users.find`...*/] },
    { status: "rejected", reason: {/*... Rejected response/Error`...*/} }
]

New MOLECULER_CONFIG environment variable in Runner

In the Moleculer Runner, you can configure the configuration filename and path with the MOLECULER_CONFIG environment variable. It means, no need to specify the config file with --config argument.

Supporting [email protected] in NATS transporter

The new nats 2.x.x version has a new breaking API which has locked the NATS transporter for [email protected] library. As of this release, the NATS transporter supports both major versions of the nats library.

The transporter automatically detects the version of the library and uses the correct API.

Async custom validator functions and ctx as metadata

Since [email protected], the FastestValidator supports async custom validators and you can pass metadata for custom validator functions.
In Moleculer, the FastestValidator passes the ctx as metadata. It means you can access to the current context, service, broker and you can make async calls (e.g calling another service) in custom checker functions.

Example

// posts.service.js
module.exports = {
    name: "posts",
    actions: {
        params: {
            $$async: true,
            owner: { type: "string", custom: async (value, errors, schema, name, parent, context) => {
                const ctx = context.meta;

                const res = await ctx.call("users.isValid", { id: value });
                if (res !== true)
                    errors.push({ type: "invalidOwner", field: "owner", actual: value });
                return value;
            } }, 
        },
        /* ... */
    }
}

Changes

• fix node crash in encryption mode with TCP transporter. #849
• expose Utils in typescript definition. #909
• other d.ts improvements. #920, #922, #934, #950
• fix etcd3 discoverer lease-loss issue #922
• catch errors in Compression and Encryption middlewares. #850
• using optional peer dependencies. #911
• add relevant packet to to serialization and deserialization calls. #932
• fix disabled balancer issue with external discoverer. #933

v0.14.13

Changes (62 commits from 12 contributors)

• update dependencies
• logging if encryption middleware can't decrypt the data instead of crashing. #853
• fix disableHeartbeatChecks option handling. #858
• force scanning only master redis nodes for deletion. #865
• add more info into waitForServices debug log messages. #870
• fix EVENT packet Avro schema. #856
• fix array & date conversion in cacher default key generator. #883
• fix Datadog tracing exporter. #890
• better elapsed time handling in tracing. #899
• improve type definitions. #843, #885, #886
• add E2E tests for CI (test all built-in transporter & serializers)

v0.14.12

Other changes

• update dependencies
• improved type definitions. #816 #817 #834 #840
• support rediss:// cacher URI. #837
• fix Event Trace exporter generated events loop. #836
• change log level of node disconnected message. #838
• improve the broker.waitForServices response. #843
• fix recursive hot-reload issue on Linux OS. #848

v0.14.11

New merged service lifecycle hook

Service has a new merged lifecycle hook which is called after the service schemas (including mixins) has been merged but before service is registered. It means you can manipulate the merged service schema before it's processed.

Example

// posts.service.js
module.exports = {
    name: "posts",

    settings: {},

    actions: {
        find: {
            params: {
                limit: "number"
            }
            handler(ctx) {
                // ...
            }
        }
    },

    merged(schema) {
        // Modify the service settings
        schema.settings.myProp = "myValue";
        // Modify the param validation schema in an action schema
        schema.actions.find.params.offset = "number";
    }
};

Other changes

• add requestID tag to all action and event spans #802
• fix bug in second level of mixins with $secureSettings #811

v0.14.10

Changes

• update dependencies
• fix issues in index.d.ts
• fix broadcast event sending issue when disableBalancer: true #799 (thanks for @ngraef)

v0.14.9

Register method in module resolver

If you create a custom module (e.g. serializer), you can register it into the built-in modules with the register method. This method is also available in all other built-in module resolver.

Example

// SafeJsonSerializer.js
const { Serializers } = require("moleculer");

class SafeJsonSerializer {}

Serializers.register("SafeJSON", SafeJSON);

module.exports = SafeJsonSerializer;

// moleculer.config.js
require("./SafeJsonSerializer");

module.exports = {
    nodeID: "node-100",
    serializer: "SafeJSON"
    // ...
});

Changeable validation property name

You can change the params property name in validator options. It can be useful if you have a custom Validator implementation.

const broker = new ServiceBroker({
    validator: {
        type: "Fastest",
        options: {
            paramName: "myParams" // using `myParams` instead of `params`
        }
    }
});

broker.createService({
    name: "posts",
    actions: {
        create: {
            myParams: {
                title: "string"
            }
        },
        handler(ctx) { /* ... */ }
    }
});

Global tracing span tags for actions & events

Thanks for @kthompson23, you can configure the action & events tracing span tags globally. These tags will be used for all actions & events where tracing is enabled. Of course, you can overwrite them locally in the action & event schema.

Example

// moleculer.config.js
module.exports = {
    tracing: {
        enabled: true,
        exporter: 'Zipkin',
        tags: {
            action: {
                meta: ['app.id', 'user.email', 'workspace.name'],
                params: false,  // overrides default behavior of all params being adding as tags
                response: true,
            },
            event: (ctx) {
                return {
                    caller: ctx.caller,
                }
            }  
        }
    }
}

Other changes

• fix multiple trace spans with same ID issue in case of retries.
• update dependencies
• add lastValue property to histogram metric type.
• update return type of context's copy method.
• add configuration hotReloadModules #779
• Remove Zipkin v1 annotations and change kind to SERVER

v0.14.8

Github Sponsoring is available for Moleculer 🎉

We have been approved in Github Sponsors program, so you can sponsor the Moleculer project via Github Sponsors.
If you have taxing problem with Patreon, change to Github Sponsors.

New Validator configuration

The validator has the same module configuration in broker options like other modules. It means you can configure the validation constructor options via broker options (moleculer.config.js).

Default usage:

//moleculer.config.js
module.exports = {
    nodeID: "node-100",
    validator: true // Using the default Fastest Validator
}

Using built-in validator name:

//moleculer.config.js
module.exports = {
    nodeID: "node-100",
    validator: "FastestValidator" // Using the Fastest Validator
}

Example with options:

//moleculer.config.js
module.exports = {
    nodeID: "node-100",
    validator: {
        type: "FastestValidator",
        options: {
            useNewCustomCheckerFunction: true,
            defaults: { /*...*/ },
            messages: { /*...*/ },
            aliases: { /*...*/ }
        }
    }
}

Example with custom validator

//moleculer.config.js
const BaseValidator = require("moleculer").Validators.Base;

class MyValidator extends BaseValidator {}

module.exports = {
    nodeID: "node-100",
    validator: new MyValidator()
}

New metrics

Added the following new metrics:

• moleculer.event.received.active: Number of active event executions.
• moleculer.event.received.error.total: Number of event execution errors.
• moleculer.event.received.time: Execution time of events in milliseconds.
• os.memory.total: OS used memory size.

Other changes

• support using moleculer.config.js with export default.
• remove some lodash methods.
• upgrade to the latest tsd.
• new dependencyInterval broker option. Using as default value for broker.waitForServices #761
• fix Datadog traceID, spanID generation logic to work with latest dd-trace.
• add error stack trace to EventLegacy trace exporter.
• fix INFO key issue in Redis Discoverer after Redis server restarting.

v0.14.7

New Discoverer module

The Discoverer is a new built-in module in Moleculer framework. It's responsible for that all Moleculer nodes can discover each other and check them with heartbeats. In previous versions, it was an integrated module inside ServiceRegistry & Transit modules. In this version, the discovery logic has been extracted to a separated built-in module. It means you can replace it to other built-in implementations or a custom one. The current discovery & heartbeat logic is moved to the Local Discoverer.

Nevertheless, this version also contains other discoverers, like Redis & etcd3 discoverers. Both of them require an external server to make them work.
One of the main advantages of the external discoverers, the node discovery & heartbeat packets don't overload the communication on the transporter. The transporter transfers only the request, response, event packets.
By the way, the external discoverers have some disadvantages, as well. These discoverers can detect lazier the new and disconnected nodes because they scan the heartbeat keys periodically on the remote Redis/etcd3 server. The period of checks depends on the heartbeatInterval broker option.

The Local Discoverer (which is the default) works like the discovery of previous versions, so if you want to keep the original logic, you'll have to do nothing.

Please note the TCP transporter uses Gossip protocol & UDP packets for discovery & heartbeats, it means it can work only with Local Discoverer.

Local Discoverer

It's the default Discoverer, it uses the transporter module to discover all other moleculer nodes. It's quick and fast but if you have too many nodes (>100), the nodes can generate a lot of heartbeat packets which can reduce the performance of request/response packets.

Example

// moleculer.config.js
module.exports = {
    registry: {
        discoverer: "Local"
    }    
}

Example with options

// moleculer.config.js
module.exports = {
    registry: {
        discoverer: {
            type: "Local",
            options: {
                // Send heartbeat in every 10 seconds
                heartbeatInterval: 10,

                // Heartbeat timeout in seconds
                heartbeatTimeout: 30,

                // Disable heartbeat checking & sending, if true
                disableHeartbeatChecks: false,

                // Disable removing offline nodes from registry, if true
                disableOfflineNodeRemoving: false,

                // Remove offline nodes after 10 minutes
                cleanOfflineNodesTimeout: 10 * 60
            }
        }
    }    
}

Redis Discoverer

This is an experimental module. Do not use it in production yet!

This Discoverer uses a Redis server to discover Moleculer nodes. The heartbeat & discovery packets are stored in Redis keys. Thanks to Redis key expiration, if a node crashed or disconnected unexpectedly, the Redis will remove its heartbeat keys from the server and other nodes can detect it.

Example to connect local Redis server

// moleculer.config.js
module.exports = {
    registry: {
        discoverer: "Redis"
    }    
}

Example to connect remote Redis server

// moleculer.config.js
module.exports = {
    registry: {
        discoverer: "redis://redis-server:6379"
    }    
}

Example with options

// moleculer.config.js
module.exports = {
    registry: {
        discoverer: {
            type: "Redis",
            options: {
                redis: {
                    // Redis connection options.
                    // More info: https://github.com/luin/ioredis#connect-to-redis
                    port: 6379,
                    host: "redis-server",
                    password: "123456",
                    db: 3
                }

                // Serializer
                serializer: "JSON",

                // Full heartbeat checks. It generates more network traffic
                // 10 means every 10 cycle.
                fullCheck: 10,

                // Key scanning size
                scanLength: 100,

                // Monitoring Redis commands
                monitor: true,
                
                // --- COMMON DISCOVERER OPTIONS ---

                // Send heartbeat in every 10 seconds
                heartbeatInterval: 10,

                // Heartbeat timeout in seconds
                heartbeatTimeout: 30,

                // Disable heartbeat checking & sending, if true
                disableHeartbeatChecks: false,

                // Disable removing offline nodes from registry, if true
                disableOfflineNodeRemoving: false,

                // Remove offline nodes after 10 minutes
                cleanOfflineNodesTimeout: 10 * 60
            }
        }
    }    
}

To be able to use this Discoverer, install the ioredis module with the npm install ioredis --save command.

Tip: To further network traffic reduction, you can use MsgPack/Notepack serializers instead of JSON.

Etcd3 Discoverer

This is an experimental module. Do not use it in production yet!

This Discoverer uses an etcd3 server to discover Moleculer nodes. The heartbeat & discovery packets are stored in the server. Thanks to etcd3 lease solution, if a node crashed or disconnected unexpectedly, the etcd3 will remove its heartbeat keys from the server and other nodes can detect it.

Example to connect local etcd3 server

// moleculer.config.js
module.exports = {
    registry: {
        discoverer: "Etcd3"
    }    
}

Example to connect remote etcd3 server

// moleculer.config.js
module.exports = {
    registry: {
        discoverer: "etcd3://etcd-server:2379"
    }    
}

Example with options

// moleculer.config.js
module.exports = {
    registry: {
        discoverer: {
            type: "Etcd3",
            options: {
                etcd: {
                    // etcd3 connection options.
                    // More info: https://mixer.github.io/etcd3/interfaces/options_.ioptions.html
                    hosts: "etcd-server:2379",
                    auth: "12345678"
                }

                // Serializer
                serializer: "JSON",

                // Full heartbeat checks. It generates more network traffic
                // 10 means every 10 cycle.
                fullCheck: 10,
                
                // --- COMMON DISCOVERER OPTIONS ---

                // Send heartbeat in every 10 seconds
                heartbeatInterval: 10,

                // Heartbeat timeout in seconds
                heartbeatTimeout: 30,

                // Disable heartbeat checking & sending, if true
                disableHeartbeatChecks: false,

                // Disable removing offline nodes from registry, if true
                disableOfflineNodeRemoving: false,

                // Remove offline nodes after 10 minutes
                cleanOfflineNodesTimeout: 10 * 60
            }
        }
    }    
}

To be able to use this Discoverer, install the etcd3 module with the npm install etcd3--save command.

Tip: To further network traffic reduction, you can use MsgPack/Notepack serializers instead of JSON.

Custom Discoverer

You can create your custom Discoverer. We recommend to copy the source of Redis Discoverer and implement the necessary methods.

Other changes

• fixed multiple heartbeat sending issue at transporter reconnecting.
• the heartbeatInterval default value has been changed to 10 seconds.
• the heartbeatTimeout default value has been changed to 30 seconds.
• the heartbeat check logics use process uptime instead of timestamps in order to avoid issues with time synchronization or daylight saving.
• Notepack serializer initialization issue fixed. It caused problem when Redis cacher uses Notepack serializer.
• new removeFromArray function in Utils.
• mcall method definition fixed in Typescript definition.
• dependencies updated.