Some Publish Composite publications not working (renamed as tracker issue was redherring)

[adamgins]

I am running locally with with Meteor 1.8.2 and NGROK
I cranked up a redis instance on DigitalOcean

the first two items in my my packages file are

cultofcoders:redis-oplog
disable-oplog

When my sever starts up I see this on the console: RedisOplog - Established connection to redis.

My login screen comes up. I can login and some of my collections work... but I am getting this exception in the browser and can see one of the main pub/subs are not working. I cannot see any exceptions on the server console.

In browser console:

Exception from Tracker recompute function: undefined
meteor.js?hash=857dafb4b9dff17e29ed8498a22ea5b1a3d6b41d:1061 Error: Expected template or null, found: true
    at Blaze.View._render (https://<someurl>/packages/spacebars.js?hash=6f2be25813c793c0b363a6a91ebb029723f294ec:61:13)
    at Blaze.View.doRender (https://<someurl>/packages/blaze.js?hash=51f4a3bdae106610ee48d8eff291f3628713d847:2086:25)
    at https://<someurl>/packages/blaze.js?hash=51f4a3bdae106610ee48d8eff291f3628713d847:1934:20
    at Function.Template._withTemplateInstanceFunc (https://<someurl>/packages/blaze.js?hash=51f4a3bdae106610ee48d8eff291f3628713d847:3769:14)
    at https://<someurl>/packages/blaze.js?hash=51f4a3bdae106610ee48d8eff291f3628713d847:1932:29
    at Object.Blaze._withCurrentView (https://<someurl>/packages/blaze.js?hash=51f4a3bdae106610ee48d8eff291f3628713d847:2271:12)
    at Spacebars.include:materialize (https://<someurl>/packages/blaze.js?hash=51f4a3bdae106610ee48d8eff291f3628713d847:1931:18)
    at Computation._compute (https://<someurl>/packages/tracker.js?hash=a9bdf2f732520305683f894b4b4d4d49d90cbe05:332:38)
    at new Computation (https://<someurl>/packages/tracker.js?hash=a9bdf2f732520305683f894b4b4d4d49d90cbe05:229:12)
    at Object.Tracker.autorun (https://<someurl>/packages/tracker.js?hash=a9bdf2f732520305683f894b4b4d4d49d90cbe05:602:11) undefined

Any ideas please?

[ramezrafla]

I see you are using blaze. Did you wait for the sub to be ready?
You broke something in your blaze template, you are returning 'true' in one of your functions
Not easy to debug this. First thing to check, are you using sub.ready before computing vars

[adamgins]

Thanks. Yes Blaze.

Mmmm. I have a bunch template autoruns that check for subscription ready, I wonder why it's manifesting with Redis vs not with Meteor "normal"?

[adamgins]

ok, the tracker exception - may be a redherring (need to dig more)...

I put put some debug on the server and it's (publishComposite) finding the docs and returning the cursor... but hey never seem to make it back to the client

Update: mmm... when I run this on Galaxy on test server... alll works, so not sure if something to do with NGROK trying to connect to DigitalOcean?

That said, some of my publications have stopped being reactive - the data is there on first visit but not reactive. trying to debug

[adamgins]

Mmm.. I think the reactivity is borked as I am using bulk insert?

const bulkNotifications = Notifications.rawCollection().initializeUnorderedBulkOp()
...
bulkNotifications.insert(notification);
...
bulkNotifications.execute();

Is there a supported way to to bulk operations?

#40

[evolross]

Indeed redis-oplog doesn't work with bulk insert as Meteor Mongo doesn't work with bulk insert. You should probably trigger some kind of reactive flag or Vent to just have your template reload.

If you still need Meteor-esque granular per-document reactivity you could use the methodology documented in Outside Mutations about handling updates to Mongo outside of your app. As that's how bulk inserts/updates are treated with regard to Meteor. It's as if they didn't happen in your app - and some outside source added them. Also keep in mind when doing bulk inserts they won't get your Simple Schema auto values or Meteor _id's. You have to add them before inserting them.

[adamgins]

ok great, thanks @evolross

[adamgins]

@evolross mmm any guesses on what the performance would be like of :

1. bulkinsert of N, where 1,000 ... 10,000

• N x redis.publish
  VS

2. N x collection.insert

?

We used Bulk insert to try and reduce CPU utilization... just wondering what the extra redis.publishes will do?

ps. I will try and resolve with Vent first

[evolross]

This should be a non-issue. As you're probably not displaying/using 1000 to 10,000 documents on the client right? If not, then there's no need to worry about publishing them. Just add them to your database via bulk insert so they're on the server and then re-fire your subscription to get however many you actually need on the client.

If this is over 100 documents you should probably rethink your client functionality, pub/sub, and UI perhaps.

[adamgins]

yes right... each client only needs a small subset (< 20) notifications

[adamgins]

I have installed redis locally and connected to

"redisOplog": {
		"redis": {

      "host":"localhost",
      "port": 6379
		},
		"retryIntervalMs": 10000,
		"mutationDefaults": {
			 "optimistic": true,
			 "pushToRedis": true
		},
		"debug": true
 	},

the issue is it does not seems to work for some of the publishComposite publications. Wondering if there is some gotcha in the way I have defined those? The weird thing is itworks on my Galaxy server but on on localhost:3000 (development mode).

for some of my subscriptions I see specific channels like Subscribing to channel:teams::-170ab77eda55ee36b9799fac but for others that are not working it seems to be generic like Subscribing to channel: resources

Should the channel name match the publication name ore the collection name?

[evolross]

Not sure. I gave up on publish composite along time ago - too many side effects like the above.

I've found you can roll your own "composite" publishes using Meteor's added, changed, removed functions inside of observeChanges in a publication. Here's a thread on the meteor forums about it - Reactive Joins Without Using Any Packages