Insertion point support?

[dandean]

Does ts-proto support insertion points?

ts-proto/protos/google/protobuf/compiler/plugin.proto

Lines 135 to 172 in 7edc991

	// If non-empty, indicates that the named file should already exist, and the
	// content here is to be inserted into that file at a defined insertion
	// point. This feature allows a code generator to extend the output
	// produced by another code generator. The original generator may provide
	// insertion points by placing special annotations in the file that look
	// like:
	// @@protoc_insertion_point(NAME)
	// The annotation can have arbitrary text before and after it on the line,
	// which allows it to be placed in a comment. NAME should be replaced with
	// an identifier naming the point -- this is what other generators will use
	// as the insertion_point. Code inserted at this point will be placed
	// immediately above the line containing the insertion point (thus multiple
	// insertions to the same point will come out in the order they were added).
	// The double-@ is intended to make it unlikely that the generated code
	// could contain things that look like insertion points by accident.
	//
	// For example, the C++ code generator places the following line in the
	// .pb.h files that it generates:
	// // @@protoc_insertion_point(namespace_scope)
	// This line appears within the scope of the file's package namespace, but
	// outside of any particular class. Another plugin can then specify the
	// insertion_point "namespace_scope" to generate additional classes or
	// other declarations that should be placed in this scope.
	//
	// Note that if the line containing the insertion point begins with
	// whitespace, the same whitespace will be added to every line of the
	// inserted text. This is useful for languages like Python, where
	// indentation matters. In these languages, the insertion point comment
	// should be indented the same amount as any inserted code will need to be
	// in order to work correctly in that context.
	//
	// The code generator that generates the initial file and the one which
	// inserts into it must both run as part of a single invocation of protoc.
	// Code generators are executed in the order in which they appear on the
	// command line.
	//
	// If |insertion_point| is present, |name| must also be present.
	optional string insertion_point = 2;

If not, do you plan on adding support, or do you have pointers to get started adding this feature on my own?

The use-case I have is that I am building my own service generation. I see that there's already quite a bit of service generation code directly in ts-proto, and each is unique and specific in its own way. I don't want to add Yet Another Service Generation option, and it seems like a perfect opportunity for insertion points so that I can write my own plugin which simply amends ts-proto output with my own extensions.

To achieve that I think I would just need one insertion point up top for imports, and one insertion point at the bottom for my own generated service additions.

[dandean]

It looks like I could probably patch this bit of main.ts:

https://github.com/stephenh/ts-proto/blob/main/src/main.ts#L81-L86

And add something like this on line 82 to define an insertion point for imports:

chunks.push(code`// @@protoc_insertion_point(imports)`);

And then add something like this on line 244 to define an insertion point at the end of the file for my own generate services to be inserted:

https://github.com/stephenh/ts-proto/blob/main/src/main.ts#L243-L245

chunks.push(code`// @@protoc_insertion_point(end_of_file)`);

If this would work, one benefit of this would be that all of the existing service-generation options could be pulled out into standalone protoc plugins, which I think would reduce ts-proto complexity a bit.

[dandean]

Ok cool - I have a hacked up version of ts-proto on my machine that's doing this. Here's the output, showing the insertion point, and the code inserted from my own protoc plugin which makes use of the insertion point:

/* eslint-disable */
import Long from 'long';
import _m0 from 'protobufjs/minimal';

import { MyBaseService } from '@fake/service';
// @@protoc_insertion_point(imports)

If you're ok with this feature, I'll work up a PR in the next few days to add these two insertion points.

[dandean]

Oh yeah, and here's the output for the end of file insertion point:

class FooService extends MyBaseService {}
// @@protoc_insertion_point(end_of_file)

[stephenh]

Hm, interesting, no I hadn't seen the insertion point option/approach before.

FWIW @aikoven recently added the outputServices=generic-definitions, which he's using in a framework to do likely similar custom-ish service stuff. (Tangentially, @aikoven wdyt about linking to your project from the ts-proto readme? Maybe either from like a "Community" section or even without the outputServices section of the docs as an example for other users to follow your approach to custom services.)

I'd be tempted to look at that first, and hopefully build your custom services on top of that.

If that really doesn't work, I guess I'm not 100% opposed to the insertion point approach, like a --ts_proto_opt=insertPoints=end_of_file,imports option...

For my own personal tendencies, it seems a little brittle; although I can see how it makes sense for the protoc stuff given their built-in code generators are C++ (!) and so pretty hard to fork/customize/etc. :-)

[aikoven]

wdyt about linking to your project from the ts-proto readme? Maybe either from like a "Community" section or even without the outputServices section of the docs as an example for other users to follow your approach to custom services.

That would be nice! I can make a PR once I have some free time.

Considering the topic of the discussion: AFAIK the core Protobuf already deprecate the insertion point feature, see protocolbuffers/protobuf#6697 (comment)

However, I too see it very useful if we could somehow extend the ts-proto generator. Maybe through some kind of plugin system, though it sounds really complicated. Another option is to be able to use ts-proto as a library, adding custom functionality on top of it.

[dandean]

@aikoven Thanks for the link to insertion points being deprecated – I hadn't seen that before.

One thing I like about the path I've been on is that it generates a complete class with all of the methods defined in the protobuf statically on the TypeScript class itself, and because those generated classes are added to the bottom of the file generated by ts-proto there are no imports to worry about.

Following the path you're recommending, I think my plugin will have to instead write to its own files (instead of amending ts-proto's) adjacent to ts-proto files, and resolve all imports associated with the request and response types for each method.

Thanks for the discussion – it's very helpful.

[stephenh]

Yeah. Just talking out loud, an earlier version of the ts-poet code generation library used to be more object-based (the current API is basically "give me a bunch of strings, I'll concat them and then auto-organize / de-conflict the imports").

The object-based API was more like "here's a function Foo, here's a class Bar with methods zaz", i.e. the library new more about the structure of the output, which had pros/cons, but it would allow a neat sort of approach like:

function myPlugin(output: TsProtoOutput) {
  const file = output.getFile("./simple.ts");
  const foo = file.getClass("FooRpcService");
  const getFoo = foo.getMethod("getFoo");
  foo.setMethod("getFoo", code`
    ...my customization...
    ${getFoo}
    ...my customization...
  `);
  return output;
}

And so you could like muck with the files / methods / functions after ts-proto had already made them, but before they go to disk.

Granted, it would take some infrastructure to build, but I think fundamentally wouldn't be too hard/complex.

That said, I do like the simplicity of ideally using the generic-definitions output.