feat: Add parser for schemarb #87

sasamuku · 2024-11-18T03:22:17Z

Summary

This is proof of concept for parsing schema.rb to DBStructure.

Related Issue

N/A

Changes

ab138d3
- fetch grammar from DBML
- create parser.js by npx peggy parser.pegjs
8939685
- add parser.ts used for entrypoint
- add tests to check if convert correctly

Testing

Other Information

sasamuku · 2024-11-18T03:29:04Z

frontend/packages/db-structure/src/schema/index.ts

+import { relationshipsSchema, tableNameSchema, tableSchema } from './database'
+import { nodeMetaDataSchema } from './erd'


there are two structure types, database and erd, so I divide structure into each file.
If this poc is good enough to merge, I am going to fix other packages where ci is failed.

sasamuku · 2024-11-18T03:41:29Z

frontend/packages/db-structure/src/schema/index.ts

  relationships: relationshipsSchema,
 })

-export type DBStructure = v.InferOutput<typeof dbStructureSchema>
+export type ERDStructure = v.InferOutput<typeof erdStructureSchema>


ERD has a metadata such as node position and colors.
As schema name, ERDStructure is better than DBStructure.
Because DBStructure doesn't seem to have any data related to erd.
How do you think?

I agree that the parser wants to remain ignorant of information about the UI of the ER Diagrams.
I think we should leave DBStructure as it is for this PR right now, since we can consider the schema when implementing ERD!

sasamuku · 2024-11-18T04:27:40Z

frontend/packages/db-structure/src/parser/index.ts

+
+type SupportedFormat = 'schemarb' | 'postgres'
+
+const convertToDBStructure = (data: any): DBStructure => {


convertToDBStructure should not be implemented.
It would be better Generated parser.js to return directly DBStructure.
So I might fix pegjs for next step if this PR is merged.

sasamuku · 2024-11-18T04:30:35Z

frontend/packages/db-structure/src/parser/index.ts

+      acc[table.name] = {
+        comment: null,
+        fields: table.fields.map((field: any) => ({
+          check: null,


there is a lack of fields such as unique and check.
This is because fields parsed by pegjs do not contain these data.

This is also issue to solve by editing pegjs.

frontend/packages/db-structure/src/parser/schemarb/parser.pegjs

hoshinotsuyoshi · 2024-11-18T04:18:20Z

frontend/packages/db-structure/src/parser/schemarb/parser.pegjs

+version = "version"
+do = "do"
+end = "end"
+lambda_function "lambda function" = "=>" / "->"


In Ruby, -> is used as a shorthand for lambda, while => is typically used to define key-value pairs in a Hash.

This was fixed in 8d86366, but => might still appear, so handling => itself might still be necessary. That said, we could also address it if and when any issues arise. Let’s proceed with this for now!

frontend/packages/db-structure/biome.jsonc

…compiler options from tsconfig.json

frontend/packages/db-structure/package.json

MH4GF · 2024-11-18T04:41:35Z

frontend/packages/db-structure/package.json

@@ -7,14 +7,18 @@
    "lint:biome": "biome check .",
    "lint:tsc": "tsc --noEmit",
    "fmt": "conc -c auto pnpm:fmt:*",
-    "fmt:biome": "biome check --write --unsafe ."
+    "fmt:biome": "biome check --write --unsafe .",
+    "test": "vitest"


good 👍🏻

hoshinotsuyoshi · 2024-11-18T05:59:20Z

frontend/packages/db-structure/src/parser/__snapshots__/index.test.ts.snap

sasamuku · 2024-11-18T07:32:35Z

Fixed:

add license comment in parser.pegjs
lodash fucntions are separately installed

Next work:

fix pegjs to fit our schema (tried in this pr but needs more time)
add types to not use Any

MH4GF · 2024-11-18T07:35:03Z

frontend/packages/db-structure/src/parser/schemarb/parser.pegjs

+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.


Thank you for adding it! 😄

sasamuku · 2024-11-18T07:35:25Z

add license comment in parser.pegjs

I refereed to https://github.com/holistics/dbml/blob/master/LICENSE

Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.

sample (MIT not Apache2.0): https://github.com/antlr/grammars-v4/blob/master/sql/postgresql/PostgreSQLLexer.g4

hoshinotsuyoshi

looks good to me!

MH4GF

LGTM 🚀🚀🚀

sasamuku added 5 commits November 18, 2024 11:33

install packages

d981b38

generate parser

ab138d3

fix db structure

8bf7728

add paser entrypoint

8939685

arrange biome and tsconfig

d1af15d

github-actions bot assigned sasamuku Nov 18, 2024

sasamuku commented Nov 18, 2024

View reviewed changes

sasamuku marked this pull request as ready for review November 18, 2024 03:29

sasamuku requested a review from a team as a code owner November 18, 2024 03:29

sasamuku requested review from hoshinotsuyoshi, FunamaYukina, junkisai and MH4GF and removed request for a team November 18, 2024 03:29

sasamuku commented Nov 18, 2024

View reviewed changes

hoshinotsuyoshi reviewed Nov 18, 2024

View reviewed changes

MH4GF added 6 commits November 18, 2024 14:13

docs(biome): Add comment for ignore settings

6051d8f

fix: Move ignore comment to each line

72b688f

Refactor: change to snapshot testing

1403bb2

Refactor: Refactor to use a named export instead of default export

a8408fd

Refactor: Add ts-nocheck comment to parser.js and remove unnecessary …

9002d12

…compiler options from tsconfig.json

fix: remove unused test snapshot

cb2131e

MH4GF reviewed Nov 18, 2024

View reviewed changes

hoshinotsuyoshi reviewed Nov 18, 2024

View reviewed changes

frontend/packages/db-structure/src/parser/__snapshots__/index.test.ts.snap Outdated

Copy link

Member

hoshinotsuyoshi Nov 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

sasamuku added 4 commits November 18, 2024 15:05

undo schema changes

9db0761

add lisense notice

022ec8e

fix syntax

8d86366

install single function from lodash

b8c0606

ts version fixed

5585547

MH4GF reviewed Nov 18, 2024

View reviewed changes

sasamuku requested review from MH4GF and hoshinotsuyoshi November 18, 2024 07:36

hoshinotsuyoshi approved these changes Nov 18, 2024

View reviewed changes

MH4GF approved these changes Nov 18, 2024

View reviewed changes

sasamuku added this pull request to the merge queue Nov 19, 2024

Merged via the queue into main with commit 71ef4cd Nov 19, 2024
8 checks passed

sasamuku deleted the add_parser_for_schemarb branch November 19, 2024 01:00

hoshinotsuyoshi mentioned this pull request Nov 20, 2024

Call function to output DBStructure in cli package #88

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add parser for schemarb #87

feat: Add parser for schemarb #87

sasamuku commented Nov 18, 2024

sasamuku Nov 18, 2024

sasamuku Nov 18, 2024

MH4GF Nov 18, 2024

sasamuku Nov 18, 2024 •

edited

Loading

sasamuku Nov 18, 2024

hoshinotsuyoshi Nov 18, 2024

hoshinotsuyoshi Nov 18, 2024

MH4GF Nov 18, 2024

hoshinotsuyoshi Nov 18, 2024

sasamuku commented Nov 18, 2024

MH4GF Nov 18, 2024

sasamuku commented Nov 18, 2024 •

edited

Loading

hoshinotsuyoshi left a comment

MH4GF left a comment

		import { relationshipsSchema, tableNameSchema, tableSchema } from './database'
		import { nodeMetaDataSchema } from './erd'


		type SupportedFormat = 'schemarb' \| 'postgres'

		const convertToDBStructure = (data: any): DBStructure => {

feat: Add parser for schemarb #87

feat: Add parser for schemarb #87

Conversation

sasamuku commented Nov 18, 2024

Summary

Related Issue

Changes

Testing

Other Information

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sasamuku Nov 18, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sasamuku commented Nov 18, 2024

Choose a reason for hiding this comment

sasamuku commented Nov 18, 2024 • edited Loading

hoshinotsuyoshi left a comment

Choose a reason for hiding this comment

MH4GF left a comment

Choose a reason for hiding this comment

sasamuku Nov 18, 2024 •

edited

Loading

sasamuku commented Nov 18, 2024 •

edited

Loading