System Architecture
Our system consists of an Android part and a server part. Let's first look at how the two parts work together and learn more about the structure of each system
Green blocks mean Android system
Oragne blocks mean server system
Blue blocks mean data type.
The black arrows show the portion of the Android app that receives a user's sketch and converts it into an architectural object.
When the user draws a sketch, the data is taken from the canvas and delivered to the manager as an array list of points.
The manager sends this data to the converter to recognize it as a building object and store the recognized building object in the stack.
More details will be covered in the Android system section.
Yellow arrows refer to situations in progress when a user attempts to export. When the user presses the export button, the exporter converts the architectural object into a Jason file and passes it to the server.
When the server receives the json file through the communicator, it calls the interpreter and turns into an architectural object.
At this time, unnecessarily overlapping lines or objects are organized.
When the interpreter has converted all the json files into architectural objects, it calls the dxf handler.
The dxf handler receives an artifact and generates a dxf file.
The generated dxf file is returned to the user.
Let's find out the details later
We made the interface into an app because we thought it was advantageous in terms of convenience for users to access the Android app and efficiently recognizing sketches.
The functions performed by the Android app can be classified into a total of three categories, and each function operates organically.
The canvas receives a sketch from a user. At this time, one input is divided using stroke and time. We classify strokes with touch up/down and fixed time, but we recommend using a learned model to advance the system. The canvas stores one input in a list format: x, y, touch down information. The stored list is analyzed through the manager.
There are so many roles that the manager plays in the app. The data input through the canvas should be processed using a converter and processed when the user enters undo or redo.
The converter receives a list of point (x, y, touch down) arrays through the manager and analyzes whether it is a number, an architectural object, or an error. First, a deep learning model was used to check whether it was a number or not.
Basically, the mnist dataset was used, but there were many errors in recognition because some numbers were different from the number handwriting of Koreans. In order to secure this, errors were reduced by additionally learning team members' numerical handwriting data.
Next, it was checked whether it was an architectural object or not if it was not a number.
We made the triangle recognize as a window, the colored square as a pillar, the square on the wall as a window, and the other square as a wall. This part solved the problem by using Douglas Parker algorithms and mathematical techniques without using deep learning models.
The manager uses two stacks to store data.
The main stack stores architectural objects converted using converters.
The redo stack temporarily stores the data removed when the user presses the undo button.
If the user presses the redo button, the data is moved back to the main stack one by one.
When the user enters new information, the redo stack data is discarded.
The stack passes information on existing objects to the converter. This is to calculate a correlation with an existing object.
The exporter operates only when the user presses the export button. Convert all architectural objects in the stack into json files. It also sends the converted data to the server
As mentioned above, we implemented the interface as an app, but the part that converts Json into a cAD file was implemented as Python and connected to a server format. If you want to create a different type of interface, you can convert it into a cad file by implementing it only in Json format and passing it to the server. Or if you want to modify it further, please use our code.
The server operates through three main handlers: communicator, interpreter, and dxfhandler Each process may be used independently.
Communicator is responsible for communicating with Android apps. Our server is simply implemented through socket communication because it is for testing purposes. For commercialization, it must be developed to communicate using http. Currently, the server receives the Json file through tcp socket communication. Hand over the received file to the interpreter.
The interpreter plays the most important role, turning the Json file into an architectural object. The json file has only coordinate and length data on the location of the wall, the location of the door, and the location of the column. Therefore, in order to draw doors, windows, and columns between walls, you must modify the information on the wall. If the walls are connected to each other, the lines of the overlapping parts should be arranged. It is the interpreter's role to organize information until just before the line is drawn in the cad file.
The dxf handler is responsible for converting the information organized by the interpreter into a cad file. To proceed with this process, we used a Python library called exdxf. Since exdxf only supports simple functions that can divide layers and draw lines and curves, we used it to draw doors, windows, and columns.
The server reads the data from the json, converts it into an architectural object, calculates it, and writes it as a dxf file.
We were able to send data in many ways, but we chose the Json format as the communication data format to open source later. This is because the Json file is in text format, so anyone can easily read it and use it as a data storage method in various environments. If you are curious about the rules for making the Json file, please refer to the API README or User guide. (https://github.com/GP-sketch2CAD/API/blob/main/README.md)
Architectural objects were developed by us to easily utilize data in code. Architectural objects are largely composed of walls, windows, pillars, and doors. All architectural objects represent straight lines in the form of lists of x and y coordinates and curves in x, y coordinates, radii, and angles. Coordinating architectural objects is a very cumbersome task, so we implemented it to create architectural objects with minimal input.
There are many different programs that can create CAD, and each program has a different data format. However, the dxf format is supported by most CAD programs because it is a format created to send and receive data even in different programs. Therefore, we chose the dxf extension, which can also be implemented on AutoCAD, which is the most used in the construction industry. Also, since dxf is composed of ASCII code, it can be easily modified with code.