diff --git a/.gitignore b/.gitignore index 84ba15c..d7b48bf 100644 --- a/.gitignore +++ b/.gitignore @@ -5,3 +5,4 @@ _site vendor .DS_Store Thumbs.db +.vs diff --git a/Gemfile.lock b/Gemfile.lock index fdf9e1d..c6e0e52 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -215,6 +215,8 @@ GEM racc (~> 1.4) nokogiri (1.13.10-x86_64-darwin) racc (~> 1.4) + nokogiri (1.13.10-x86_64-linux) + racc (~> 1.4) octokit (4.25.1) faraday (>= 1, < 3) sawyer (~> 0.9) @@ -256,6 +258,7 @@ GEM PLATFORMS arm64-darwin-23 universal-darwin-22 + x86_64-linux DEPENDENCIES github-pages (~> 228) diff --git a/_mydocs/contoso-data-generator/config-data.md b/_mydocs/contoso-data-generator/config-data.md new file mode 100644 index 0000000..040ad57 --- /dev/null +++ b/_mydocs/contoso-data-generator/config-data.md @@ -0,0 +1,37 @@ +--- +layout: page +title: Configuration data (data.xlsx) +menu_title: Configuration data +published: true +order: /07 +--- + +The Excel configuration file contains both fixed data and parameters to control the distribution of random data. For example, from here you can decide the relative percentage of orders for categories and subcategories. + +The file contains several sheets, further described here. Each sheet contains multiple columns. The software reads some of the columns recognizing them by name. Columns with names that do not follow the standard requirements of the software are ignored. Columns have been conveniently colored in yellow if they are used by the software. Any non-yellow color is considered a comment and it is useful only for human purposes. + +### Categories +From here you can configure sales of categories using two curves: W and PPC. "W" define the relative weight of each category in the set of all categories for different periods in the entire timeframe. "PPC" define the variation in the price of items of each category during the whole period (Price percent). Normally the last column is 100%. + +### Subcategories +From here you can configure sales of subcategories using a weight curve with columns marked with W. The values are used to define the weight of a subcategory inside its category. Therefore, the numbers are summed by category and then used to weight subcategories inside the category. + +### Subcatlinks +On this page, you can configure the likelihood that one product in a subcategory triggers the purchase of another product in another subcategory. The values are in percentage: <17, 18, 80%> means that if a product of subcategory 17 is added to an order, there is an 80% chance that a product of subcategory 18 will be added to the same order. + +### Products +On this page, you configure, for each product, the initial price and the distribution of sales of the product over different periods. The weights identified in the W columns are relative to the subcategory to which the product belongs. + +### CustomerClusters +On this page, you define clusters of customers. Each cluster is defined by two columns: OW (OrderWeight) and CW (CustomerWeight). Order Weight defines the percentage of orders assigned to customers belonging to the cluster, whereas CustomerWeight defines the percentage of the total customers used to fill the cluster. + +It is possible to define a large cluster of customers that generates a small number of orders. The number of clusters is free. + +### GeoAreas +This page is intended to define geographical areas, each with a set of weights to change the activity of the area over time. Each area is independent of the other and the definition of geographical areas needs to be done at the leaf level: no grouping is provided. +For each geographic area, you define the W columns to provide the activity spline. + +### Stores +On this page, you enumerate the stores. For each store, you provide its geographical area and the open and close date. A store is active only between the two dates. +You do not provide weight activity for the stores, as the behavior is dictated by the customer clusters. A special store marked -1 as StoreID defines the online store. +Each order is assigned to either the online store or to a local store depending on the country of the customer. \ No newline at end of file diff --git a/_mydocs/contoso-data-generator/config-json.md b/_mydocs/contoso-data-generator/config-json.md new file mode 100644 index 0000000..9b650c5 --- /dev/null +++ b/_mydocs/contoso-data-generator/config-json.md @@ -0,0 +1,58 @@ +--- +layout: page +title: Configuration file (config.json) +menu_title: Configuration file +published: true +order: /06 +--- + +This file contains the main configuration of the data generator. +- **OrdersCount**: (int) total number of orders to be generated. + +- **StartDT**: (datetime) date of the first order. + +- **YearsCount**: (int) total number of years generated. Orders are distributed over the years. + +- **CutDateBefore**, **CutDateAfter**: (datetime optional parameters) the 2 parameters allow to create data starting from a day different from January 1st and ending on a date different from December 31st. Data before CutDateBefore and after CutDateAfter is removed + +- **CustomerPercentage** : percentage of customers to be used. Range: 0.001 - 1.000 + +- **OutputFormat** : format of the data to be generated. Values: CSV, PARQUET, DELTATABLE + +- **SalesOrders** : type if data to be generated. Values: SALES ORDERS BOTH. SALES = creates the "sales" table. ORDERS = creates the "orders" and the "orders details" table. BOTH = creates all the previous tables. + +- **CustomerFakeGenerator**: (int) number of full random customers. Only used during tests for speeding up the process. + +- **DaysWeight** (section) + + - **DaysWeightConstant**: (bool) if set to true, the configuration about days is ignored. + + - **DaysWeightPoints**, **DaysWeighValues**: (double[]) points for interpolating the curve of distribution of orders over time. It covers the entire YearsCount period. + + - **DaysWeightAddSpikes**: (bool) if set to false, annual spikes are ignored. + + - **WeekDaysFactor**: (double[] - length 7) weight multiplication factor for each day of the week. The first day is Sunday. + + - **DayRandomness**: (double) percentage of randomness add to days, to avoid having a too-perfect curve over time. + +- **OrderRowsWeights**: (double[]) distribution of the number of rows per order. Each element is a weight. The first element is the weight of orders with one row, the second is the weight of orders with two rows. and so on + +- **OrderQuantityWeights**: (double[]) distribution of the quantity applied to each order row. Each element is a weight. The first element is the weight of rows with quantity=1, the second element is the weight of rows with quantity=2, and so on. + +- **DiscountWeights**: (double[]) distribution of the discounts applied to order rows. Each element is a weight. The first element is the weight of rows with a discount of 0%, the second element is the weight of rows with a discount of 1%, and so on. + +- **OnlinePerCent**: (double[]) distribution of the percentage of orders sold online, over the orders total. + +- **DeliveryDateLambdaWeights**: (double[]) distribution of the days for delivery. The delivery date is computed by adding one day plus a random number generated using the distribution built from this parameter. + +- **CountryCurrency**: table mapping Country to Currency + +- **AnnualSpikes** : set of periods where orders show a spike. For each spike, you define the start day, the end day, and the multiplication factor. + +- **OneTimeSpikes**: set of spikes with a fixed start and end date. For each spike, you define the start end, the end date, and the multiplication factor. + +- **CustomerActivity** : contains the configuration for customer start/end date + + - **StartDateWeightPoints**, **StartDateWeightValues**: configuration for the spline of customer start date + + - **EndDateWeightPoints**, **EndDateWeightValues**: configuration for the spline of customer end dates diff --git a/_mydocs/contoso-data-generator/databasegenerator.md b/_mydocs/contoso-data-generator/databasegenerator.md deleted file mode 100644 index cf44709..0000000 --- a/_mydocs/contoso-data-generator/databasegenerator.md +++ /dev/null @@ -1,123 +0,0 @@ ---- -layout: page -title: DatabaseGenerator -menu_title: DatabaseGenerator -published: true -order: /2 ---- -The DatabaseGenerator.exe is a tool to generate orders for the Contoso data model in order to provide demo data. - -## Overview and usage -The DatabaseGenerator.exe is a command-line application, running on .NET Core 3.1. It requires a working folder to run on, containing a configuration file and data input files. After executing, the working folder also contains generated output data files and logs. - -Usage: -```bash -databasegenerator.exe inputfolder outputfolder configfile [param:OrdersCount=nnnnnnn] -``` - -Input files: -- config.json : main app configuration -- data.xlsx : static data (products, categories, …) and their configuration (distribution, weights, …) -- customer.rpt : customers list - -Output files: -- orders.csv -- orderRows.csv -- customerCluster.csv -- log.log - -## Model - - - -## Json Configuration file (config.json) - -This file contains the main configuration of the data generator. -- **OrdersCount**: (int) total number of orders to be generated. - -- **StartDT**: (datetime) date of the first order. - -- **YearsCount**: (int) total number of years generated. Orders are distributed over the years. - -- **CutDateBefore**, **CutDateAfter**: (datetime optional parameters) the 2 parameters allow to create data starting from a day different from January 1st and ending on a date different from December 31st. Data before CutDateBefore and after CutDateAfter is removed - -- **CustomerFakeGenerator**: (int) number of fake customers. If > 0, customers.rpt file is ignored and a number of fake customers are generated. - -- **DaysWeight** (section) - - - **DaysWeightConstant**: (bool) if set to true, the configuration about days is ignored. - - - **DaysWeightPoints**, **DaysWeighValues**: (double[]) points for interpolating the curve of distribution of orders over time. It covers the entire YearsCount period. - - - **DaysWeightAddSpikes**: (bool) if set to false, annual spikes are ignored. - - - **WeekDaysFactor**: (double[] – length 7) weight multiplication factor for each day of the week. The first day is Sunday. - - - **DayRandomness**: (double) percentage of randomness add to days, to avoid having a too-perfect curve over time. - -- **OrderRowsWeights**: (double[]) distribution of the number of rows per order. Each element is a weight. The first element is the weight of orders with one row, the second is the weight of orders with two rows. and so on - -- **OrderQuantityWeights**: (double[]) distribution of the quantity applied to each order row. Each element is a weight. The first element is the weight of rows with quantity=1, the second element is the weight of rows with quantity=2, and so on. - -- **DiscountWeights**: (double[]) distribution of the discounts applied to order rows. Each element is a weight. The first element is the weight of rows with a discount of 0%, the second element is the weight of rows with a discount of 1%, and so on. - -- **OnlinePerCent**: (double[]) distribution of the percentage of orders sold online, over the orders total. - -- **DeliveryDateLambdaWeights**: (double[]) distribution of the days for delivery. The delivery date is computed by adding one day plus a random number generated using the distribution built from this parameter. - -- **CountryCurrency**: table mapping Country to Currency - -- **AnnualSpikes** : set of periods where orders show a spike. For each spike, you define the start day, the end day, and the multiplication factor. - -- **OneTimeSpikes**: set of spikes with a fixed start and end date. For each spike, you define the start end, the end date, and the multiplication factor. - -- **CustomerActivity** : contains the configuration for customer start/end date - - - **StartDateWeightPoints**, **StartDateWeightValues**: configuration for the spline of customer start date - - - **EndDateWeightPoints**, **EndDateWeightValues**: configuration for the spline of customer end dates - - -## Customers list (customers.rpt) - -A file containing customers' information. Format: text, tab-delimited. Only the first two columns are used by the tool: CustomerID (int) and GeoAreaID (int). The following columns are ignored. - - -## Excel configuration file (data.xlsx) - -The Excel configuration file contains both fixed data and parameters to control the distribution of random data. For example, from here you can decide the relative percentage of orders for categories and subcategories. - -The file contains several sheets, further described here. Each sheet contains multiple columns. The software reads some of the columns recognizing them by name. Columns with names that do not follow the standard requirements of the software are ignored. Columns have been conveniently colored in yellow if they are used by the software. Any non-yellow color is considered a comment and it is useful only for human purposes. - -### Categories -From here you can configure sales of categories using two curves: W and PPC. “W” define the relative weight of each category in the set of all categories for different periods in the entire time frame. “PPC” define the variation in the price of items of each category during the whole period (Price percent). Normally the last column is 100%. - -### Subcategories -From here you can configure sales of subcategories using a weight curve with columns marked with W. The values are used to define the weight of a subcategory inside its category. Therefore, the numbers are summed by category and then used to weight subcategories inside the category. - -### Subcatlinks -On this page, you can configure the likelihood that one product in a subcategory triggers the purchase of another product in another subcategory. The values are in percentage: <17, 18, 80%> means that if a product of subcategory 17 is added to an order, there is an 80% chance that a product of subcategory 18 will be added to the same order. - -### Products -On this page, you configure, for each product, the initial price and the distribution of sales of the product over different periods. The weights identified in the W columns are relative to the subcategory to which the product belongs. - -### CustomerClusters -On this page, you define clusters of customers. Each cluster is defined by two columns: OW (OrderWeight) and CW (CustomerWeight). Order Weight defines the percentage of orders assigned to customers belonging to the cluster, whereas CustomerWeight defines the percentage of the total customers used to fill the cluster. - -It is possible to define a large cluster of customers that generates a small number of orders. The number of clusters is free. - -### GeoAreas -This page is intended to define geographical areas, each with a set of weights to change the activity of the area over time. Each area is independent of the other and the definition of geographical areas needs to be done at the leaf level: no grouping is provided. -For each geographic area, you define the W columns to provide the activity spline. - -### Stores -On this page, you enumerate the stores. For each store, you provide its geographical area and the open and close date. A store is active only between the two dates. -You do not provide weight activity for the stores, as the behavior is dictated by the customer clusters. A special store marked -1 as StoreID defines the online store. -Each order is assigned to either the online store or to a local store depending on the country of the customer. - -### Output files - -- customersClusters.csv : ClusterID, CustomerID -- orders.csv : OrderID, CustomerID, StoreID, DT -- orderRows.csv: OrderID. RowNumber, ProductID, Quantity, Price1, Price2 -- log.log diff --git a/_mydocs/contoso-data-generator/details.md b/_mydocs/contoso-data-generator/details.md new file mode 100644 index 0000000..bf7c1e7 --- /dev/null +++ b/_mydocs/contoso-data-generator/details.md @@ -0,0 +1,41 @@ +--- +layout: page +title: Details +menu_title: Details +published: true +order: /01 +--- + +## Data structure + +Output elements: + - Customers + - Stores + - Dates + - CurrencyExchanges + - Sales + - Orders & OrderRows (optional) + +Data schema (Sales version): + +![Schema Sales](images/schema-sales.svg) + + +Data schema (Orders & OrderRows version): + +![Schema Sales](images/schema-orders.svg) + +
+ +Customers set of data is filled with fake, but realistic, customers data. + + +## Pre-data-preparation: static data from SQLBI repository + +The tool needs some files containing static data: fake customers, exchange rates, postal codes, etc. The files are cached under cache folder specified as a parameter on the command line. The files are downloaded from a specific SQLBI repository if not found in the cache folder. In normal usage, if you reuse the same cache folder, the files are downloaded only on the first run. +After downloading, some files are processed to create a consistent set of fake customers. The output file, customersall.csv, is placed under cache folder. If you delete it, it will be recreated on the following run. + +https://github.com/sql-bi/Contoso-Data-Generator-V2-Data/releases/tag/static-files + + + diff --git a/_mydocs/contoso-data-generator/formats.md b/_mydocs/contoso-data-generator/formats.md new file mode 100644 index 0000000..5aae658 --- /dev/null +++ b/_mydocs/contoso-data-generator/formats.md @@ -0,0 +1,68 @@ +--- +layout: page +title: Output formats and related parameters +menu_title: Output formats +published: true +order: /02 +next_reading: true +--- + +Every output format has specific parameter to be set inside config.json + +## CSV + +| Parameter | Values | Notes | +| -- | -- | -- | +| OutputFormat | "CSV" | | +| CsvMaxOrdersPerFile | -1 or a number >1 | Maximum number of Orders per file | +| CsvGzCompression | 0 or 1 | Apply GZ compression to output CSV files | + +For creating a single big CSV file: +``` +"OutputFormat": "CSV" +"CsvMaxOrdersPerFile": -1 +"CsvGzCompression": 0 +``` + +For creating multiple CSV files: +``` +"OutputFormat": "CSV" +"CsvMaxOrdersPerFile": 50000 +"CsvGzCompression": 0 +``` + +For creating multiple CSV.GZ files: +``` +"OutputFormat": "CSV" +"CsvMaxOrdersPerFile": 50000 +"CsvGzCompression": 1 +``` + +## Parquet + +| Parameter | Values | Notes | +| -- | -- | -- | +| OutputFormat | "PARQUET" | | +| ParquetOrdersRowGroupSize | integer | Number of orders per parquet Row Group. Default value is 500000. Do not change if not strictly required. + +Example: + +``` +"OutputFormat": "PARQUET" +``` + + +## Delta Table + +| Parameter | Values | Notes | +| -- | -- | -- | +| OutputFormat | "DELTATABLE" | | +| DeltaTableOrdersPerFile | integer | Number of orders per parquet file. | +| ParquetOrdersRowGroupSize | integer | Number of orders per parquet Row Group. Default value is 500000. Do not change if not strictly required. + +Example: + +``` +"OutputFormat": "DELTATABLE" +"DeltaTableOrdersPerFile": 250000 +``` diff --git a/_mydocs/contoso-data-generator/images/contoso-dataset-model.png b/_mydocs/contoso-data-generator/images/contoso-dataset-model.png deleted file mode 100644 index 04c810c..0000000 Binary files a/_mydocs/contoso-data-generator/images/contoso-dataset-model.png and /dev/null differ diff --git a/_mydocs/contoso-data-generator/images/fabric_01.png b/_mydocs/contoso-data-generator/images/fabric_01.png new file mode 100644 index 0000000..83660f5 Binary files /dev/null and b/_mydocs/contoso-data-generator/images/fabric_01.png differ diff --git a/_mydocs/contoso-data-generator/images/onelake-01.png b/_mydocs/contoso-data-generator/images/onelake-01.png new file mode 100644 index 0000000..3ca4869 Binary files /dev/null and b/_mydocs/contoso-data-generator/images/onelake-01.png differ diff --git a/_mydocs/contoso-data-generator/images/onelake-02.png b/_mydocs/contoso-data-generator/images/onelake-02.png new file mode 100644 index 0000000..250774d Binary files /dev/null and b/_mydocs/contoso-data-generator/images/onelake-02.png differ diff --git a/_mydocs/contoso-data-generator/images/onelake-03.png b/_mydocs/contoso-data-generator/images/onelake-03.png new file mode 100644 index 0000000..b1a919f Binary files /dev/null and b/_mydocs/contoso-data-generator/images/onelake-03.png differ diff --git a/_mydocs/contoso-data-generator/images/onelake-10.png b/_mydocs/contoso-data-generator/images/onelake-10.png new file mode 100644 index 0000000..dda0353 Binary files /dev/null and b/_mydocs/contoso-data-generator/images/onelake-10.png differ diff --git a/_mydocs/contoso-data-generator/images/onelake-11.png b/_mydocs/contoso-data-generator/images/onelake-11.png new file mode 100644 index 0000000..7e0c265 Binary files /dev/null and b/_mydocs/contoso-data-generator/images/onelake-11.png differ diff --git a/_mydocs/contoso-data-generator/images/onelake-12.png b/_mydocs/contoso-data-generator/images/onelake-12.png new file mode 100644 index 0000000..08d750f Binary files /dev/null and b/_mydocs/contoso-data-generator/images/onelake-12.png differ diff --git a/_mydocs/contoso-data-generator/images/schema-orders.svg b/_mydocs/contoso-data-generator/images/schema-orders.svg new file mode 100644 index 0000000..5bbbab8 --- /dev/null +++ b/_mydocs/contoso-data-generator/images/schema-orders.svg @@ -0,0 +1,4 @@ + + + +
Orders
Orders
Customers
Customers
Products
Products
Stores
Stores
OrderRows
OrderRows
Dates
Dates
CurrencyExchanges
CurrencyExchangesText is not SVG - cannot display \ No newline at end of file diff --git a/_mydocs/contoso-data-generator/images/schema-sales.svg b/_mydocs/contoso-data-generator/images/schema-sales.svg new file mode 100644 index 0000000..afb3937 --- /dev/null +++ b/_mydocs/contoso-data-generator/images/schema-sales.svg @@ -0,0 +1,4 @@ + + + +
Sales
Sales
Customers
Customers
Dates
Dates
CurrencyExchanges
CurrencyExchanges
Products
Products
Stores
StoresText is not SVG - cannot display \ No newline at end of file diff --git a/_mydocs/contoso-data-generator/images/sql-01.png b/_mydocs/contoso-data-generator/images/sql-01.png new file mode 100644 index 0000000..f5d8047 Binary files /dev/null and b/_mydocs/contoso-data-generator/images/sql-01.png differ diff --git a/_mydocs/contoso-data-generator/index.md b/_mydocs/contoso-data-generator/index.md index c9dc455..2445a1c 100644 --- a/_mydocs/contoso-data-generator/index.md +++ b/_mydocs/contoso-data-generator/index.md @@ -7,16 +7,57 @@ order: /contoso-data-generator next_reading: true --- -The Contoso Data Generator is a tool to generate sample databases with random generated orders for the Contoso data model in order to provide demo data. +The Contoso Data Generator is a tool for generating sample data with random generated orders for the Contoso data model in order to provide demo data. Generated data is ready to be imported into PowerBI, Fabric OneLake and other platforms. -It consist of a c# program, to generate the orders and a set of powershell and t-sql scripts, to generate the database, that import the generated orders files, and fixed customers, product, stores, and currency exchange files. +It consist of a c# program, to generate the data plus additional scripts for simplifying the activity, importing data to sql-server, etc. The tool is available on GitHub: [https://github.com/sql-bi/Contoso-Data-Generator-V2/](https://github.com/sql-bi/Contoso-Data-Generator-V2/) -The tool is available on github: +If you are just interested in using **ready to use sets of data** generated by the tool, [download them here.](https://github.com/sql-bi/Contoso-Data-Generator-V2-Data) -[Contoso-Data-Generator](https://github.com/sql-bi/Contoso-Data-Generator) +Supported [output formats](formats.md): + - Parquet + - Delta Table (files) + - CSV + - CSV multi file + - CSV multi file - gz compressed + - Sql Server, via bulk-insert script of the generated CSV files -The release v1.0.0 contains a set of downloadable Contoso databases of different sizes, a ready-to-use version of the tool, and the source code in compressed formats. +
-The pre-generated database files require the SQL Server 2019 or newer, but we expect the tool to be working starting from SQL Server 2016 (we did just a few tests with SQL Server 2017). +Delta Table output can be directly used in Fabric LakeHouse without any conversion: -[Contoso-Data-Generator-release-v1.0.0](https://github.com/sql-bi/Contoso-Data-Generator/releases/tag/v1.0.0) +

+ +## Usage + +DataGenerator runs at command-line and requires four mandatory elements to run: + - a configuration file (json) + - a data file (excel) + - an output folder + - a cache folder + - [optional parameters] + +``` +databasegenerator.exe configfile datafile outputfolder cachefolder [param:AAAAA=nnnn] [param:BBBBB=mmmm] +``` +Example: + +``` +databasegenerator.exe c:\temp\config.json c:\temp\data.xlsx c:\temp\OUT\ c:\temp\CACHE\ +``` + +
+ +**Note**: the tool needs some files containing static data: fake customers, exchange rates, postal codes, etc. The files are cached after been downloaded over the Internet from a specific SQLBI repository. [More details](details.md). + +
+ +To simplify running the tool, [a set of scripts](scripts.md) is available. + +
+
+
+ +The current version is the evolution of the older one, still available on GitHub: +[Previous version](https://github.com/sql-bi/Contoso-Data-Generator) + + \ No newline at end of file diff --git a/_mydocs/contoso-data-generator/scripts.md b/_mydocs/contoso-data-generator/scripts.md index 9e3371a..1f119d5 100644 --- a/_mydocs/contoso-data-generator/scripts.md +++ b/_mydocs/contoso-data-generator/scripts.md @@ -3,25 +3,16 @@ layout: page title: Scripts menu_title: Scripts published: true -order: /1 +order: /03 --- -## QuickRun.ps1 -QuickRun purpose is to create the working folders and then run the **GenerateDatabase.ps1** script -## GenerateDatabases.ps1 +Under `script/dataset`, there are 3 scripts: + - `make_tool.cmd` : compiles the tool in release mode, using dotnet from the command line. + - `build_all.cmd` : creates the sets of data published on the ready-to-use repository. + - `build_single.cmd` : create a single set of data. -Generate several Contoso databases per run, by iterating over an array of configuration parameters to run in sequence - - - **DatabaseGenerator** to generate the csv files with the orders - - SQL Script to create the database, import the data, backup the database and detach the .mdf and .ldf files - - zip/7-Zip to compress the database files - -## Parameters - -The parameters are documented into the script inside the **param()** section - -## Configurations - -The configuration for the databases to be generated are specified as Tuples into the **$databases** array +Steps: + - run `build_tool.cmd` + - run `build_single.cmd`. When asked, enter the code of the set of data you want to create. E.g.: `csv-100k`, `delta-1m`, `parquet-10m`, etc. \ No newline at end of file diff --git a/_mydocs/contoso-data-generator/sqlscripts.md b/_mydocs/contoso-data-generator/sqlscripts.md new file mode 100644 index 0000000..274309d --- /dev/null +++ b/_mydocs/contoso-data-generator/sqlscripts.md @@ -0,0 +1,40 @@ +--- +layout: page +title: SQL Scripts +menu_title: SQL Scripts +published: true +order: /11 +--- + +# Scripts for importing data into SQL Server + +The set of scripts under `scripts/sql` allow you to import CSV output files to a Sql Server database. The scripts create the required tables, indexes and relationship and import data using sql bulk insert. The underlying SQL scripts are executed using SQLCMD from the command line. + + +## Sql_ImportData.cmd + +**Warning**: on every run, the script deletes and recreates the tables from scratch. Be careful: do not point the script to a production database or to a database in which you already have data. **You risk to lose the existing data**. + +Steps: + - Create the set of data, as CSV files, as usual. + - Copy the output files under `scripts/sql/inputcsv` + - Adapt the script to your SQL server instance. SQLCMD requires sql server name and other parameters to connect to you sql server. The script defaults are: + `sqlcmd -S (LocalDb)\MSSQLLocalDB -d ContosoDGV2Test` + - Run the import script. When asked, choose what to import: + - sales : mamages base tables + sales table + - orders : mamages base tables + orders/order-rows tables + - both: mamages base tables + sales/orders/order-rows tables + + +Resulting database: + +![Img2](images/sql-01.png) + + + +## SQLBI_ALL_DB.cmd + +Specific scripts used by SQLBI for creating Sql Server database backups, available in the ready-to-use repository. The database are: Contoso 100k, Contoso 1M, Contoso 10M and Contoso 100M. + + - `SQLBI_ALL_DB.cmd` : import data into the 4 databases, backup them and compress the resulting file. + - `SQLBI_CreateSqlDatabases.ps1` : creates the 4 databases on the specified SQL Server \ No newline at end of file diff --git a/_mydocs/contoso-data-generator/uploadtofabric.md b/_mydocs/contoso-data-generator/uploadtofabric.md new file mode 100644 index 0000000..6456e95 --- /dev/null +++ b/_mydocs/contoso-data-generator/uploadtofabric.md @@ -0,0 +1,65 @@ +--- +layout: page +title: Upload data to Fabric LakeHouse +menu_title: LakeHouse +published: true +order: /10 +--- + +Delta Table output files generated by the tool can be easily and directly uploaded to Fabric LakeHouse without any conversions. Just copy all the files, respecting the folders structures, directly under "Tables" LakeHouse folder, and data will be ready to use. + +

+ + +There are 3 ways to upload files to Fabric LakeHouse directly: + - OneLake File Explorer (Windows only) + - Azure Storage Explorer + - AzCopy from the command line + +## OneLake File Explorer + +Currently in preview. It integrates with Windows File Explorer and allows you to manage files on Fabric with a OneDrive-like user-experience. Details here: https://learn.microsoft.com/en-us/fabric/onelake/onelake-file-explorer + +Data generated by the tool, uploaded to LakeHouse: + +

+

+

+ +## Azure Storage Explorer + +Details here: https://learn.microsoft.com/en-us/fabric/onelake/onelake-azure-storage-explorer + +Data generated by the tool, uploaded to LakeHouse: + +

+

+

+ + +## AzCopy + +First of all, login with AzCopy: + +``` +azcopy login +``` + +To test connectivity, try to list the files on the storage Tables partition: + +``` +azcopy list https://onelake.blob.fabric.microsoft.com//.Lakehouse/Tables --trusted-microsoft-suffixes onelake.blob.fabric.microsoft.com +``` + +Upload all file: + +``` +azcopy copy "../out/*" https://onelake.blob.fabric.microsoft.com//.Lakehouse/Tables --recursive --check-length --put-md5 --trusted-microsoft-suffixes onelake.blob.fabric.microsoft.com +``` + +It also possible to delete all existing files. **Be very careful!** +Import: for some unknown reasons, the command often fails. Retry many times till all files have been deleted successfully. + +``` +azcopy remove https://onelake.blob.fabric.microsoft.com//.Lakehouse/Tables/* --recursive=true --trusted-microsoft-suffixes onelake.blob.fabric.microsoft.com +``` \ No newline at end of file