Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added Logging Capabilities #118

Draft
wants to merge 3 commits into
base: main
Choose a base branch
from
Draft

Added Logging Capabilities #118

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
21384a0
Added Logging Capabilities
marvinbuss Feb 11, 2022
351b887
linting
marvinbuss Feb 11, 2022
35cd936
linting
marvinbuss Feb 11, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/linters/.arm-ttk.psd1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,5 +11,7 @@
'IDs Should Be Derived From ResourceIDs'
'Parameters Must Be Referenced'
'Variables Must Be Referenced'
'URIs Should Be Properly Constructed'
'apiVersions Should Be Recent'
)
}
1 change: 1 addition & 0 deletions docs/DataManagementAnalytics-AzureDevOpsDeployment.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,7 @@ To begin, please open the [infra/params.dev.json](/infra/params.dev.json). In th
| `purviewManagedStorageId` | Specifies the Resource ID of the managed storage account of the central Purview instance. | `/subscriptions/{subscription-id}/resourceGroups/{rg-name}/providers/Microsoft.Storage/storageAccounts/{storage-account-name}` |
| `purviewManagedEventHubId` | Specifies the Resource ID of the managed Event Hub of the central Purview instance. | `/subscriptions/{subscription-id}/resourceGroups/{rg-name}/providers/Microsoft.EventHub/namespaces/{eventhub-namespace-name}` |
| `enableRoleAssignments` | Specifies whether role assignments should be enabled. | `true` or `false` |
| `enableObservability` | Specifies whether observability capabilities should be enabled. | `true` or `false` |
| `privateDnsZoneIdKeyVault` | Specifies the Resource ID of the private DNS zone for KeyVault. | `/subscriptions/{subscription-id}/resourceGroups/{rg-name}/providers/Microsoft.Network/privateDnsZones/privatelink.vaultcore.azure.net` |
| `privateDnsZoneIdSynapseDev` | Specifies the Resource ID of the private DNS zone for Synapse Dev. | `/subscriptions/{subscription-id}/resourceGroups/{rg-name}/providers/Microsoft.Network/privateDnsZones/privatelink.dev.azuresynapse.net` |
| `privateDnsZoneIdSynapseSql` | Specifies the Resource ID of the private DNS zone for Synapse Sql. | `/subscriptions/{subscription-id}/resourceGroups/{rg-name}/providers/Microsoft.Network/privateDnsZones/privatelink.sql.azuresynapse.net` |
Expand Down
1 change: 1 addition & 0 deletions docs/DataManagementAnalytics-GitHubActionsDeployment.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,7 @@ To begin, please open the [infra/params.dev.json](/infra/params.dev.json). In th
| `purviewManagedStorageId` | Specifies the Resource ID of the managed storage account of the central Purview instance. | `/subscriptions/{subscription-id}/resourceGroups/{rg-name}/providers/Microsoft.Storage/storageAccounts/{storage-account-name}` |
| `purviewManagedEventHubId` | Specifies the Resource ID of the managed Event Hub of the central Purview instance. | `/subscriptions/{subscription-id}/resourceGroups/{rg-name}/providers/Microsoft.EventHub/namespaces/{eventhub-namespace-name}` |
| `enableRoleAssignments` | Specifies whether role assignments should be enabled. | `true` or `false` |
| `enableObservability` | Specifies whether observability capabilities should be enabled. | `true` or `false` |
| `privateDnsZoneIdKeyVault` | Specifies the Resource ID of the private DNS zone for KeyVault. | `/subscriptions/{subscription-id}/resourceGroups/{rg-name}/providers/Microsoft.Network/privateDnsZones/privatelink.vaultcore.azure.net` |
| `privateDnsZoneIdSynapseDev` | Specifies the Resource ID of the private DNS zone for Synapse Dev. | `/subscriptions/{subscription-id}/resourceGroups/{rg-name}/providers/Microsoft.Network/privateDnsZones/privatelink.dev.azuresynapse.net` |
| `privateDnsZoneIdSynapseSql` | Specifies the Resource ID of the private DNS zone for Synapse Sql. | `/subscriptions/{subscription-id}/resourceGroups/{rg-name}/providers/Microsoft.Network/privateDnsZones/privatelink.sql.azuresynapse.net` |
Expand Down
97 changes: 81 additions & 16 deletions docs/reference/portal.dataProduct.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -433,13 +433,21 @@
{
"name": "enableSqlPool",
"label": "Enable SQL Pool",
"type": "Microsoft.Common.CheckBox",
"visible": "[equals(steps('generalSettings').dataProcessingServiceDeploymentSettings.processingService, 'synapse')]",
"defaultValue": false,
"type": "Microsoft.Common.OptionsGroup",
"visible": true,
"toolTip": "Enable the deployment of an Azure SQL Pool (DW100).",
"defaultValue": "No",
"constraints": {
"required": false,
"validationMessage": "Enable the deployment of an Azure SQL Pool (DW100)."
"allowedValues": [
{
"label": "Yes",
"value": "yes"
},
{
"label": "No",
"value": "no"
}
]
}
}
]
Expand Down Expand Up @@ -564,13 +572,21 @@
{
"name": "enableCosmos",
"label": "Enable Azure Cosmos DB",
"type": "Microsoft.Common.CheckBox",
"type": "Microsoft.Common.OptionsGroup",
"visible": true,
"defaultValue": false,
"toolTip": "Enable the deployment of Azure Cosmos DB.",
"defaultValue": "Yes",
"constraints": {
"required": false,
"validationMessage": "Enable the deployment of Azure Cosmos DB."
"allowedValues": [
{
"label": "Yes",
"value": "yes"
},
{
"label": "No",
"value": "no"
}
]
}
},
{
Expand All @@ -585,13 +601,61 @@
{
"name": "enableRoleAssignments",
"label": "Enable role assignments",
"type": "Microsoft.Common.CheckBox",
"type": "Microsoft.Common.OptionsGroup",
"visible": true,
"defaultValue": false,
"toolTip": "Enable role assignments.",
"defaultValue": "Yes (recommended)",
"constraints": {
"allowedValues": [
{
"label": "Yes (recommended)",
"value": "yes"
},
{
"label": "No",
"value": "no"
}
]
}
}
]
},
{
"name": "observabilitySettings",
"label": "Observability Settings",
"type": "Microsoft.Common.Section",
"visible": true,
"elements": [
{
"name": "observabilitySettingsText",
"type": "Microsoft.Common.TextBlock",
"visible": true,
"options": {
"text": "Specify observability settings for this deployment.",
"link": {
"label": "",
"uri": ""
}
}
},
{
"name": "enableObservability",
"label": "Deploy Observability Extension",
"type": "Microsoft.Common.OptionsGroup",
"visible": true,
"toolTip": "If 'Yes' is selected, a Log Analytics workspace as well as a set of Azure dashboards will be deployed.",
"defaultValue": "Yes (recommended)",
"constraints": {
"required": false,
"validationMessage": "Enable role assignments of the Synapse MSI. Please read infobox above for more details."
"allowedValues": [
{
"label": "Yes (recommended)",
"value": "yes"
},
{
"label": "No",
"value": "no"
}
]
}
}
]
Expand Down Expand Up @@ -977,12 +1041,13 @@
"administratorPassword": "[if(empty(steps('generalSettings').generalSettings.administratorPassword), '', steps('generalSettings').generalSettings.administratorPassword)]",
"processingService": "[if(empty(steps('generalSettings').dataProcessingServiceDeploymentSettings.processingService), '', steps('generalSettings').dataProcessingServiceDeploymentSettings.processingService)]",
"synapseDefaultStorageAccountFileSystemId": "[if(empty(steps('generalSettings').dataProcessingServiceDeploymentSettings.synapseDefaultStorageAccountFileSystemId), '', steps('generalSettings').dataProcessingServiceDeploymentSettings.synapseDefaultStorageAccountFileSystemId)]",
"enableSqlPool": "[steps('generalSettings').dataProcessingServiceDeploymentSettings.enableSqlPool]",
"enableCosmos": "[steps('generalSettings').generalSettings.enableCosmos]",
"enableSqlPool": "[equals(steps('generalSettings').dataProcessingServiceDeploymentSettings.enableSqlPool, 'yes')]",
"enableCosmos": "[equals(steps('generalSettings').generalSettings.enableCosmos, 'yes')]",
"purviewId": "[if(empty(steps('generalSettings').dataGovernanceSettings.purviewId.id), '', steps('generalSettings').dataGovernanceSettings.purviewId.id)]",
"purviewManagedStorageId": "[if(empty(steps('generalSettings').dataGovernanceSettings.purviewId.id), '', steps('generalSettings').dataGovernanceSettings.purviewApi.properties.managedResources.storageAccount)]",
"purviewManagedEventHubId": "[if(empty(steps('generalSettings').dataGovernanceSettings.purviewId.id), '', steps('generalSettings').dataGovernanceSettings.purviewApi.properties.managedResources.eventHubNamespace)]",
"enableRoleAssignments": "[steps('generalSettings').generalSettings.enableRoleAssignments]",
"enableRoleAssignments": "[equals(steps('generalSettings').generalSettings.enableRoleAssignments, 'yes')]",
"enableObservability": "[equals(steps('generalSettings').observabilitySettings.enableObservability, 'yes')]",
"subnetId": "[if(empty(steps('connectivitySettings').virtualNetwork.subnetId), '', steps('connectivitySettings').virtualNetwork.subnetId)]",
"privateDnsZoneIdKeyVault": "[if(empty(steps('connectivitySettings').privateDnsZones.privateDnsZoneIdKeyVault), '', steps('connectivitySettings').privateDnsZones.privateDnsZoneIdKeyVault)]",
"privateDnsZoneIdSynapseDev": "[if(empty(steps('connectivitySettings').privateDnsZones.privateDnsZoneIdSynapseDev), '', steps('connectivitySettings').privateDnsZones.privateDnsZoneIdSynapseDev)]",
Expand Down
13 changes: 13 additions & 0 deletions infra/main.bicep
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,8 @@ param purviewManagedStorageId string = ''
param purviewManagedEventHubId string = ''
@description('Specifies whether role assignments should be enabled.')
param enableRoleAssignments bool = false
@description('Specifies whether observability capabilities should be enabled.')
param enableObservability bool = true

// Network parameters
@description('Specifies the resource ID of the subnet to which all services will connect.')
Expand Down Expand Up @@ -101,6 +103,7 @@ var sql001Name = '${name}-sqlserver001'
var mysql001Name = '${name}-mysql001'
var mariadb001Name = '${name}-mariadb001'
var potsgresql001Name = '${name}-postgresql001'
var logAnalytics001Name = '${name}-la001'

// Resources
module keyVault001 'modules/services/keyvault.bicep' = {
Expand Down Expand Up @@ -236,4 +239,14 @@ module postgresql001 'modules/services/postgresql.bicep' = if (sqlFlavour == 'po
}
}

module logAnalytics001 'modules/services/loganalytics.bicep' = if(enableObservability) {
name: 'logAnalytics001'
scope: resourceGroup()
params: {
location: location
tags: tagsJoined
logAnanalyticsName: logAnalytics001Name
}
}

// Outputs
123 changes: 121 additions & 2 deletions infra/main.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
"_generator": {
"name": "bicep",
"version": "0.4.1124.51302",
"templateHash": "2607815037718360205"
"templateHash": "17229161188929746269"
}
},
"parameters": {
Expand Down Expand Up @@ -122,6 +122,13 @@
"description": "Specifies whether role assignments should be enabled."
}
},
"enableObservability": {
"type": "bool",
"defaultValue": true,
"metadata": {
"description": "Specifies whether observability capabilities should be enabled."
}
},
"subnetId": {
"type": "string",
"metadata": {
Expand Down Expand Up @@ -219,7 +226,8 @@
"sql001Name": "[format('{0}-sqlserver001', variables('name'))]",
"mysql001Name": "[format('{0}-mysql001', variables('name'))]",
"mariadb001Name": "[format('{0}-mariadb001', variables('name'))]",
"potsgresql001Name": "[format('{0}-postgresql001', variables('name'))]"
"potsgresql001Name": "[format('{0}-postgresql001', variables('name'))]",
"logAnalytics001Name": "[format('{0}-la001', variables('name'))]"
},
"resources": [
{
Expand Down Expand Up @@ -2066,6 +2074,117 @@
]
}
}
},
{
"condition": "[parameters('enableObservability')]",
"type": "Microsoft.Resources/deployments",
"apiVersion": "2020-10-01",
"name": "logAnalytics001",
"properties": {
"expressionEvaluationOptions": {
"scope": "inner"
},
"mode": "Incremental",
"parameters": {
"location": {
"value": "[parameters('location')]"
},
"tags": {
"value": "[variables('tagsJoined')]"
},
"logAnanalyticsName": {
"value": "[variables('logAnalytics001Name')]"
}
},
"template": {
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"metadata": {
"_generator": {
"name": "bicep",
"version": "0.4.1124.51302",
"templateHash": "6316627051499014174"
}
},
"parameters": {
"location": {
"type": "string"
},
"tags": {
"type": "object"
},
"logAnanalyticsName": {
"type": "string"
}
},
"variables": {
"dataFactoryAnalyticsName": "[format('{0}-adf-analytics', parameters('logAnanalyticsName'))]",
"sqlAnalyticsName": "[format('{0}-sql-analytics', parameters('logAnanalyticsName'))]"
},
"resources": [
{
"type": "Microsoft.OperationalInsights/workspaces",
"apiVersion": "2020-10-01",
"name": "[parameters('logAnanalyticsName')]",
"location": "[parameters('location')]",
"tags": "[parameters('tags')]",
"properties": {
"features": {},
"publicNetworkAccessForIngestion": "Enabled",
"publicNetworkAccessForQuery": "Enabled",
"retentionInDays": 120,
"sku": {
"name": "PerGB2018"
}
}
},
{
"type": "Microsoft.OperationsManagement/solutions",
"apiVersion": "2015-11-01-preview",
"name": "[variables('dataFactoryAnalyticsName')]",
"location": "[parameters('location')]",
"tags": "[parameters('tags')]",
"plan": {
"name": "[format('AzureDataFactoryAnalytics{0}', parameters('logAnanalyticsName'))]",
"product": "OMSGallery/AzureDataFactoryAnalytics",
"promotionCode": "",
"publisher": "Microsoft"
},
"properties": {
"workspaceResourceId": "[resourceId('Microsoft.OperationalInsights/workspaces', parameters('logAnanalyticsName'))]"
},
"dependsOn": [
"[resourceId('Microsoft.OperationalInsights/workspaces', parameters('logAnanalyticsName'))]"
]
},
{
"type": "Microsoft.OperationsManagement/solutions",
"apiVersion": "2015-11-01-preview",
"name": "[variables('sqlAnalyticsName')]",
"location": "[parameters('location')]",
"tags": "[parameters('tags')]",
"plan": {
"name": "[format('AzureSQLAnalytics{0}', parameters('logAnanalyticsName'))]",
"product": "OMSGallery/AzureSQLAnalytics",
"promotionCode": "",
"publisher": "Microsoft"
},
"properties": {
"workspaceResourceId": "[resourceId('Microsoft.OperationalInsights/workspaces', parameters('logAnanalyticsName'))]"
},
"dependsOn": [
"[resourceId('Microsoft.OperationalInsights/workspaces', parameters('logAnanalyticsName'))]"
]
}
],
"outputs": {
"logAnalyticsWorkspaceId": {
"type": "string",
"value": "[resourceId('Microsoft.OperationalInsights/workspaces', parameters('logAnanalyticsName'))]"
}
}
}
}
}
]
}
Loading