bskyweb: add robots disallow and 'canonical' config flags (DO NOT MERGE YET)

bskyweb: add robots disallow and 'canonical' config flags (DO NOT MERGE YET) #18804