Make astra scraper, schemas & docs

[jpahm]

We would like to make an Astra scraper to develop a database of room and room usage data for the API.

We'll also need to develop schemas for the room data we're scraping.

• Determine what we want to scrape from Astra (what data can we reliably provide via scraping?)
• Create an Astra scraper (contact me if you want to work on this, I'll need to provide you the auth for it)

[TyHil]

The data format for the Astra API (https://www.aaiscloud.com/UTXDallas/~api/calendar/CalendarWeekGrid) is a list of these:

[
	"fedac217-4536-11ef-b8d2-85d9c567bf4d",
	357226,
	"FN 2.204 Media Upgrade",
	null,
	null,
	"Meeting",
	null,
	"2024-09-26T00:00:00",
	"2024-09-26T23:30:00",
	5,
	0,
	1410,
	2,
	"86212fa5-5b3a-4a64-bc63-cc9c6e22bac2",
	"075080UTDM",
	"FN",
	"2.204",
	" ",
	"FN 2.204",
	"9845f293-fd21-4dd8-95b4-95eb023fd533",
	null,
	null,
	null,
	0,
	0,
	0,
	"fedac210-4536-11ef-b8d2-85d9c567bf4d",
	85175,
	"Scheduled",
	null,
	"228B22",
	false,
	null,
	0,
	null,
	22
]

which follow the format specified in the fields section of the body:

ActivityId,
ActivityPk,
ActivityName,
ParentActivityId,
ParentActivityName,
MeetingType,
Description,
StartDate,
EndDate,
DayOfWeek,
StartMinute,
EndMinute,
ActivityTypeCode,
ResourceId,
CampusName,
BuildingCode,
RoomNumber,
RoomName,
LocationName,
InstitutionId,
SectionId,
SectionPk,
IsExam,
IsCrosslist,
IsAllDay,
IsPrivate,
EventId,
EventPk,
CurrentState,
NotAllowedUsageMask,
UsageColor,
UsageColorIsPrimary,
EventTypeColor,
MaxAttendance,
ActualAttendance,
Capacity

[democat3457]

Reposting from Discord thread:

API overview: https://help.aais.com/Help/75/index.html?ad_astra_api.htm
Section schema:
https://www.aaiscloud.com/UTXDallas/javascript/astra/entities/Section.js
Sample section query:
https://www.aaiscloud.com/UTXDallas/~api/query/section?fields=Id,SectionTitle,SectionCode&limit=100
Sample section by id:
https://www.aaiscloud.com/UTXDallas/~api/entity/section/byid/c0e18e7c-46be-4307-985a-b45091bbf731
Sample room by id:
https://www.aaiscloud.com/UTXDallas/~api/entity/room/byid/fb0a8704-efb7-4ccc-8e7b-289d53feff66

[TyHil]

Thanks!

[TyHil]

I'm not sure how I'd go about making a schema for this. Seems that may be in the nebula-api repo. @jpahm can you point me in the right direction?

And are parser and uploader changes in the scope of this issue?

[jpahm]

I'm not sure how I'd go about making a schema for this. Seems that may be in the nebula-api repo. @jpahm can you point me in the right direction?

And are parser and uploader changes in the scope of this issue?

@TyHil Sorry for the slow response on this Tyler!

Indeed this would involve adding schemas to the nebula-api repo, and I do agree that it may be a bit outside of scope for this issue. I'm thinking of maybe making a new issue for it and letting some new members of the team try to tackle it if that sounds good to you?

[TyHil]

No worries and sounds great. I ran the scraper again and put the data to base the schema off of here: https://drive.google.com/file/d/1MzcPu7dFYjpIz5AQd8ZjqaLiP4Mfr-rP/view?usp=sharing

[jpahm]

No worries and sounds great. I ran the scraper again and put the data to base the schema off of here: https://drive.google.com/file/d/1MzcPu7dFYjpIz5AQd8ZjqaLiP4Mfr-rP/view?usp=sharing

Awesome, thanks!

[jpahm]

Closing and moving schema creation to #34.