Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Changes for migration to be postgres to be merged to main #86

Open
wants to merge 28 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
102103b
addind print
sasi2312 Jul 31, 2024
0c23308
query converted to ORM
sasi2312 Aug 19, 2024
2908558
Merge branch 'development' of https://github.com/Code4GovTech/discord…
sasi2312 Aug 20, 2024
20eee57
Merge pull request #75 from Code4GovTech/migration_orm
sasi2312 Aug 21, 2024
353b425
Update build-and-push.yml
jaanbaaz Sep 12, 2024
1f6976c
Updated requirements.txt for psycopg2-binary
Srijan-SS02 Sep 30, 2024
5404627
MIgration from supabase to postgres changes
jaanbaaz Nov 14, 2024
4799c8e
Merge branch 'dev' of github.com:Code4GovTech/discord-bot into dev
jaanbaaz Nov 14, 2024
d36cb2e
MIgration from supabase to postgres changes
jaanbaaz Nov 15, 2024
c20b815
MIgration from supabase to postgres changes
jaanbaaz Nov 15, 2024
83c4199
MIgration from supabase to postgres changes
jaanbaaz Nov 15, 2024
469f82a
Fix env variables for postgres
karntrehan Nov 25, 2024
3364a90
postgres initialization changes
jaanbaaz Nov 26, 2024
6a70571
Merge branch 'dev' of github.com:Code4GovTech/discord-bot into dev
jaanbaaz Nov 26, 2024
cd8c28b
Add dependencies and fix logs
karntrehan Nov 28, 2024
ec6de60
migration to postgres bug fixes in postgres queries
jaanbaaz Dec 10, 2024
3644aed
Merge pull request #87 from Code4GovTech/feature/migration-to-postgres
jaanbaaz Dec 12, 2024
ced3991
Print token
karntrehan Dec 12, 2024
8b7aaf4
Print token
karntrehan Dec 12, 2024
23c444e
Adding logs to token
karntrehan Dec 12, 2024
92e9af3
add dev branch in the build worflow
singhalkarun Dec 13, 2024
0fd0464
is_active added to contributors discord
jaanbaaz Dec 13, 2024
74486e9
is_active added
jaanbaaz Dec 13, 2024
163e7d7
registration bug fix
jaanbaaz Dec 13, 2024
3dfb1e8
email check added
jaanbaaz Dec 15, 2024
bc255cc
discord_username and name fix
jaanbaaz Dec 24, 2024
4a0f5de
log added
jaanbaaz Dec 24, 2024
7319844
Merge pull request #88 from Code4GovTech/feature/unmerged-fix
jaanbaaz Dec 24, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 26 additions & 17 deletions .github/workflows/build-and-push.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,41 +7,50 @@ on:
workflow_dispatch:

env:
DOCKER_USERNAME: ${{ github.actor }}
DOCKER_IMAGE_NAME: ${{ github.repository }}
DOCKER_REGISTRY: ghcr.io
DOCKER_IMAGE_TAG: ${{ github.ref_name }}
REGISTRY: ghcr.io
IMAGE_NAME: ${{ github.repository }}

jobs:
build-and-push:
runs-on: ubuntu-latest
# Sets the permissions granted to the `GITHUB_TOKEN` for the actions in this job.
permissions:
contents: read
packages: write
steps:
- id: lower-repo
shell: pwsh
run: |
"::set-output name=repository::$($env:DOCKER_IMAGE_NAME.ToLowerInvariant())"

- name: Checkout code
uses: actions/checkout@v2

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2

- name: Log in to the Container registry
uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: Login to Docker registry
uses: docker/login-action@v1
- name: Extract metadata (tags, labels) for Docker
id: meta
uses: docker/metadata-action@9ec57ed1fcdbf14dcef7dfbe97b2010124a938b7
with:
registry: ${{ env.DOCKER_REGISTRY }}
username: ${{ env.DOCKER_USERNAME }}
password: ${{ secrets.PAT }}
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
tags: |
# minimal
type=pep440,pattern={{version}},value=${{ github.ref_name }},enable=${{ github.event_name == 'release' }}
# branch event
type=ref,event=branch
type=raw,value=latest,enable=${{ github.event_name == 'release' }}

- name: Build and Push Docker image
uses: docker/build-push-action@v4
with:
build-args: |
"SERVER_RELEASE_VERSION=${{ github.sha }}"
# build-args:
context: .
push: true
cache-from: type=gha
cache-to: type=gha,mode=max
tags: ${{ env.DOCKER_REGISTRY }}/${{ steps.lower-repo.outputs.repository }}:${{env.DOCKER_IMAGE_TAG}}
labels: org.opencontainers.image.source=https://github.com/${{steps.lower-repo.outputs.repository}}
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
12 changes: 6 additions & 6 deletions cogs/badges.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
import discord
from discord.ext import commands

from helpers.supabaseClient import SupabaseClient
from helpers.supabaseClient import PostgresClient


class BadgeModal(discord.ui.Modal, title="Your Badges"):
Expand Down Expand Up @@ -133,7 +133,7 @@ def get_user_badges(self, discord_id):
userBadges = {"points": [], "achievements": []}
if (
len(
SupabaseClient().read(
PostgresClient().read(
"contributors_registration",
query_key="discord_id",
query_value=discord_id,
Expand All @@ -143,7 +143,7 @@ def get_user_badges(self, discord_id):
):
userBadges["achievements"].append(self.discordXGithubBadge)

discordMemberData = SupabaseClient().read(
discordMemberData = PostgresClient().read(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we create 1 class level instance of postgresclient please? Creating a new one each time seems unncessary.

"discord_engagement", "contributor", discord_id
)
if discordMemberData:
Expand All @@ -153,16 +153,16 @@ def get_user_badges(self, discord_id):
userBadges["achievements"].append(self.rockstarBadge)
if discordMemberData[0]["has_introduced"]:
userBadges["achievements"].append(self.apprenticeBadge)
contributorData = SupabaseClient().read(
contributorData = PostgresClient().read(
"contributors_registration", query_key="discord_id", query_value=discord_id
)
if contributorData:
github_id = contributorData[0]["github_id"]
prData = {
"raised": SupabaseClient().read(
"raised": PostgresClient().read(
table="connected_prs", query_key="raised_by", query_value=github_id
),
"merged": SupabaseClient(table="connected_prs").read(
"merged": PostgresClient(table="connected_prs").read(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems to be a different syntax. We should have a uniform syntax across the board.

table="connected_prs", query_key="merged_by", query_value=github_id
),
}
Expand Down
28 changes: 14 additions & 14 deletions cogs/discordDataScraper.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
from discord.channel import TextChannel
from discord.ext import commands, tasks

from helpers.supabaseClient import SupabaseClient
from helpers.supabaseClient import PostgresClient

with open("config.json") as config_file:
config_data = json.load(config_file)
Expand All @@ -28,12 +28,12 @@ def __init__(self, bot) -> None:
@commands.Cog.listener()
async def on_message(self, message):
pass
# contributor = SupabaseClient().read(
# contributor = PostgresClient().read(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets delete the commented code.

# "discord_engagement", "contributor", message.author.id
# )
# print("message", len(message.content))
# if not contributor:
# SupabaseClient().insert(
# PostgresClient().insert(
# "discord_engagement",
# {
# "contributor": message.author.id,
Expand All @@ -46,13 +46,13 @@ async def on_message(self, message):
# if len(message.content) > 20:
# if message.channel.id == INTRODUCTIONS_CHANNEL_ID:
# print("intro")
# SupabaseClient().update(
# PostgresClient().update(
# "discord_engagement",
# {"has_introduced": True},
# "contributor",
# message.author.id,
# )
# SupabaseClient("discord_engagement").update(
# PostgresClient("discord_engagement").update(
# "discord_engagement",
# {"total_message_count": contributor[0]["total_message_count"] + 1},
# "contributor",
Expand All @@ -62,11 +62,11 @@ async def on_message(self, message):
@commands.Cog.listener()
async def on_reaction_add(self, reaction, user):
message = reaction.message
contributor = SupabaseClient().read(
contributor = PostgresClient().read(
"discord_engagement", "contributor", message.author.id
)[0]
if not contributor:
SupabaseClient().insert(
PostgresClient().insert(
"discord_engagement",
{
"contributor": message.author.id,
Expand All @@ -77,7 +77,7 @@ async def on_reaction_add(self, reaction, user):
)
return
print("reaction")
SupabaseClient().update(
PostgresClient().update(
"discord_engagement",
{"total_reaction_count": contributor["total_reaction_count"] + 1},
"contributor",
Expand All @@ -89,7 +89,7 @@ async def add_engagement(self, ctx):
await ctx.channel.send("started")

def addEngagmentData(data):
client = SupabaseClient()
client = PostgresClient()
client.insert("discord_engagement", data)
return

Expand Down Expand Up @@ -145,7 +145,7 @@ async def enable_webhook(self, ctx):
channels = await guild.fetch_channels()
enabled = [
channel["channel_id"]
for channel in SupabaseClient().read_all("discord_channels")
for channel in PostgresClient().read_all("discord_channels")
]
for channel in channels:
try:
Expand All @@ -156,7 +156,7 @@ async def enable_webhook(self, ctx):
webhook = await channel.create_webhook(name="New Ticket Alert")
feedback = f"""URL: {webhook.url}\n Token:{"Yes" if webhook.token else "No"}"""
await ctx.send(feedback)
SupabaseClient().insert(
PostgresClient().insert(
"discord_channels",
{
"channel_id": channel.id,
Expand All @@ -177,7 +177,7 @@ async def update_applicants(self, ctx):
await ctx.send("Member List Count: " + str(len(members)))
for member in members:
try:
SupabaseClient().insert(
PostgresClient().insert(
"applicant",
{"sheet_username": member.name, "discord_id": member.id},
)
Expand Down Expand Up @@ -216,12 +216,12 @@ async def collect_all_messages(self):

async def add_messages(self):
def addMessageData(data):
client = SupabaseClient()
client = PostgresClient()
client.insert("unstructured discord data", data)
return

def getLastMessageObject(channelId):
last_message = SupabaseClient().read_by_order_limit(
last_message = PostgresClient().read_by_order_limit(
table="unstructured discord data",
query_key="channel",
query_value=channelId,
Expand Down
6 changes: 3 additions & 3 deletions cogs/listeners/member_events_cog.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
import discord
from discord.ext import commands

from helpers.supabaseClient import SupabaseClient
from helpers.supabaseClient import PostgresClient


class MemberEventsListener(commands.Cog):
Expand All @@ -11,7 +11,7 @@ def __init__(self, bot) -> None:

@commands.Cog.listener("on_member_join")
async def on_member_join(self, member: discord.Member):
SupabaseClient().updateContributor(member)
PostgresClient().updateContributor(member)

@commands.Cog.listener("on_member_remove")
async def on_member_remove(self, member: discord.Member):
Expand All @@ -20,7 +20,7 @@ async def on_member_remove(self, member: discord.Member):

@commands.Cog.listener("on_member_update")
async def on_member_update(self, before: discord.Member, after: discord.Member):
SupabaseClient().updateContributor(after)
PostgresClient().updateContributor(after)


async def setup(bot: commands.Bot):
Expand Down
6 changes: 3 additions & 3 deletions cogs/listeners/message_events_cog.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,10 @@
from discord.ext import commands

from config.server import ServerConfig
from helpers.supabaseClient import SupabaseClient
from helpers.supabaseClient import PostgresClient

serverConfig = ServerConfig()
supabaseClient = SupabaseClient()
postgresClient = PostgresClient()


async def grantVerifiedRole(member: discord.Member):
Expand All @@ -32,7 +32,7 @@ def __init__(self, bot) -> None:
async def on_message(self, message: discord.Message):
# Listen for Introduction
if message.channel.id == serverConfig.Channels.INTRODUCTION_CHANNEL:
if await supabaseClient.memberIsAuthenticated(message.author):
if await postgresClient.memberIsAuthenticated(message.author):
await grantVerifiedRole(message.author)
else:
return
Expand Down
12 changes: 6 additions & 6 deletions cogs/listeners/role_events_cog.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
import discord
from discord.ext import commands

from helpers.supabaseClient import SupabaseClient
from helpers.supabaseClient import PostgresClient


class RoleEventsListener(commands.Cog):
Expand All @@ -13,27 +13,27 @@ def __init__(self, bot) -> None:
async def on_guild_role_create(self, role: discord.Role):
if role.name.startswith("College:"):
orgName = role.name[len("College: ") :]
SupabaseClient().addChapter(roleId=role.id, orgName=orgName, type="COLLEGE")
PostgresClient().addChapter(roleId=role.id, orgName=orgName, type="COLLEGE")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Class level instance here as well. Lets do it for all classes.

elif role.name.startswith("Corporate:"):
orgName = role.name[len("Corporate: ") :]
SupabaseClient().addChapter(
PostgresClient().addChapter(
roleId=role.id, orgName=orgName, type="CORPORATE"
)

@commands.Cog.listener()
async def on_guild_role_delete(self, role: discord.Role):
SupabaseClient().deleteChapter(roleID=role.id)
PostgresClient().deleteChapter(roleID=role.id)

@commands.Cog.listener()
async def on_guild_role_update(self, before: discord.Role, after: discord.Role):
if after.name.startswith("College:"):
orgName = after.name[len("College: ") :]
SupabaseClient().addChapter(
PostgresClient().addChapter(
roleId=after.id, orgName=orgName, type="COLLEGE"
)
elif after.name.startswith("Corporate:"):
orgName = after.name[len("Corporate: ") :]
SupabaseClient().addChapter(
PostgresClient().addChapter(
roleId=after.id, orgName=orgName, type="CORPORATE"
)

Expand Down
16 changes: 8 additions & 8 deletions cogs/serverManagement.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
from discord.ext import commands, tasks

from config.server import ServerConfig
from helpers.supabaseClient import SupabaseClient
from helpers.supabaseClient import PostgresClient

serverConfig = ServerConfig()

Expand Down Expand Up @@ -40,20 +40,20 @@ async def updateServerData(self, ctx):
if role.name.startswith("College:"):
orgName = role.name[len("College: ") :]
chapterRoles.append(role)
SupabaseClient().addChapter(
PostgresClient().addChapter(
roleId=role.id, orgName=orgName, type="COLLEGE"
)
elif role.name.startswith("Corporate:"):
orgName = role.name[len("Corporate: ") :]
chapterRoles.append(role)
SupabaseClient().addChapter(
PostgresClient().addChapter(
roleId=role.id, orgName=orgName, type="CORPORATE"
)

print("added chapters")

contributorsGithub = SupabaseClient().read_all("contributors_registration")
contributorsDiscord = SupabaseClient().read_all_active("contributors_discord")
contributorsGithub = PostgresClient().read_all("contributors_registration")
contributorsDiscord = PostgresClient().read_all_active("contributors_discord")

## Give contributor role
contributorIds = [
Expand All @@ -71,22 +71,22 @@ async def updateServerData(self, ctx):

print(count)

SupabaseClient().updateContributors(guild.members)
PostgresClient().updateContributors(guild.members)
recordedMembers = [
contributor["discord_id"] for contributor in contributorsDiscord
]
print(f"{len(recordedMembers)} have their data saved")
currentMembers = [member.id for member in guild.members]
membersWhoLeft = list(set(recordedMembers) - set(currentMembers))
print(f"{len(membersWhoLeft)} members left")
SupabaseClient().invalidateContributorDiscord(membersWhoLeft)
PostgresClient().invalidateContributorDiscord(membersWhoLeft)
print("Updated Contributors")

@tasks.loop(minutes=30)
async def assign_contributor_role(self):
guild = self.bot.get_guild(serverConfig.SERVER)
contributorRole = guild.get_role(serverConfig.Roles.CONTRIBUTOR_ROLE)
contributorsGithub = SupabaseClient().read_all("contributors_registration")
contributorsGithub = PostgresClient().read_all("contributors_registration")

contributorIds = [
contributor["discord_id"] for contributor in contributorsGithub
Expand Down
Loading